text:
Feature viz saving on missing fields

Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #511 (abce43d) into main (558aebc) will increase coverage by 18.36%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #511       +/-   ##
=============================================
+ Coverage     54.26%   72.62%   +18.36%
  Complexity      272      272
=============================================
  Files           238       38      -200
  Lines          7519     2130     -5389
  Branches       1537      236     -1301
=============================================
- Hits           4080     1547     -2533
+ Misses         3244      421     -2823
+ Partials        195      162       -33

Flag
Coverage Δ

dashboards-observability
?

opensearch-observability
72.62% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...er/visualizations/config_panel/configPanelItem.tsx

...ublic/components/visualizations/charts/bar/bar.tsx

...ic/components/visualizations/charts/lines/line.tsx

...ublic/components/visualizations/charts/pie/pie.tsx

...helpers/ppl_docs/language_structure/identifiers.ts

...public/components/explorer/docTable/docViewRow.tsx

...onfig_editor/config_controls/config_data_links.tsx

...ts/visualizations/charts/financial/gauge/gauge.tsx

...blic/components/visualizations/charts/vis_types.ts

...vability/common/constants/application_analytics.ts

... and 190 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 558aebc...abce43d. Read the comment docs.

bug bashh

Let's go

authors: codecov-commenter mengweieric

repo: opensearch-project/observability


text:
Where is PercolateQueryBuilder class in opensearch java client ?

I don't see org.opensearch.percolator.PercolateQueryBuilder in opensearch high level client, i am using below dependency
    <dependency>
        <groupId>org.opensearch.client</groupId>
        <artifactId>opensearch-rest-high-level-client</artifactId>
        <version>1.2.3</version>
    </dependency>

What is the alternate for org.elasticsearch.percolator.PercolateQueryBuilder

@ankampraveen the org.opensearch.percolator.PercolateQueryBuilder is provided by percolator-client (comes with transport module), seems like you need to add:
<dependency>
    <groupId>org.opensearch.plugin</groupId>
    <artifactId>percolator-client</artifactId>
    <version>1.2.3</version>
</dependency>

source: gharchive/issue

authors: ankampraveen reta

repo: opensearch-project/opensearch-java

license_type: permissive

license_source: github-api


text:
[BUG] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]]

What is the bug?
When I attempt to start OpenSearch 2.11.0 with data from our ElasticSearch 7.5.2 setup I get the following exception in the logs:
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]
How can one reproduce the bug?
Use the following docker-compose.yml to startup elasticsearch with kibana:
services:
  elasticsearch-node:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
    container_name: elasticsearch-node
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - ./volumes/data/:/usr/share/elasticsearch/data
      - ./volumes/logs/:/usr/share/elasticsearch/logs
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - elasticsearch-net

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.5.2
    environment:
      ELASTICSEARCH_HOSTS: '["http://elasticsearch-node:9200"]'
    ports:
      - 5600:5601
    networks:
      - elasticsearch-net

networks:
  elasticsearch-net:

bring down ElasticSearch and Kabana (docker-compose down)
Use the following docker-compose to start OpenSearch making sure that the volumes mapped on opensearch-node are the same folders that were used when starting ElasticSearch previously
version: "3"
services:
  opensearch-node:
    image: opensearchproject/opensearch:2.11.0
    container_name: opensearch-node
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "DISABLE_SECURITY_PLUGIN=true"
      - action.auto_create_index=false
    ports:
      - 9201:9200
      - 9601:9600 # required for Performance Analyzer
    volumes:
      - ./volumes/data/:/usr/share/opensearch/data
      - ./volumes/logs/:/usr/share/opensearch/logs
    networks:
      - opensearch-net

  opensearch-dashboard:
    image: opensearchproject/opensearch-dashboards:2.11.0
    container_name: opensearch-dashboard
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["http://opensearch-node:9200"]'
    networks:
      - opensearch-net

networks:
  opensearch-net:

The exception should show in the logs.
What is the expected behavior?
No exception and OpenSearch starts successfully (at the moment it doesn't)
What is your host/environment?
POP_OS 22.04 LTS
Do you have any screenshots?
Nothing of value
Do you have any additional context?
Here's the stacktrace in the OpenSearch logs
opensearch-node         | [2023-11-17T23:03:55,449][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [0fdcca6d2139] uncaught exception in thread [main]
opensearch-node         | org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         | Caused by: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:252) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:289) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:271) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:167) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.node.Node.start(Node.java:1368) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       ... 6 more
opensearch-node         | Caused by: org.opensearch.index.mapper.MapperParsingException: Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [local_metadata]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:440) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:426) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.merge(MapperService.java:412) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:248) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:289) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:271) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:167) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.node.Node.start(Node.java:1368) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       ... 6 more
opensearch-node         | Caused by: org.opensearch.index.mapper.MapperParsingException: No handler for type [flattened] declared on field [local_metadata]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:381) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:296) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:265) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:389) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:296) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:183) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:144) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:133) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:438) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:426) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.merge(MapperService.java:412) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:248) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:289) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:271) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:167) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.node.Node.start(Node.java:1368) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       ... 6 more
opensearch-node         | uncaught exception in thread [main]
opensearch-node         | java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]]
opensearch-node         | Likely root cause: MapperParsingException[No handler for type [flattened] declared on field [local_metadata]]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:381)
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:296)
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:265)
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:389)
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:296)
opensearch-node         |       at org.opensearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:183)
opensearch-node         |       at org.opensearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:144)
opensearch-node         |       at org.opensearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:133)
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:438)
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:426)
opensearch-node         |       at org.opensearch.index.mapper.MapperService.merge(MapperService.java:412)
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:248)
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121)
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:289)
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:271)
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:167)
opensearch-node         |       at org.opensearch.node.Node.start(Node.java:1368)
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339)
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413)
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
opensearch-node         |       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
opensearch-node         |       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
opensearch-node         |       at org.opensearch.cli.Command.main(Command.java:101)
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
opensearch-node         | For complete error details, refer to the log at /usr/share/opensearch/logs/docker-cluster.log

Hi @reformit  - Have you tried updating to OpenSearch 1.x before moving on to OpenSearch 2.x? It's important to note that multi-version upgrades are not supported in both Elasticsearch and OpenSearch. Could you please wrap your *.yml configuration in triple backticks in the bug description? This will help preserve the indentation in the YAML files, which is crucial for parsing and building with docker-compose.

Hi @sumobrian, thanks for your reply!  I've updated with backticks as requested.  I don't completely understand your question about moving from OS 1.x to 2.x?  I'm trying to migrate from our ElasticSearch 7.5.2 stack to OpenSearch.  Are you suggesting I migrate to OS 1.x from 7.5.2 and then upgrade OS 1.x to 2.x?  If that's the case I have tried it with OS 1.x and still had the same issue.  Appreciate any suggestions/guidance you can give.

The flattened data type doesn't exist in OpenSearch. Here is the equivalent: https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/

Anybody have any suggestion how to perform upgrade without removing Kibana index?

Our solution to migration from ES 7.5.2 to OS 1/2.x is documented here. The documentation and feature set has been greatly enhanced since this issue was opened. If you have any questions please open another issue. Although migrations are currently not available on the migration console, it is available within this repo and noted in the referenced wiki documentation. Here is the internal documentation. Please reach out to me or open another issue if you run into issues with your upgrade/migration process.

authors: ComBin reformit sumobrian wbeckler


text:
Add exit code to compile-scss script on failure

Description
Add exit code to compile-scss script on failure. Previously, if this script had an error, it wouldn't propagate, which would cause false positives in our CI workflows.
Check List

[ ] New functionality includes testing.
[ ] New functionality has been documented.
[x] All tests pass

[x] yarn lint
[x] yarn test-unit

[x] Update CHANGELOG.md
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Do you want these to be the same exit code, or different to distinguish between them?

I don't think it matters much for this case, as long as it isn't 0, the workflows will fail. Maybe it could be useful for some case, but the point of this PR is to make the command propagate errors. If we find a use case in the future, this is an easy thing to change

source: gharchive/pull-request

repo: opensearch-project/oui


text:
[BUG] ODFE docker-compose.yml refers to wrong version of Kibana.

Description
The docker-compose.yml file linked to on https://opendistro.github.io/for-elasticsearch/  has a typo. The kibana: subsection lists the docker image: (amazon/opendistro-for-elasticsearch-kibana:1.13.3) which doesn't exist. It was suggested in https://github.com/opendistro-for-elasticsearch/opendistro-build/issues/793 to change to latest which worked just fine for me.
To Reproduce

Download docker-compose.yml from link in description.
Run docker-compose up
See error:

Status: Downloaded newer image for amazon/opendistro-for-elasticsearch:1.13.3
Pulling kibana (amazon/opendistro-for-elasticsearch-kibana:1.13.3)...
ERROR: manifest for amazon/opendistro-for-elasticsearch-kibana:1.13.3 not found: manifest unknown: manifest unknown

Expected behavior
docker-compose up should work with no modifications to the yml.
Additional context
Just kicking the tires on ODFE.

My apologies - I misfiled this under the wrong repo. Will re-file.

repo: opensearch-project/project-website


text:
[OCSI]: Adds new author ruilin808

Description
Adds new author ruilin808
Issues Resolved
[List any issues this PR will resolve]
Check List

[ x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Closing as we are past deadline and there are outstanding changes.


text:
Internal authentication DB username spec inconsistency

While developing a custom kibana plugin for a client looking to deploy an ODfE cluster, my team has been asked to configure user access using the internal authentication database.
We've developed functionality to add users to the internal auth db via the REST API without an issue, initially.
While updating some of our test users via the Kibana security plugin rather than the REST API, we've been presented with the following error trying to save the modified user:
Username must not contain '*', '.' or curly brackets
However, the REST API enforces no such requirements on usernames for newly created or modified users. Users created via the REST API using an email address as a username are able to log in as we had expected without issue.
Could someone address this inconsistency and (preferably) remove the restriction on . characters in usernames via the kibana security plugin, or alternatively (and definitely not preferred) add the restriction to the REST API for consistency?

Can we get some feedback on this so we know whether or not we need to switch to a different solution?

Still nothing?

Thanks for raising this. We will look into this.

I currently hit this issue also. We have several FQDN of servers used as username which worked very well (I am not sure if this restiricton was already in ODFE 1.3.0 and we just did not reognize it) but now with ODFE 1.9.0 I get this error.
Please remove the dots-restriction as it would be difficult to get non-conflicting usernames otherwise

@DJSundog @WennySoft This appears to be fixed in latest Opensearch version 1.2.0, perhaps you might want to think about upgrading if you are still using old odfe versions.

We are doing some "spring cleaning in the fall", and to make sure we focus our energies on the right issues and we get a better picture of the state of the repo, we are closing all issues that we are carrying over from the ODFE era (ODFE is no longer supported/maintained, see post here).
If you believe this issue should still be considered for current versions of OpenSearch, apologies! Please let us know by re-opening it.
Thanks!

authors: Anthony7774 DJSundog WennySoft davidlago hardik-k-shah

repo: opensearch-project/security


text:
Fixing minor typos.

Reading the spec for the first time, these jumped out as minor mistakes.

once the CLA is signed, I'm ok with these edits.
LGTM

hmm @mattmcneeney shouldn't those links be good now?

@duglin yes they should. I think the problem may be that this branch needs to be rebased on master?

Ok let me try to rebase.

I think I had rebased already but I had not pushed my local master to github. Done now.

Looks like it's going to pass this time :)

LGTM

LGTM

LGTM

LGTM

repo: openservicebrokerapi/servicebroker

license_source: bigquery


text:
[release-v1.9] Conformance tests against MTChannelBasedBroker

This is a backport of upstream commit which was supposed to get to midstream automatically but somehow didn't get there.
This is required for finishing https://github.com/openshift-knative/serverless-operator/pull/2065

/assign @pierDipi

repo: openshift-knative/eventing-kafka-broker


text:
bmh: add HostFirmwareSettings resource and unit tests

Adds the HostFirmwareSettings resource to the bmh package and unit tests for the functions it exports.

cc: @dgonyier

repo: openshift-kni/eco-goinfra


text:
{{pull_request_number}} does not work.

I have a pipelinerun with params:
apiVersion: tekton.dev/v1beta1 kind: PipelineRun metadata:   name: pr-pipeline   annotations:     pipelinesascode.tekton.dev/on-event: "[pull_request]"     pipelinesascode.tekton.dev/on-target-branch: "master"     pipelinesascode.tekton.dev/task: "[git-clone]"     pipelinesascode.tekton.dev/max-keep-runs: "2" spec:   params:     - name: repo_url       value: ""     - name: gitrevision       value: {{revision}}     - name: repo_path       value: {{repo_name}}     - name: pull_request_number       value: {{pull_request_number}}
Here when i add the pull request number params, the pipeline stops working. I've tested it through both github and bitbucket. I get the same error and PR number is not extracted.
invalid map key: map[interface {}]interface {}{"pull_request_number":interface {}(nil)}"}

@AsfaMumtaz
Which PAC version you are using ?

@AsfaMumtaz
Which PAC version you are using ?

Pipelines as Code: v0.5.10

authors: AsfaMumtaz savitaashture

repo: openshift-pipelines/pipelines-as-code


text:
data race in unittests

[go-ninja-go : unittest] === RUN   TestHandleEvent
[go-ninja-go : unittest] === RUN   TestHandleEvent/get_http_call
[go-ninja-go : unittest] === RUN   TestHandleEvent/invalid_json_body
[go-ninja-go : unittest] === RUN   TestHandleEvent/invalid_json_body_only_when_payload_has_been_set
[go-ninja-go : unittest] === RUN   TestHandleEvent/valid_event
[go-ninja-go : unittest] === RUN   TestHandleEvent/skip_event
[go-ninja-go : unittest] ==================
[go-ninja-go : unittest] WARNING: DATA RACE
[go-ninja-go : unittest] Write at 0x00c0006b1800 by goroutine 44:
[go-ninja-go : unittest]   github.com/openshift-pipelines/pipelines-as-code/pkg/adapter.listener.handleEvent.func1()
[go-ninja-go : unittest]       /workspace/source/pkg/adapter/adapter.go:111 +0x2a4
[go-ninja-go : unittest]   net/http.HandlerFunc.ServeHTTP()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:2122 +0x4d
[go-ninja-go : unittest]   net/http.serverHandler.ServeHTTP()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:2936 +0x682
[go-ninja-go : unittest]   net/http.(*conn).serve()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:1995 +0xbd4
[go-ninja-go : unittest]   net/http.(*Server).Serve.func3()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:3089 +0x58
[go-ninja-go : unittest]
[go-ninja-go : unittest] Previous read at 0x00c0006b1800 by goroutine 40:
[go-ninja-go : unittest]   github.com/openshift-pipelines/pipelines-as-code/pkg/adapter.listener.handleEvent.func1.1()
[go-ninja-go : unittest]       /workspace/source/pkg/adapter/adapter.go:189 +0x77
[go-ninja-go : unittest]
[go-ninja-go : unittest] Goroutine 44 (running) created at:
[go-ninja-go : unittest]   net/http.(*Server).Serve()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:3089 +0x817
[go-ninja-go : unittest]   net/http/httptest.(*Server).goServe.func1()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/httptest/server.go:310 +0xbb
[go-ninja-go : unittest]
[go-ninja-go : unittest] Goroutine 40 (running) created at:
[go-ninja-go : unittest]   github.com/openshift-pipelines/pipelines-as-code/pkg/adapter.listener.handleEvent.func1()
[go-ninja-go : unittest]       /workspace/source/pkg/adapter/adapter.go:188 +0x1185
[go-ninja-go : unittest]   net/http.HandlerFunc.ServeHTTP()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:2122 +0x4d
[go-ninja-go : unittest]   net/http.serverHandler.ServeHTTP()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:2936 +0x682
[go-ninja-go : unittest]   net/http.(*conn).serve()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:1995 +0xbd4
[go-ninja-go : unittest]   net/http.(*Server).Serve.func3()
[go-ninja-go : unittest]       /usr/lib/golang/src/net/http/server.go:3089 +0x58
[go-ninja-go : unittest] ==================
[go-ninja-go : unittest]     testing.go:1446: race detected during execution of test


text:
Disable exluders in inventory

@sdodson here Im desabling the excluders globally for intall/upgrade plyabooks. I guess this should be enough. Thoughts ?
ccing @stevekuznetsov

Yes, assuming that's used for all playbooks that should work.

If we are shipping with them on by default I do not see why this is something we want to do.

Until we have 1.5 builds without broken excluders I think this is the right thing to do.

Yeah, that was my point in https://github.com/openshift/origin/issues/13814#issuecomment-295644998
But I though we want to get rid of flakes that are cause by issue in the openshift-ansible-3.5.46

I confirmed with @smarterclayton yesterday that we can probably drop the GA install for good -- we get 80% of the benefit from last install + upgrade, and we don't get the whole host of flakes from it. Haven't had time to communicate that to @jhadvig  yet, sorry. I'd rather do that first, if that doesn't fix it disable them ONLY for the install, if that doesn't work it may be reasonable to disable globally, but that needs to be a last resort.

@stevekuznetsov so just to sum it up... we are going to install the latest release of previous verions and then upgrade to the one we build ? is that it ?
So not:  install GA -> upgrade latest -> upgrade to build
But:  install latest -> upgrade to build
Also you said probably ... how much probable is that ? since we will need to modify few things(jobs/scripts/...) :smile:

Yeah, it's going to be a bunch of work for you... I would confirm this morning with @smarterclayton but that "probably" was 99% as I understand it. We should see if it's ever going to be a long term plan to do GA -> latest -> upgrade, or if it's never going to be, since that will color how you change up the scripts.

Ok so shall I start to work on it ? or creating an issue should be enough for now ?

Let's wrap up the card for this sprint -- email, etc -- then we can start a WIP for the changes/

Alternatively if the bug is severe (happening on more than 1/10 jobs) let's do this change for now and get them back in later. @sdodson your call on frequency

Looks like about 7/12 test_pull_request_origin jobs since 10pm last night failed in this way.

2/2 merge_pull_request_origin failed in this way too

OK, let's get this in and clean up the rest of it later.

authors: jhadvig sdodson stevekuznetsov

repo: openshift/aos-cd-jobs


text:
OPNET-415: Revert the revert "Extend infra CR to store VIPs and MachineNetwork""

In https://github.com/openshift/api/pull/1593 we have implemented a new API targeting OCP 4.15
In https://github.com/openshift/api/pull/1694 we have reverted this change as the feature has been deferred till OCP 4.16
Here we revert the revert with the desire to merge it after the 4.15 branching so that it lands in master but not in release-4.15

/cc @JoelSpeed
/cc @cybertron
/hold
To unhold only after branching

Hey @JoelSpeed, back in 4.16, can we get this back again?

/lgtm

/test e2e-aws-serial-techpreview
Failure in namespace/openshift-monitoring

authors: JoelSpeed mkowalski

repo: openshift/api


text:
[OSD-4992] Remove unused func SetBYOCAccountClaimStatusAWSAccountInUse

Ref : OSD-4992

Codecov Report

Merging #626 (16e09ef) into master (1fb6a5b) will increase coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #626      +/-   ##
==========================================
+ Coverage   20.20%   20.26%   +0.05%
==========================================
  Files          46       46
  Lines        4439     4426      -13
==========================================
  Hits          897      897
+ Misses       3482     3469      -13
  Partials       60       60

Impacted Files
Coverage Δ

pkg/apis/aws/v1alpha1/accountclaim_types.go
10.00% <ø> (ø)

pkg/controller/utils/conditions.go
15.62% <ø> (+1.17%)
:arrow_up:

pkg/controller/utils/utils.go
47.69% <ø> (ø)

authors: codecov-commenter dkeohane

repo: openshift/aws-account-operator


text:
Raise file and process limits before building an image

Attempt to set the limit on open files and processes while we're building an image to 0x100000 (1048576), overriding the default open files limit that we get when generating a runtime configuration.

/test e2e-aws-builds

@bparees @adambkaplan ptal

/lgtm

@gabemontero seeing the same jenkins test failure here: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_builder/69/pull-ci-openshift-builder-master-e2e-aws-builds/203

/retest

/test e2e-aws-builds

authors: bparees mrunalp nalind

repo: openshift/builder


text:
Revert "OCPCLOUD-2051: Merge https://github.com/kubernetes/cloud-provider-aws:master (cab0100) into master"

Reverts openshift/cloud-provider-aws#41
Per OpenShift policy, we are reverting this breaking change to get CI and/or nightly payloads flowing again.
This change broke SLB PDB disruption on 4.14 both ci and nightly payloads, as proven by the payload-aggregate job run below.
To unrevert this, revert this PR, and layer an additional separate commit on top that addresses the problem. Before merging the unrevert, please run these jobs on the PR and check the result of (job/X or job/X, test/Y tuple) to confirm the fix has corrected the problem:
/payload-aggregate periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-aws-ovn-upgrade 10
And examine the output of each of the 10 sub-jobs for occurrences of [sig-network-edge] Application behind service load balancer with PDB remains available using new connections failing.

/payload-aggregate periodic-ci-openshift-release-master-ci-4.14-upgrade-from-stable-4.13-e2e-aws-ovn-upgrade 10

No failures of the SLB PDB test, this is our culprit. Proceeding with revert.

/lgtm

/approve

/lgtm

/label jira/valid-bug

Bug for re-introduction: https://issues.redhat.com/browse/OCPBUGS-14696

/verify-owners

Replaced by https://github.com/openshift/cloud-provider-aws/pull/43
/close

authors: DennisPeriquet JoelSpeed dgoodwin xueqzhan

repo: openshift/cloud-provider-aws


text:
add trust for new admin kubeconfig client-ca

pulls the admin client-ca.crt from installer and sets it up to be included in the client-ca for our server.

/lgtm

/retest

/retest

/retest


text:
Device plugin

Supersedes https://github.com/openshift/dpu-operator/pull/26

/override ci/prow/e2e-dpu-operator-tests

go vet ./...
# github.com/openshift/dpu-operator/daemon/device-plugin
daemon/device-plugin/deviceplugin.go:120:44: cannot use &emptypb.Empty{} (value of type *emptypb.Empty) as *dpu_api.Empty value in argument to nf.client.GetDevices

/lgtm

/lgtm

/override ci/prow/e2e-dpu-operator-tests

repo: openshift/dpu-operator


text:
lab 11 run through

updated with blog posts.

Fixed by #159

authors: mvazquezc scollier

repo: openshift/federation-dev


text:
[WIP] fix(e2e): fix and stabilize e2e tests

Description
Fix and stabilize tests. Debug the Advanced test behavior on a CI system.

[test]

[test]

[test]

[test]

[test]

[test]

[test]

[test]

[test]

[test]

[test]

/retest

/retest

[test]

[test]

[test]

[test]

/lgtm

[test]

/lgtm

authors: JoaoPedroPP Katka92 testcara

repo: openshift/hac-dev


text:
test: e2e: exclude catalog pods from restart check

Switching to file-based catalogs (FBC) in 4.11 has made registry container start times highly variable, sometimes in excess of the 150s StartupProbe timeout.
Exclude those pods from the restart check for now so they don't flake the e2e runs.
https://github.com/openshift/hypershift/pull/1746
https://github.com/operator-framework/operator-lifecycle-manager/pull/2791

/lgtm

authors: enxebre sjenning

repo: openshift/hypershift


text:
Release 4.8 backport 20211121

weekly backport of changes into 4.8 branch

/hold

/retest

/lgtm
/approve

/hold cancel


text:
MGMT-19154: BMO never mounted dataimage into IBI provisioned SNO

There's a race between the DataImage creation and the reboot annotation added in updateBMHProvisioningState
This PR adds a check that the DataImage creationTimestamp is more than the DataImageCoolDownPeriod ago (set to 1 second by default).
In case the the dataImage was created withing the DataImageCoolDownPeriod the controller will
return early and requeue the ICI CR after the DataImageCoolDownPeriod.
This should prevent IBIO from adding the reboot annotation before BMO cached client notice that a DataImage exists.

/lgtm

Is this supposed to resolve that bug? If so should we move it to MGMT or ACM and file it against IBIO properly?
Or are we waiting for some fix from the BMO team? In which case we should open a separate issue to mitigate this until it is fixed on their side.

Is this supposed to resolve that bug? If so should we move it to MGMT or ACM and file it against IBIO properly?
Or are we waiting for some fix from the BMO team? In which case we should open a separate issue to mitigate this until it is fixed on their side.

This should resolve the bug, it's still unknown if BMO can going to make a proper fix for it.
So I guess let's open another one for MGMT and link it to OCPBUGS-43330

/hold
No more changes to 2.7 as we're in RC now

/hold
No more changes to 2.7 as we're in RC now

Can we push this already?

/lgtm

Not quite yet. Release is still not out

/unhold

/ok-to-test

repo: openshift/image-based-install-operator


text:
Refactor to create and destroy kind cluster in e2e tests

          I think we could refactor this PR to create the kind cluster in the Makefile? Correct?

Originally posted by @rphillips in https://github.com/openshift/instaslice-operator/pull/177#discussion_r1806676712

/assign

Separating the targets to create and destroy kind clusters leaves the test suite to just fetch the existing kubeconfig and proceed running the e2e tests.
This helps the test suite to be agnostic to the cluster type and so the extra checks can be removed from the code and restrict the changes just to the Makefile

/close
Closing this issue as the change is part of the following PRs
https://github.com/openshift/instaslice-operator/pull/203
https://github.com/openshift/instaslice-operator/pull/232

repo: openshift/instaslice-operator


text:
Bug 1970962: Remove dependency to acegi-security in favor of jbcrypt (because aceg…

Remove dependency to acegi-security in favor of jbcrypt (because acegi was remove from jenkins 2.277.x

/bugzilla refresh

/bugzilla refresh

/test e2e-aws-jenkins

/bugzilla refresh

/bugzilla refresh

/test e2e-aws-jenkins

/retest

/lgtm
/override ci/prow/e2e-aws-jenkins
/override ci/prow/e2e-aws

/lgtm

/lgtm
/override ci/prow/e2e-aws-jenkins
/override ci/prow/e2e-aws

/lgtm

/override ci/prow/e2e-aws-jenkins
/override ci/prow/e2e-aws

/override ci/prow/images

authors: akram jkhelil

repo: openshift/jenkins


text:
controllercmd: Only observe config file if provided

Since providing --config to a controller is optional, the config file should not be observed if unspecified.
This is required for https://github.com/openshift/service-ca-operator/pull/88

/lgtm
/approve

authors: marun sttts

repo: openshift/library-go


text:
controllercmd: pass context via func argument

Passing context.Context via struct field is a Go anti-pattern. We need to pass the context directly via function arguments instead.
After this PR land, the bumps in operator will have to fix the way they pass the context.

/override ci/prow/verify-alpha-build-machinery
This is unrelated to this change and @tnozicka is looking into the failure

/override ci/prow/verify-alpha-build-machinery


text:
OLS-1058 - Adding exception to operator installation for konflux

Description

Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library
[ ] Bump-up library or tool used for development (does not change the final image)
[ ] CI configuration change
[ ] Konflux configuration change

Related Tickets & Documents

Related Issue #
Closes #

Checklist before requesting a review

[ ] I have performed a self-review of my code.
[ ] PR has passed all pre-merge test jobs.
[ ] If it is a core feature, I have added thorough tests.

Testing

Please provide detailed steps to perform tests related to this code change.
How were the fix/results from this change verified? Please provide relevant screenshots or results.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.45%. Comparing base (12c8083) to head (9d4487b).
Report is 24 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1673      +/-   ##
==========================================
- Coverage   94.88%   92.45%   -2.44%
==========================================
  Files          60       60
  Lines        2681     2679       -2
==========================================
- Hits         2544     2477      -67
- Misses        137      202      +65

Flag
Coverage Δ

92.45% <ø> (-2.44%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 6 files with indirect coverage changes

/approve

/override "Red Hat Konflux / lightspeed-service-on-pull-request"

/lgtm

authors: JoaoFula codecov-commenter tisnik

repo: openshift/lightspeed-service


text:
OCPBUGS-41978: Update go.mod to fix CVE - 4.14

Its a back port of PR from main branch: https://github.com/openshift/machine-api-provider-powervs/pull/81

/test images

/jira refresh

/label backport-risk-assessed

repo: openshift/machine-api-provider-powervs


text:
WIP DO-NOT-MERGE: investigate rpm upgrade test failures

/hold
/cc

/test microshift-rpm-upgrade

/test microshift-rpm-upgrade

/test microshift-rpm-upgrade

/test microshift-rpm-upgrade

repo: openshift/microshift


text:
OCPBUGS-43242: upgrade dynamic plugin sdk to remove vulnerable dependencies 4.15

This PR

updates the dynamic plugin sdk to remove vulnerable dependencies
updates other vulnerable dependencies directly

This PR is done directly to release-4.15 as the sdk dependency support varies for 4.15

/hold
until QE checks

/lgtm

PR tested with cluster-bot, monitoring-plugin worked well
/label qe-approved

/unhold

/jira refresh

authors: PeterYurkovich Tai-RedHat jgbernalp

repo: openshift/monitoring-plugin


text:
Provide information on the starter projects available to each component

/kind user-story
/assign
/area devfile
User Story
As a user, I want odo to be able to list starter projects available to each component, and provide some basic information on each starter project including:

Name
Description (this would need to be added to the devfile v2 schema)

This information would give me a better indication as to which starter project fits my needs.
Acceptance Criteria

[ ] It should provide information for all available starter projects associated with each component
[ ] This list command will only be run if the following conditions are met:

Experimental flag has been enabled
The component is a devfile component
There is at least one project defined in the devfile of the given component

Possible Implementations

Include a "project" column in the output of odo catalog list components with a comma-separated list of available projects for each component by the project name.
Allow an argument for the component name in the odo catalog list components command, and then only list details of that component and it's projects. Here we could list both the project name and project description. e.g. odo catalog list components nodejs would list the available projects for the nodejs component.

Another flag could also be used here to show all information about the component's projects.
We could also fall to interactive mode if the user does not specify which project they want, but there are multiple projects defined

Add a flag --showProjects (or another term) to the odo catalog list components and that will mean we show project details in the same table.

Hey @skoh7645 here are my thoughts:
With regards to "starter" project information. To be blunt (and just my opinion, please everyone else chime in), but couldn't this be remedied in either an odo init command or ample online documentation?
The problem being is no having to rely on a command that we have to maintain consistency on with each new feature of Devfile being implemented.
For example, when we type just odo we have this helpful output:
▶ odo
(OpenShift Do) odo is a CLI tool for running OpenShift applications in a fast and automated manner.
Reducing the complexity of deployment, odo adds iterative development without the worry of deploying your source code.

Find more information at https://github.com/openshift/odo

Get started by creating a new application:

 git clone https://github.com/openshift/nodejs-ex && cd nodejs-ex
 odo create nodejs
 odo push

Your nodejs application has now been deployed. odo has pushed the source code, built the application and deployed it on OpenShift.
You can now edit your code in real time and watch as odo automatically deploys your application.

 odo watch

To access your application, create a URL:

 odo url create myurl
 odo push

More information such as logs or what components you've deployed can be accessed with these commands:

 odo describe
 odo list
 odo log

To see a full list of commands, run 'odo --help'

Same for odo create --help:
▶ odo create --help
Create a configuration describing a component.

 If a component name is not provided, it'll be auto-generated.

 A full list of component types that can be deployed is available using: 'odo catalog list'

 By default, builder images (component type) will be used from the current namespace. You can explicitly supply a namespace by using: odo create namespace/name:version If version is not specified by default, latest will be chosen as the version.

Usage:
  odo create <component_type> [component_name] [flags]

Examples:
  # Create new Node.js component with the source in current directory.
  odo create nodejs

  # Create new Node.js component named 'frontend' with the source in './frontend' directory
  odo create nodejs frontend --context ./frontend

  # Create new Java component with binary named sample.jar in './target' directory
  odo create java:8  --binary target/sample.jar

  # Create new Node.js component with source from remote git repository
  odo create nodejs --git https://github.com/openshift/nodejs-ex.git

  # Create new Node.js component with custom ports, additional environment variables and memory and cpu limits
  odo create nodejs --port 8080,8100/tcp,9100/udp --env key=value,key1=value1 --memory 4Gi --cpu 2

In my opinion, instead of implementing "starter project" information into odo catalog list components, I would rather improve the current documentation that we lack with regards to what starter projects to use.
If we were to implement this, a user would have to hunt through running --help commands before finding the one line that mentions starter projects.
Also, instead of implementing a command, couldn't we add a COLUMN to the odo catalog list components output that indicates it's a good starter project?
With regards to your implementation and using -p, we already have the command odo catalog describe that will output all information with regards to the component / service.

@cdrage  Here's my 2 cents:

I don't think it's feasible to just document the starter projects for each devfile component, especially as the list of supported devfiles in odo grow.
I'm concerned that adding an additional column for projects to odo catalog list would make the output too cluttered (especially if the components has multiple projects available to it).

Now, if we only showed a subset of the project info (like repo or project name) in odo catalog list, and kept the full info under odo catalog describe, I think I'd be good with that.

My concern is that if we add subset of the project info for devfile component we may create terminology confusion for user (given we already displayed PROJECT for s2i component) as in the same result page the "Project" points to two definitions respectively:

Project for s2i component: Similar to namespace in k8s world
Project for devfile component: It's the project template that is hosted on GitHub or other source host

odo catalog list components                                                                                 ✔  5s   11:26:40 
Odo OpenShift Components:
NAME          PROJECT       TAGS                                        SUPPORTED
nodejs        openshift     10,8,8-RHOAR,latest                         YES
openjdk18     openshift     latest                                      YES
dotnet        openshift     2.0,latest                                  NO
httpd         openshift     2.4,latest                                  NO
nginx         openshift     1.10,1.12,1.8,latest                        NO
nodejs        openshift     6                                           NO
perl          openshift     5.24,5.26,latest                            NO
php           openshift     7.0,7.1,latest                              NO
python        openshift     2.7,3.5,3.6,latest                          NO
ruby          openshift     2.3,2.4,2.5,latest                          NO
wildfly       openshift     10.0,10.1,11.0,12.0,13.0,8.1,9.0,latest     NO

Odo Devfile Components:
NAME                 DESCRIPTION                           SUPPORTED
maven                Upstream Maven and OpenJDK 11         YES
nodejs               Stack with NodeJS 10                  YES
openLiberty          Open Liberty microservice in Java     YES
java-spring-boot     Spring Boot® using Java               YES

@GeekArthur If we make the clarification that it's a starter project or or template project that shouldn't be an issue.

@GeekArthur If we make the the column header starter project(s) or a template project(s) that shouldn't be an issue, especially if that output in the column is a git repo link.

for odo catalog list components output
I would actually just output the names of the projects, so the user knows what options are available for --downloadSource flag for odo create.
Showing details like git repo link should be displayed in odo catalog describe component <devfile name>
odo catalog list components                                                                                Devfile Components:
NAME                 DESCRIPTION                           SUPPORTED       STARTER PROJECTS
maven                Upstream Maven and OpenJDK 11         YES
nodejs               Stack with NodeJS 10                  YES                nodejs-ex
openLiberty          Open Liberty microservice in Java     YES
java-spring-boot     Spring Boot® using Java               YES                petclinic, other-example

odo catalog describe component java-spring-boot

odo catalog list components                                                                                Devfile Components:
NAME                 DESCRIPTION                           SUPPORTED       STARTER PROJECTS
maven                Upstream Maven and OpenJDK 11         YES
nodejs               Stack with NodeJS 10                  YES                nodejs-ex
openLiberty          Open Liberty microservice in Java     YES
java-spring-boot     Spring Boot® using Java               YES                petclinic, other-example

odo catalog describe component java-spring-boot

apiVersion: 1.0.0
metadata:
  generateName: java-spring-boot
projects:
  - name: petclicnic
    source:
      type: git
      location: "https://github.com/spring-projects/spring-petclinic.git"
  - name: other-example
    source:
      type: git
      location: "https://github.com/other-example/other-example.git"
components:
  - type: dockerimage
    alias: tools
    image: quay.io/eclipse/che-java11-maven:nightly
    memoryLimit: 512Mi
    mountSources: true
    endpoints:
      - name: '8080/tcp'
        port: 8080
    volumes:
      - name: m2
        containerPath: /home/user/.m2
commands:
  - name: devBuild
    actions:
      - type: exec
        component: tools
        command: "mvn package"
  - name: devRun
    actions:
      - type: exec
        component: tools
        command: "java -jar target/*.jar"

doing odo catalog list components nodejs goes against how odo catalog list services and  odo catalog describe services work. If we really want odo catalog list components nodejs that we will have to implement the same behavior for odo catalog list services!

The main agreement we’ve come to is to move the main solution to under odo catalog describe component <component>, which sounds good to me!
The following is what I would like to have agreed on:

Is this command going to accept both devfile and odov1 components? If so, how do we handle the case that a component exists as both a devfile and odov1 component? Would we print the devfile.yaml for all components with that component name?
What would the output of odo catalog describe component be? Do you only want to output certain information or the entire devfile.yaml? Easiest way would be to pretty print the yaml rather than having a table, to ensure we print “all” info (to handle schema spec changes).
Do we want to show project info in odo catalog list components or not? @johnmcollier would like to show a subset of project info under odo catalog list components <component> whilst @kadel  would like to add a “Starter Projects” column to the outputted “Odo devfile components” table, whilst some others think we shouldn’t add anything. (Whilst I would like a “starter projects” column here, we would have to download the devfile.yaml for each devfile component to get this information, which would be very expensive.)
There’s also the case that @GeekArthur pointed out here, where we have multiple components with the same name from different registries. Do we want a “registry” column to be added to odo catalog list components to handle this? And how would we handle it in the new odo catalog describe component command. Would we prioritise in this case or print all yamls?

If I’ve missed anything out please bring it up! These were the decision points I came up with from trying to solve this issue

The main agreement we’ve come to is to move the main solution to under odo catalog describe component <component>, which sounds good to me!
The following is what I would like to have agreed on:
1. Is this command going to accept both devfile and odov1 components? If so, how do we handle the case that a component exists as both a devfile and odov1 component? Would we print the `devfile.yaml` for all components with that component name?

2. What would the output of `odo catalog describe component` be? Do you only want to output certain information or the entire `devfile.yaml`? Easiest way would be to pretty print the yaml rather than having a table, to ensure we print “all” info (to handle schema spec changes).

3. Do we want to show project info in `odo catalog list components` or not? @johnmcollier would like to show a subset of project info under `odo catalog list components <component>` whilst @kadel  would like to add a “Starter Projects” column to the outputted “Odo devfile components” table, whilst some others think we shouldn’t add anything. (Whilst I would like a “starter projects” column here, we would have to download the `devfile.yaml` for each devfile component to get this information, which would be very expensive.)

4. There’s also the case that @GeekArthur pointed out [here](https://github.com/openshift/odo/pull/3057#issuecomment-622020128), where we have multiple components with the same name from different registries. Do we want a “registry” column to be added to `odo catalog list components` to handle this? And how would we handle it in the new `odo catalog describe component` command. Would we prioritise in this case or print all yamls?

If I’ve missed anything out please bring it up! These were the decision points I came up with from trying to solve this issue

For point 4, the new "registry" column will be added by this PR https://github.com/openshift/odo/pull/2940 which implements dynamic registry support.

Is this command going to accept both devfile and odov1 components? If so, how do we handle the case that a component exists as both a devfile and odov1 component? Would we print the devfile.yaml for all components with that component name?

For old-style s2i based components, I would just print the component name for now. At some point, we are going to depreciate and I wouldn't put too much effort into it. But for compatibility sakes, I would make the command work for both. Similar to what we do for odo catalog list components
If there is a name collision I would just show both (or all available options). Something like this
odo catalog describe component java

Devfile components:

WARNING: There are multiple components named "java" indifferent multiple devfile registries.

* Registry: my-registry
<devfile.yaml content>

* Registry: other-registry
<devfile.yaml content>

s2i based components:
- java

Please note that in all odo commands we want to have an option to show output in machine-readable format (json), to make it easier to build additional tools on top of odo (like ide plugins).
odo catalog describe component java -o json should output the same information but in nice JSON structured format

2. What would the output of odo catalog describe component be? Do you only want to output certain information or the entire devfile.yaml? Easiest way would be to pretty print the yaml rather than having a table, to ensure we print “all” info (to handle schema spec changes).

yes, pretty-printing devfile.yaml is ok

3. Do we want to show project info in odo catalog list components or not? @johnmcollier would like to show a subset of project info under odo catalog list components <component> whilst @kadel  would like to add a “Starter Projects” column to the outputted “Odo devfile components” table, whilst some others think we shouldn’t add anything. (Whilst I would like a “starter projects” column here, we would have to download the devfile.yaml for each devfile component to get this information, which would be very expensive.)

I don't like odo catalog list components <component>, I think that it should be handled by odo catalog describe component <component>.
Regarding showing it in odo catalog list components. Currently, we already download all devfile from the registry, so showing the list of the project wouldn't add any overhead. But downloading devfiles is only a temporary solution until there is a proper devfile v2 spec and devfile registry implementation.
We could make sure that a new devfile registry design has some kind of API that would allow listing available projects for devfile without downloading it first.
We currently don't know how the registry will look like, so I would defer showing the starter projects in odo catalog list components for now, unless you feel that we need it (then I would go with parsing it from devfiles as a temporary solution)

Do we want to show project info in odo catalog list components or not? @johnmcollier would like to show a subset of project info under odo catalog list components  whilst @kadel would like to add a “Starter Projects” column to the outputted “Odo devfile components” table, whilst some others think we shouldn’t add anything. (Whilst I would like a “starter projects” column here, we would have to download the devfile.yaml for each devfile component to get this information, which would be very expensive.)

I think with odo catalog list components showing project info for the devfiles and odo catalog describe component <component> showing the devfile's content, we don't need odo catalog list components <component>. And agreed with @kadel in that it's a bit of an anti-pattern.

@GeekArthur thank you for pointing that out to me!
@kadel @johnmcollier thank you for the very comprehensive reply! I think that covers all my questions (at least for now!), so I'll work on getting the command to follow your proposed format

authors: GeekArthur cdrage johnmcollier kadel skoh7645

repo: openshift/odo


text:
chore: added link to Slack

For chat-addicted people

/assign @mik-dass

The CoreOS slack is for RedHat accounts only. Closing


text:
Removes code related to s2i storage

What type of PR is this?
/kind cleanup
What does this PR do / why we need it:
It removes code related to the s2i storage.
Which issue(s) this PR fixes:
Fixes part of #4932
PR acceptance criteria:

[ ] Unit test

[ ] Integration test

[ ] Documentation

[ ] I have read the test guidelines

How to test changes / Special notes to the reviewer:
Tests should pass.

/approve

/retest-required

@mik-dass Can you resolve the merge conflict?

@mik-dass Just to make sure I got it right, you removed the unit tests from utils_test.go because Philippe's #4948 got merged, right?

/lgtm

@mik-dass Just to make sure I got it right, you removed the unit tests from utils_test.go because Philippe's #4948 got merged, right?

Yes

authors: feloy mik-dass valaparthvi


text:
README file needs update

Uninstall command is wrong. Should be:
ansible-playbook -i inventory/aws/hosts -e 'region=us-east-1 stack_name=openshift-infra ci=false' playbooks/teardown.yaml
Currently has "playook"

Good find!

repo: openshift/openshift-ansible-contrib


text:
add node fix due to playbook movement

This is a fix due to some playbooks being moved. This will serve as the generic starting point for adding nodes within AWS and with other providers

@cooktheryan yamllint errors from travis.

Cool ill fix. Attempted to write code too late in the day

@cooktheryan late in the day is the best time for me, fewer distractions :)

Learned a lesson in committing too close

authors: cooktheryan detiber


text:
[WIP] UPSTREAM: : fix xplat compile for of-tools image; hide utest f…

…iles from git; use rhel8 art builder image

/retest

repo: openshift/operator-framework-olm


text:
Inconsistencies on nightly releases

This change will visualize the "nightlies" that are not in a shippable state.
For many reasons, nightlies are created not in a way that meets different stakeholder requirements.
The "inconsistency" annotations are embedded in the metadata (imagestreamtag):
 - annotations:
      release.openshift.io/inconsistency: '["Found outdated RPM (python3-cachetools-3.1.0-3.el8ost)
        installed in ironic-container-v4.11.0-202202152258.p0.gfcae3c5.assembly.stream
        when python3-cachetools-3.1.0-2.el8ost was available"]'
    from:
      kind: DockerImage
      name: quay.io/openshift-release-dev/ocp-v4.0-art-dev:sha256-9c6bcaa6e10d4068c2f055a522e37a89e05921e1e9044f1c124ceed8bd2ad86f
    generation: 381
    importPolicy: {}
    name: ironic
    referencePolicy:
      type: Source

These inconsistencies will be displayed on the release controller page:

/hold

repo: openshift/release-controller


text:
ensure the PreserveOriginal import mode policy image tag synchronization

e.x
    - name: 4.11.0-multi
      annotations:
        release.openshift.io/creationTimestamp: '2022-08-09T00:01:01Z'
        release.openshift.io/hash: 4.11.0-multi-102
        release.openshift.io/name: 4-stable-multi
        release.openshift.io/phase: Accepted
        release.openshift.io/rewrite: 'false'
        release.openshift.io/source: ocp-multi/release-multi
      from:
        kind: DockerImage
        name: 'quay.io/openshift-release-dev/ocp-release:4.11.0-multi'
      generation: 102
      importPolicy:
        importMode: Legacy
      referencePolicy:
        type: Source

The quay.io/openshift-release-dev/ocp-release:4.11.0-multi image holds a manifest list. After 4.13, Openshift imagestreams are eligible to preserve the manifest list by using the PreserveOriginal import mode.
/cc @bear-redhat
Signed-off-by: Nikolaos Moraitis nmoraiti@redhat.com

/assign @AlexNPavel

/retest


text:
Enable passing a kubeconfig file to apiserver

Upstream cherry-pick of https://github.com/kubernetes-incubator/service-catalog/pull/2158
Also contains https://github.com/kubernetes-incubator/service-catalog/pull/2136 which we miss but on which 2158 is based on.
@jpeeler @jim-minter @pweil-

/cherrypick release-3.10

@kargakis generally we try to avoid carrying patches, but since 3.10 is frozen it would have to be done there. Do you need this for just 3.10 only? If so, I'll rebase this branch next release and apply the desired patches to the 3.10 branch (now).

Yeah, ideally we want this in 3.10 even though we are most likely going to
GA in 3.11.
On Thu, Jul 5, 2018, 17:18 Jeff Peeler notifications@github.com wrote:

@kargakis https://github.com/kargakis generally we try to avoid
carrying patches, but since 3.10 is frozen it would have to be done there.
Do you need this for just 3.10 only? If so, I'll rebase this branch next
release and apply the desired patches to the 3.10 branch (now).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/service-catalog/pull/11#issuecomment-402757976,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADuFfxBVJHPonRzVsDgHeFPEtWrWprJjks5uDi44gaJpZM4VCE7c
.

@jpeeler we need these patches to be able to deploy the service catalog as part of the "openshift on azure" project.  We need these changes to go into origin and enterprise 3.10 in the next release possible, as well as origin master as soon as possible.  Without them it won't deploy at all.

/lgtm
I was asking so that I see if I could rebase master with a new upstream release next week. But I'll go ahead and merge and sort the patch rebasing out later.

authors: jim-minter jpeeler kargakis

repo: openshift/service-catalog


text:
Hash code in download link

I'm try looking a way to download dynamic the last version of Linux zip file, but have a hash on the link. Is possible remove this from the links? Thanks.
S2I_VERSION="$(curl -ks https://api.github.com/repos/openshift/source-to-image/releases/latest | grep tag_name | cut -d '"' -f 4)"; \
  curl -L "https://github.com/openshift/source-to-image/releases/download/${S2I_VERSION}/source-to-image-${S2I_VERSION}-????????-linux-amd64.tar.gz" | \
    sudo tar -xzC '/usr/local/bin' --strip-components 1 && \
      sudo chmod +x '/usr/local/bin/s2i'

It'd be nice!!!

Agree - the hash is not necessary, especially since GitHub releases are always tied to a tag.

/remove-lifecycle stale

/reopen

authors: adambkaplan brunowego erdii gustavodescs

repo: openshift/source-to-image


text:
Remove vtk

Fixes issue #975
Brief summary of changes
Remove dependencies on vtk (already broken functionality since libraries aren't distributed any more) and remove vtk module altogether to cleanup code base and simplify code maintenance.
Testing I've completed
Built GUI and launched it.
CHANGELOG.md (choose one)

no need to update because cleanup

Can anybody try this PR branch to see if something broke? This PR removes all traces of vtk, remove dependency and makes the distribution smaller at the same time. Also makes it easier to search the code. Whoever has cycles to donate would be great: @chrisdembia @carmichaelong @nickbianco @jimmyDunne Thank you.

I ran the three tutorial GUI scripts. I also ran the standard pipeline of Scale --> IK --> RRA --> CMC using gait2392 and the accompanying files. These all seemed to work fine.
I ran into one problem where I (incorrectly) tried to run IK on a model without markers then tried to open the plotter. I tested this on the master branch and the problem was also there, so the issue is not with this branch (#1004 details this).

Thanks much @carmichaelong 🥇

authors: aymanhab carmichaelong

repo: opensim-org/opensim-gui


text:
Safe/unsafe configurable proxy calls

Document #30.
Implemented:

proxy.safe.<methodcall> and proxy.unsafe.<methodcall> for per-call configuration.
Proxy(..., safe=True/False) initialization parameter.
Global OSBRAIN_DEFAULT_SAFE environment variable.

Closed in favor of #57.

repo: opensistemas-hub/osbrain


text:
推荐嘉宾 - 水歌（个人成长系列）

GitHub: TechQuery
所在公司：idea2app
岗位：全栈工程师
推荐理由：开源社区深度贡献者，开源理念积极践行者。
推荐人：stevending1st

感谢推荐，如果 @TechQuery 感觉没问题的话，我们可以拉进微信群里（所有嘉宾的群）

感谢推荐，很高兴来分享开源心得，哈哈

已经邀请 @TechQuery 加入《开源面对面》的 GitHub Org

已经发送了 pre-talk 的会议邀请， @stevending1st 你要是有兴趣的话，可以一起来聊，下面是日历和 Google Group 的链接

https://calendar.google.com/calendar/u/0?cid=NnYxNzcxc2w4Y2owZG9sMWxuYjRwbGJyMm9AZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ
https://groups.google.com/g/osf2f

下面是 pre-talk 的录屏资料
https://www.youtube.com/watch?v=z30ZIEYtkUk

已完成

authors: LinuxSuRen TechQuery stevending1st


text:
Tutorial training an AI badgeholder using Vertex AI

What is it?
https://cloud.google.com/vertex-ai/docs/tutorials
Data is in BigQuery, so this should be easy.
Tutorial should probably go

from training data (key/timeseries metrics by project)
to funding allocations by project

Got some things started here: https://github.com/opensource-observer/insights/tree/main/community/ai_agents

repo: opensource-observer/oso


text:
🛑 Weblate is down

In 42f104b, Weblate (https://translate.opensourcepos.org) was down:

HTTP code: 500
Response time: 1091 ms

Resolved: Weblate is back up in f72d5be after 41 minutes.

repo: opensourcepos/upptime

license: MIT


text:
[Remote Sync] Fix resources sync to the server.

Technical implementation specific

[ ] Sync all the data found on the Local SQLite DB to the FHIR server when sync is automatically or manually triggered.
[ ] Confirm that QRs are correctly tagged based on this issue.

NB: - This issue does not need to handle sync down to the server since the Remote Sync issues defined here will adress that.
Originally posted by @dubdabasoduba in https://github.com/opensrp/fhircore/issues/1460#issuecomment-1275404837

@FikriMilano Please add the LOE on this

@FikriMilano not really, this issue is to refactor sync to the server to sync all resources. We do not want to filter data synced to the server. We only want to filter data synced from then server

Ah ok, you mean like, syncing the latest update from local DB to the server

Confirm that QRs are correctly tagged based on this issue.
I think this point can be solved by the resource tagging. Whatever resource, it will be tagged when inserted to the local DB.
If we do this point, then it's just repeating the same thing.
@dubdabasoduba

Confirm that QRs are correctly tagged based on this issue.
I think this point can be solved by the resource tagging. Whatever resource, it will be tagged when inserted to the local DB.
If we do this point, then it's just repeating the same thing.
@dubdabasoduba

@agreed! That point is more of a confirmation that the tagging is happening correctly

authors: FikriMilano dubdabasoduba

repo: opensrp/fhircore


text:
Runchecker failure with no-asm: tests hang

Output is:
05-test_cast.t ..................... ok
05-test_cmac.t ..................... ok
05-test_des.t ...................... ok
05-test_hmac.t ..................... ok
05-test_idea.t ..................... ok
05-test_rand.t ..................... ok
05-test_rc2.t ...................... ok
05-test_rc4.t ...................... ok
05-test_rc5.t ...................... skipped: rc5 is not supported by this OpenSSL build
06-test_algorithmid.t .............. ok
06-test_rdrand_sanity.t ............ ok
make[1]: *** [Makefile:3274: _tests] Terminated
make[1]: *** wait: No child processes.  Stop.
make[1]: *** Waiting for unfinished jobs....
make[1]: *** wait: No child processes.  Stop.
make: *** [Makefile:3271: tests] Terminated

The test following rdrand is the big number test which is slow.

The ending point isn't deterministic.  For example, these have been seen too:

25-test_x509.t ..................... ok
15-test_ecdsa.t .................... ok

So apparently it is just reallyreally slow. Still can be problem with seeding the rng if the runchecker OS is old and does not support getrandom.

I still can't login to the run-checker machine to figure this out.  @levitte.

Hmmm....
This test runs on the run-checker host in just over 25 minutes.
The two longest running tests are:

04-test_encoder_decoder_legacy.t ... ok  1191522 ms
20-test_dhparam_check.t ............ ok   700571 ms
which amount to 32 minutes (4 processor test, so it's possible)

This could depend heavily on system load at the time.

This is probably addressed by openssl/tools#86

Optimised, the build time doesn't change a lot.  The test time is circa 6 minutes which is much better.

Should be fixed, closing.

repo: openssl/openssl


text:
test/namemap_internal_test.c: use "cookie" instead of "foo"...

... in test_namemap()
Because tests may sometimes run in random order (subject of the
environment variable OPENSSL_TEST_RAND_ORDER being defined), and we're
dealing with the global namemap, each test must use names that are
globally unique for that test.  Unfortunately, we used "foo" in two of
them, which might lead to surprising results.
Fixes #10401

So this just kicks the can down the road, right?  Perhaps a test-function that does the equivalent of $(basename $(mktemp)) to generate a random name when needed?

Yeah, an individual OPENSSL_CTX might be the answer.

Merged.
97ba39547d4c79b60131475a1512cc7d7e8952b2 test/namemap_internal_test.c: use "cookie" instead of "foo"...

authors: levitte richsalz


text:
Refactor BN_R_NO_INVERSE logic in internal functions

Closes #12129
NOTE for reviewers: I strongly suggest reviewing commit by commit, as one of the separate commits is reordering function definitions, which makes it look like a big set of changes even when the changes are quite minimal.
As described in https://github.com/openssl/openssl/issues/12129 the readability of the internal functions providing the two alternative implementations for BN_mod_inverse() is a bit lacking.
Both these functions are now completely internal, so we have the flexibility needed to slightly improve readability and remove unnecessary NULL checks.
The main changes here are:

rename BN_mod_inverse_no_branch() as bn_mod_inverse_no_branch(): this function is static so it is not even visible within the rest of libcrypto. By convention upcase prefixes are reserved for public functions.
remove if (pnoinv == NULL) checks in int_bn_mod_inverse(): this function is internal to the BN module and we can guarantee that all callers pass non-NULL arguments.
bn_mod_inverse_no_branch() takes an extra int *pnoinv argument, so that it can signal if no inverse exists for the given inputs: in this way the caller is in charge of raising BN_R_NO_INVERSE as it is the case for the non-consttime implementation of int_bn_mod_inverse().

I kept the following 3 commits separated as they might require further discussion.
Create a temporary BN_CTX if needed in the public function

commit 1824fadc9dc2e1bcaf8cc217c445efbf5134d476
- `BN_mod_inverse()` is a public function and must guarantee that the
  internal functions providing the actual implementation receive valid
  arguments. If the caller passes a NULL `BN_CTX` we create a temporary
  one for internal use.

Reorder function definitions to avid forward declarations

commit f22d828c5c708b5276c756a4f0523479d1bd85a9
@ This commit only reorders the existing functions, no other edit is
@ included.
@
- reorder function definitions in `crypto/bn/bn_gcd.c` to avoid forward
  declaration of `static` functions (in preparation for inlining)

Inline bn_mod_inverse_no_branch()

commit 7e7cb711b5585827e1722736c85aeffef5fa6b23
- inline `bn_mod_inverse_no_branch()`.

Backport to 1.1.1
Most of the changes here should apply trivially also to 1.1.1, but I expect some required changes related to the internal header and to the error function codes.
I will open a separate PR once this is approved dedicated to 1.1.1 and will update this comment.

This pull request is ready to merge

Merged to master as 5d8b3a3ef2941b8822523742a0408ca6896aa65d
Thanks!

See #12169 for a dedicated backport PR.

authors: openssl-machine romen


text:
Fix a memory leak if using a custom X509_LOOKUP_METHOD

I also add a test for this leak - the test will fail prior to the patch in this PR being applied if OpenSSL is configured with enable-crypto-mdebug.
Fixes #8707

A rebase, maybe ?

If custom method is implemented properly X.509 lookup methods does not leak memory!
Proposed modification in by_dir.c will introduce memory leak!
It is not backward compatible and so ti is not acceptable for release branches.

If custom method is implemented properly X.509 lookup methods does not leak memory!

How would you suggest that it should be implemented properly? AFAICT the only way to set an X509 or X509_CRL object into an X509_OBJECT is by using X509_OBJECT_set1_X509() or X509_OBJECT_set1_X509_CRL(). How can it be done without increasing the ref count?

It is not backward compatible and so ti is not acceptable for release branches.

This is usually true for all bug fixes. If you happen to rely on the bug then fixing the bug isn't backwards compatible. That doesn't mean it isn't an acceptable change.

The fact that by_dir.c doesn't increase the refcount properly is a bug and has always been.  This has been a pain for anyone writing custom X509_LOOKUP methods with objects where the refcount is automagically set to 1 on creation and no way to decrease it without freeing the object... I do recall having more than one conversation on the matter.

Here's an example of what the bug has lead to: 5306d1094f3e0d4304614ff8e99db90523f8d980

mattcaswell wrote:

If custom method is implemented properly X.509 lookup methods does not leak memory!

How would you suggest that it should be implemented properly? AFAICT the only way to set an X509 or X509_CRL object into an X509_OBJECT is by using X509_OBJECT_set1_X509() or X509_OBJECT_set1_X509_CRL(). How can it be done without increasing the ref count?

One again let see #6970 =>  #8442 .
I already point in original issue that "sample X.509 code" is not
correct. It leaks memory in 1.0.2* and may be crash.

It is not backward compatible and so ti is not acceptable for release branches.

This is usually true for all bug fixes. If you happen to rely on the bug then fixing the bug isn't backwards compatible. That doesn't mean it isn't an acceptable change.

No. Not for this defect.

It is not acceptable change as currently in PKIX-SSH ldap X.509 method
does not leak memory .Remark: It has memory leak but it is due to ldap
and sasl initialization not due to OpenSSL or PKIS-SSH code.
How third party method will know to increment data for X509_OBJECT? If
is run with 1.0.2 or 1.1.1b. If is compiled with 1.1.1b but run with
1.1.1c what will be result - crash!
I start to think that is more save to backport  #8442 instead to correct
current situation.
Regards,
Roumen

One again let see #6970 =>  #8442 .

The "solution" implemented by #8442 is not an approach that non-OpenSSL implementations of an X509_LOOKUP_METHOD can use. It dives under the covers of the X509 and X509_CRL objects and directly modifies the reference counts. That is a horrific hack and should be removed from #8442 IMO. It is only necessary because of this bug.
#6970 just seems to describe the same bug in point 2.
So, so far, I still don't understand how you think this is "supposed" to be done to not leak memory in a custom lookup method.

That is a horrific hack and should be removed from #8442 IMO. It is only necessary because of this bug.

Agreed on all points, and if this PR does make it in, that horrific hack is gone.

Richard Levitte wrote:

The fact that by_dir.c doesn't increase the refcount properly is a bug and has always been.
Hmm,
It is increased indirectly:

X509_load_cert_file ->  X509_STORE_add_cert -> x509_store_add --> [*]
X509_load_crl_file -> X509_STORE_add_crl -> x509_store_add -> [*]

[*] X509_OBJECT_up_ref_count  with X509_OBJECT_free under condition.
This has been a pain for anyone writing custom X509_LOOKUP methods with
objects where the refcount is automagically set to 1 on creation and no
way to decrease it without freeing the object... I do recall having more
than one conversation on the matter.

This is historic .

Richard Levitte wrote:

That is a horrific hack and should be removed from #8442 IMO. It is only necessary because of this bug.

Agreed on all points, and if this PR does make it in, that horrific hack is gone.

With implementation on X.509 lookup based on Store API  "X.509  store
cache" will become  real "opaque" . External implementation will not
care how to manage "X.509  cache" entries. So you could rewrite X.509
lookup to best development practice.
Roumen

This is historic .

In 1.0.2 this was a badly designed interface, but you could work around it by diving directly into the structures. In 1.1.0 that option is no longer possible and you must only use the provided API functions. At that point the badly designed interface became a bug because there is no way to do it correctly. Yes that means things will work slightly differently between 1.0.2 and 1.1.0+. But that's ok, because you have to do things differently between 1.0.2 and 1.1.0 anyway.
This is a bug, pure and simple and should be fixed.

Proposed modification in by_dir.c will introduce memory leak!

No - it doesn't. The by_dir.c modification ups the ref count on the contained objects. The modification in x509_lu.c then means that we don't also up the ref count in X509_STORE_CTX_get_by_subject(). Therefore the number of references remain the same before and after these modifications for the built-in  X509_LOOKUP_METHOD.

Richard Levitte wrote:

The fact that by_dir.c doesn't increase the refcount properly is a bug and has always been.
Hmm,
It is increased indirectly:

X509_load_cert_file ->  X509_STORE_add_cert -> x509_store_add --> [*]
X509_load_crl_file -> X509_STORE_add_crl -> x509_store_add -> [*]

[*] X509_OBJECT_up_ref_count  with X509_OBJECT_free under condition.

But X509_load_cert_file and X509_load_crl_file have nothing to do with y_dirr.c's get_cert_by_subject...  Rather, using X509_load_cert_fileandX509_load_crl_fileis a way to ensure thatget_cert_by_subjectwon't be called, at least as a consequence of callingX509_STORE_CTX_get_by_subject`, because the objects you have loaded are already in the X509 STORE.  That pattern of use should be unaffected by this PR!

This has been a pain for anyone writing custom X509_LOOKUP methods with objects where the refcount is automagically set to 1 on creation and no way to decrease it without freeing the object... I do recall having more than one conversation on the matter.

This is historic .

How so?  The situation is unchanged, so the challenge is still the same.

@petrovr, I just had a closer look a the ldap store in #6970, and I have to hand it to you, that's a fairly elegant hack to get around the problem that we're seeing here.
Matt, FYI, what he does is to put everything retrieves from LDAP into the X509 STORE on the fly, then just retrieve one of them and return that as the get_by_subject X509_OBJECT return.  This works, because the resulting "extra" refcount is for the X509 STORE, and will thereby be cleared as soon as that store is freed.  by_dir.c essentially does the same, but has its own internal cache (which gets cleared by its free_dir function) instead of the X509 STORE.
And then comes by_store.c in #8442, which does not cache in any way.  KABOOM

Richard Levitte wrote:

Richard Levitte wrote:

The fact that by_dir.c doesn't increase the refcount properly is a bug and has always been.
Hmm,
It is increased indirectly:

X509_load_cert_file ->  X509_STORE_add_cert -> x509_store_add --> [*]
X509_load_crl_file -> X509_STORE_add_crl -> x509_store_add -> [*]

[*] X509_OBJECT_up_ref_count  with X509_OBJECT_free under condition.

But X509_load_cert_file and X509_load_crl_file have nothing to do with y_dirr.c's get_cert_by_subject...  Rather, using X509_load_cert_fileandX509_load_crl_fileis a way to ensure thatget_cert_by_subjectwon't be called, at least as a consequence of callingX509_STORE_CTX_get_by_subject`, because the objects you have loaded are already in the X509 STORE.  That pattern of use should be unaffected by this PR!

Let start with empty 'X.509 cache'  and application try to build X.509
chain.
Cache is empty so get_cert_by_subject is called and it push to "cache"
x509/crls found so indirect increase.
Or may be my understanding is not correct.
Regards,
Roumen

Cache is empty so get_cert_by_subject is called and it push to "cache" x509/crls found so indirect increase. Or may be my understanding is not correct.

The thing with that line of thinking is that it assumes that get_by_subject and similar always cache objects.  However, that assumptions turns out to be flawed.  I've read through the docs many times now, and I still haven't found anything that says that get_by_subject et al must cache the stuff they return.

Richard Levitte wrote:

Cache is empty so get_cert_by_subject is called and it push to "cache" x509/crls found so indirect increase. Or may be my understanding is not correct.

The thing with that line of thinking is that it assumes that get_by_subject and similar always cache objects.  However, that assumptions turns out to be flawed.  I've read through the docs many times now, and I still haven't found anything that says that get_by_subject et al must cache the stuff they return.

No! There is some misunderstanding. Neither by_dir or by_ldap cache
objects. by_ldap is about 15 year old implementation based on by_dir.
All the times algorithm was one and the same.
(1) for each object found add object to x.509 cache.
(2) retrieve object from "x.509 cache" and copy to 'return' value.
Let skip that in (2) there is difference in lock. Also for CRL by_dir
loop (1) perform some 'cache'(3). This cache is not for object
found(loaded)!
Cache in by_dir loop is for 'counter' of last file name. CRL suffix is
r00, r01, r02 and etc.
Usually certificates and not changed(replaces) but crl are issued quite
often. If I remember well in the pass once loaded into 'x.509 cache'
openssl never try to 'reload' neither cert nor crl. This was changed
long time ago for crl. I did not look yet into code but if I remember
well for crl 'by_subject' is always called. To prevent loading again of
all crl  'by_dir' remember counter of last loaded crl.
Sample:
Let X.509 name hash is 1234567.
Let on file system we have crl files 1234567.r00, 1234567.r01 and
1234567.r02.
On first load for crl  by_dir will load all those files and will
remember next "counter', i.e. 3.
On next call with same name it will try to load crl from files
1234567.r03, 1234567.r04 and etc. assuming than newly issued crl are
stored into files with next index(counter).
There is no cache neither of X.509 nor CRL.
There is no way in database like queries(by_ldap/by_store) to implement
caching similar to (3) - by_dir "counter" cache. We should expect that
external system returns items in arbitrary order. But this part is out
of scope to this issue. Let discuss in future.
Regards,
Roumen
P.S. May be I will not participate into discussion next few day.

Richard Levitte wrote:

Cache is empty so get_cert_by_subject is called and it push to "cache" x509/crls found so indirect increase. Or may be my understanding is not correct.

The thing with that line of thinking is that it assumes that get_by_subject and similar always cache objects.  However, that assumptions turns out to be flawed.  I've read through the docs many times now, and I still haven't found anything that says that get_by_subject et al must cache the stuff they return.

No! There is some misunderstanding. Neither by_dir or by_ldap cache
objects. by_ldap is about 15 year old implementation based on by_dir.
All the times algorithm was one and the same.
(1) for each object found add object to x.509 cache.
(2) retrieve object from "x.509 cache" and copy to 'return' value.

So uhhhhmmmm, here you're saying "add object to x.509 cache", and yet you're trying to say that by_ldap doesn't cache objects?

Let skip that in (2) there is difference in lock. Also for CRL by_dir
loop (1) perform some 'cache'(3). This cache is not for object
found(loaded)!
Cache in by_dir loop is for 'counter' of last file name. CRL suffix is
r00, r01, r02 and etc.

Regarding by_dir, I didn't remember things quite right.  It does use X509_load_cert_file and X509_load_crl_file, which do store the found objects in the X509 STORE.  Exactly like what you're doing in by_ldap.

Usually certificates and not changed(replaces) but crl are issued quite
often. If I remember well in the pass once loaded into 'x.509 cache'
openssl never try to 'reload' neither cert nor crl. This was changed
long time ago for crl. I did not look yet into code but if I remember
well for crl 'by_subject' is always called. To prevent loading again of
all crl  'by_dir' remember counter of last loaded crl.

This code seems to be relevant:
https://github.com/openssl/openssl/blob/ccf453610f48fe88968f0cfc63784b503eae33a0/crypto/x509/by_dir.c#L335-L371

Sample:
Let X.509 name hash is 1234567.
Let on file system we have crl files 1234567.r00, 1234567.r01 and
1234567.r02.
On first load for crl  by_dir will load all those files and will
remember next "counter', i.e. 3.
On next call with same name it will try to load crl from files
1234567.r03, 1234567.r04 and etc. assuming than newly issued crl are
stored into files with next index(counter).
There is no cache neither of X.509 nor CRL.

I think you don't quite understand what I'm talking about.  These lines are doing exactly what I'm talking about, i.e. storing away all found object...  that's what I call "caching" in this context:
https://gitlab.com/secsh/pkixssh/blob/87c242950cbd86ec5761939ac6b54452bfb56ae5/x509_by_ldap.c#L913
https://gitlab.com/secsh/pkixssh/blob/87c242950cbd86ec5761939ac6b54452bfb56ae5/x509_by_ldap.c#L921

Richard Levitte wrote:
[SNIP]

This code seems to be relevant:
https://github.com/openssl/openssl/blob/ccf453610f48fe88968f0cfc63784b503eae33a0/crypto/x509/by_dir.c#L335-L371

This is update on 'counter' cache for crl -  after loop that push to
'x5.09 store'  by_dir method store  next "counter' .." for future use.
On next invocation with same X.509 name crl-load will start for this
"crl counter".
Regards,
Roumen
P.S.:
Correction of my previous post : crl suffix  is .r0 , .r1, r2 and etc.
by example 1234567.r0, 1234567.r1 and  1234567.r2.

Please wait with commit.

I looked through the documentation again, and it is true what @petrovr says, that by_dir.c behaves exactly like he says (i.e. get_cert_by_subject doesn't just fetch one certificate, but all matching ones and puts them in the X509 STORE), and this is actually documented in X509_LOOKUP_hash_dir.pod.  At the same time, [X509_LOOKUP_meth_new.pod}(https://github.com/openssl/openssl/blob/master/doc/man3/X509_LOOKUP_meth_new.pod) doesn't mention any such requirements, and in fact says that the returned object should be assigned using X509_OBJECT_set1_X509() or X509_OBJECT_set1_X509_CRL(), depending on the type (those two functions do increment the object's reference counter, exactly like this PR does fir by_dir).
So there's a very clear disconnect between the documentation and what's actually expected, or at least implemented.
Looking through the history of by_dir.c, its basic implementation hasn't changed much since the dawn of OpenSSL, i.e. get_cert_by_subject has always loaded all matching objects into the X509 STORE.  This does suggest that the documentation in X509_LOOKUP_meth_new is incorrect.

As of https://github.com/openssl/openssl/issues/8707#issuecomment-483281460, I've changed my mind and oppose this PR.
-1
(I will also have to modify #8442 to match, or not push for that PR at all...)

Ok, I'll close this. However if we're going to live with this then it suggests we should have an X509_OBJECT_set0_X509 and an X509_OBJECT_set0_X509_CRL function (as opposed to just the set1 functions we have now). Otherwise it just way too hard to get the ref counting right. And we need much better documentation for the get_by_subject stuff that explains the ref counting requirements. @levitte will you take that on, or am I doing it?

I've already started on the documentation bit.

Great - I'll leave it with you.

authors: levitte mattcaswell petrovr


text:
BIO_f_zlib(): add pending / wpending controls

This filter was lacking support to check if there's any more pending
data, which may result in the last zlib block being lost.
Fixes #9866

@petrovr The #9877 was merged in both master and 1.1.1 stable so that fixes the #9866 issue.

Why is #9877 discussed here?


text:
Remove user visible traces of the "engine" OSSL_PARAM.

The OSSL_PARAM "engine" is present only to allow backwards compatibility for engines until they are wrapped in to a engine-provider.  Because this means the parameter will be relatively short lived, it is better if it was never exposed publicly.

[x] documentation is added or updated
[ ] tests are added or updated

I think I'd like to hear at least what @mattcaswell has to say before this gets merged.

I think I'd like to hear at least what @mattcaswell has to say before this gets merged.

I'm kind of undecided. On the one hand I'd rather this sort of thing is not exposed. On the other hand its not yet clear to me exactly how the engine provider is going to work, and I kind of think that we might have to expose these parameters in order for that to work. It would be really nice if we actually had that in place before doing this kind of work.

Thanks @mattcaswell.  It isn't clear to be that engine support for the algorithms used by KDFs and MACs will need this kind of support.
I can think of a way where it would be possible to set something up but it requires going via the PKEY wrapper not the KDF/PRF/MAC apis.
I'll merge this, we can put something back once we know how engines are to be dealt with -- odds are it won't be identical to this.

Pushed thanks guys.

authors: levitte mattcaswell paulidale


text:
Always name database "cinder"

Current code names the database for cinder based on the Cinder CR name, and this is ok if the instance is called cinder, but if the name is something like "mycloud-cinder" then the database creation doesn't work as it fails with:
  ERROR 1064 (42000) at line 1: You have an error in your SQL syntax;
  check the manual that corresponds to your MariaDB server version for
  the right syntax to use near '-cinder' at line 1

We have a constant called DatabaseName that contains the name that we should be using for the database creation, and this patch uses it.
Jira: OSPRH-7396

/lgtm

authors: Akrog fmount


text:
Add TLS support for multiple service endpoints

Some services create multiple endpoints; therefore, they require a specific TLS secret for each endpoint. The TypedSecretName field is added to the Service struct, which allows to specify unique TLS secret for each endpoint.

I've implemented the changes to the structs and their fields as suggested by @stuggi ^^ Following these updates the functions and tests were revised within the tls package, so it should now be in place for operator testing.

Correcting right now, so the PR consists only of corresponding commits...

With changes TLS support is moved here ---> https://github.com/openstack-k8s-operators/lib-common/pull/428


text:
Add sample for network isolation using ceph for glance and cinder

Adds a sample to deploy the ctplane using isolated networks and ceph. The ceph configs get attached from ceph-conf-files secret. The FSID in the cinder custome config needs to be replaces with the one from the ceph cluster:
rbd_secret_uuid=_FSID_

One nit about a comment, not the PR itself.
  sed -i "s/FSID/${FSID}/" $OPENSTACK_CTLPLANE

vs
  sed -i "s/_FSID_/${FSID}/" $OPENSTACK_CTLPLANE

You want the second so you don't end up with a line like this:
            rbd_secret_uuid=_b07ec152-274d-4c8c-b939-32362336ee9f_

authors: fultonj stuggi


text:
"patch" crashes when an existing .patch file is not from git-format-patch

Seeing the following crash with master (4dd0bef9b99a409ff59055a48526b31e778019ef)
$ rdopkg patch
## get_package_env
## ensure_patches_branch
## get_patches_branch
git update-ref refs/heads/ceph-2-rhel-patches refs/remotes/patches/ceph-2-rhel-patches
## check_new_patches
Traceback (most recent call last):
  File "/home/kdreyer/dev/rdopkg/venv/bin/rdopkg", line 9, in <module>
    load_entry_point('rdopkg', 'console_scripts', 'rdopkg')()
  File "/home/kdreyer/dev/rdopkg/rdopkg/cli.py", line 46, in main
    sys.exit(rdopkg(*cargs))
  File "/home/kdreyer/dev/rdopkg/rdopkg/cli.py", line 38, in rdopkg
    return shell.run(runner, cargs=cargs, version=VERSION)
  File "/home/kdreyer/dev/rdopkg/rdopkg/shell.py", line 84, in run
    action_runner.engage()
  File "/home/kdreyer/dev/rdopkg/rdopkg/core.py", line 194, in engage
    new_args = self.action_manager.run_action(step, self.args)
  File "/home/kdreyer/dev/rdopkg/rdopkg/action.py", line 208, in run_action
    return action_fun(*fun_args)
  File "/home/kdreyer/dev/rdopkg/rdopkg/actions/distgit/actions.py", line 515, in check_new_patches
    if helpers.is_same_hash(hash, old_hash):
  File "/home/kdreyer/dev/rdopkg/rdopkg/helpers.py", line 86, in is_same_hash
    if h1 == h2 or h1.startswith(h2) or h2.startswith(h1):
TypeError: startswith first arg must be str, unicode, or tuple, not NoneType

I should mention that in this case the ceph-2-rhel-patches branch was modified such that the last couple of commits were deleted and that was all - none were added.

I found that this crash happens because I have an additional .patch file at the root of my dist-git tree that lacks the standard headers that git-format-patch creates. specfile.get_patches_from_files() returns None for the hash in that case.

This is fallout from the fact that we have to occasionally ship .patch files for Git submodules :(
https://review.rdoproject.org/r/4533

repo: openstack-packages/rdopkg