text:
Trace events lib changes

Description
As part of tracing framework https://github.com/opensearch-project/OpenSearch/pull/7648, we added tracing capabilities in OpenSearch, this proposal is to add diagnosis metrics which can be associated with spans. It can help provide additional insights on resource consumption around different parts of the system.  When enabled, resource consumption will be recorded between start and end of the span and will publish the net consumption. It will consider resources consumed by all threads involved in the lifecycle of a span. Example usage, assuming segment merge is instrumented with tracing framework, on enabling diagnostic on segment merge traces, it will start emitting resource usage by merges such as heap allocation, cpu usage etc. Such diagnostic metrics could pin point the bottlenecks in the system and exact scaling requirements.
Tenet

Minimal latency overhead. No impact when disabled.
Failure in computing diagnosis metrics should not fail the request execution.
Should be behind dynamic cluster setting based feature flag.

Trace Event Listener
In order to add diagnosis metrics associated with span, trace event listeners are required which are called on span start and span end. To handle thread context switch and record resource consumption associated with the new thread spawned,  Runnables associated with a span can be wrapped with RunnableWrapper, which is provided by this framework. Wrapped Runnable generates start and end events to accurately measure the resource consumption. TraceEventListener are generic and can be used for other purposes if needed and not just diagnosis.

Above figure illustrates the expected life cycle of a span, which is started in the middle of an execution of a task, started by Thread 1 (of ThreadPool 1). It performs an IO operation, and on completion of IO, Thread 2 (of ThreadPool 1) resumes the task. Thread 2 also does an IO operation and task is resumed later by the Thread 3 (of ThreadPool 1) on IO completion. Thread 3 performs 3 different sub-tasks, which it executes in parallel by spawning three different threads in different executor services, and is non-blocking in nature, so it continues to work on rest of the task, while sub-tasks are delegated to different threads. However, it does wait for sub-tasks completion, without blocking the thread, and on response to the last sub-task, the task is resumed by Thread 2 (of ThreadPool 1) to its completion and the Span is closed somewhere in between. ThreadPool 3, which doesn't have the Runnable wrapped and doesn't have the span and registered event listeners information, thus span events are not generated.
Diagnostic Trace Event Listener
This is a generic implementation of trace event listeners which records resource consumption using provided ThreadResourceRecorder and emit corresponding metrics using provided MetricEmitter.
Thread Resource Recorder
Thread resource recorder exposes start and end recording method which are used by DiagnosticTraceEventListener to record resource consumption. On start recording, it observes the current usage of a resource using provided resource observer. On end recording, it again observes the current readings, computes the difference in observed metric from start point and emits them. The metric is of delta temporality type.
Thread Resource Observer
It simply exposes observe() method to observe the current reading associated with a thread.
Related Tasks

[ ] OpenTelemetry meter integration to emit JMX based diagnosis metrics - https://github.com/rishabhmaurya/OpenSearch/pull/47
[ ] OpenSearch server integration - wrapping tracer, spans and runnable for diagnostics support - https://github.com/rishabhmaurya/OpenSearch/pull/46

Related Issues
Resolves #[Issue number to be closed when this PR is merged]

Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@rishabhmaurya, Thanks a lot for putting this up. I have a few question.

I want to understand the motivation behind combining the span and diagnostic information. Do we want to capture the diagnostic info for a span/trace or to capture the diagnostic info we are creating the span.
Anyways as of now these metrics are being collated at span level but emitted independently. So can we keep these 2 pieces separate and link with exemplars through spanContext?
How are you thinking of linking these metrics and spans later for analysis? I see you are passing the attributes. Do attributes always have traceID?
Maybe too early but how do these metrics work in case of sampling? Will these also be sampled along with span or not?
I have concerns about exposing the Span publicly. If you just need SpanType and hasEnded attribute then can we just expose these properties through either SpanScope or minimum version of span?

I want to understand the motivation behind combining the span and diagnostic information. Do we want to capture the diagnostic info for a span/trace or to capture the diagnostic info we are creating the span.

the motivation is to capture additional metrics around resource usage associated with a span. This is an optional capability which can be enabled on per span basis if needed and default behavior would be disabled diagnostics. This is helpful for long running jobs, where resource consumption can be recorded for a span like segment merges, peer recovery, index engine start etc and for such long running job, the overhead would be very minimal when compared to the runtime of these jobs. Diagnosis shouldn't be enabled for operations which have very tight SLA of order of 10ms latency. We do something similar with resource tracking framework at task level and this is a generic way to solve the problem and is built in an extensible way.

Make sense, I was just thinking if we can keep metric and span separate and link them with traceID then metrics can be supported throughout as most of the work is already done by you.

Anyways as of now these metrics are being collated at span level but emitted independently. So can we keep these 2 pieces separate and link with exemplars through spanContext?

I didn't quite get this part. But you're right about collection and emission and they are independent. This framework leaves emission logic open ended. For example, This PR is using opentelemetry for metric emission. - https://github.com/rishabhmaurya/OpenSearch/pull/44/commits. Whereas, if we want to use something else, like stats API to emit into or logging exporter, it can be integrated too.

My point is, we are not doing any aggregation at span level so there metrics captured for thread are independent and can be independently emitted without clubbing with the span logic.

How are you thinking of linking these metrics and spans later for analysis? I see you are passing the attributes. Do attributes always have traceID?

traceID would a regular field associated with metric which can used to filter when visualizing, it will not be a dimension attribute to avoid high cardinality dimension.

How this is being done? Can you please point me to the code path?

Maybe too early but how do these metrics work in case of sampling? Will these also be sampled along with span or not?

It is the right time to discuss it. It depends on when we are sampling, if it is at collector level, then all of these diagnosis metrics will be available to collector. Metric sampling works differently at collector, they can both be aggregated or sampled at collector and a lot depends on the aggregation function. I will add more details around it.

I am talking for head sampling.

I have concerns about exposing the Span publicly. If you just need SpanType and hasEnded attribute then can we just expose these properties through either SpanScope or minimum version of span?

Adding points we discussed offline around it a few days ago - SpanScope is getting bloated with a lot of redundant methods same as Span. We wanted to avoid exposing Span because of endSpan() method, which we only want to be exposed to framework internal classes like SpanScope, which can call it when SpanScope is closed. Is there any other reason why we don't want to expose Span publicly?

No this is the main concern. Anyways this is being discussed here - https://github.com/opensearch-project/OpenSearch/pull/8831/files/7bac1e33dea887e9c35e08e54b8e7d5d2bc53bcf#r1281897313

cc @reta

How are you thinking of linking these metrics and spans later for analysis? I see you are passing the attributes. Do attributes always have traceID?

traceID would a regular field associated with metric which can used to filter when visualizing, it will not be a dimension attribute to avoid high cardinality dimension.

How this is being done? Can you please point me to the code path?

I would relying on the default attributes and traceId being one of them, that doesn't seems to be the case, so I will add it as a default attribute. I will address it in the follow up commit.

Maybe too early but how do these metrics work in case of sampling? Will these also be sampled along with span or not?

It is the right time to discuss it. It depends on when we are sampling, if it is at collector level, then all of these diagnosis metrics will be available to collector. Metric sampling works differently at collector, they can both be aggregated or sampled at collector and a lot depends on the aggregation function. I will add more details around it.

I am talking for head sampling.

I will think more about it and we may have to address it when we add head sampling logic for traces.

cc @reta looking forward to your comments. Retagging in case you missed it earlier, thanks.

@rishabhmaurya could you please rebase? there are conflicts, thank you

cc @reta looking forward to your comments. Retagging in case you missed it earlier, thanks.

@rishabhmaurya could you please rebase? there are conflicts, thank you

There were some disruptive changes with last rebase, but its rebased now. Thank you.

@rishabhmaurya I am not sure what is the state of this pull request but here are my general observations:

the telemetry support is split across 2 pieces (right now): tracing and metric
the telemetry may provide both or just one
the metrics and tracing should be separated (they are very much coupled now)

In case of OTel (our plugin):

the metrics, if available, could be integrated with tracing using SpanProcessor
the tracing should be using semantic conventions (https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/span-general.md#general-thread-attributes) to facilitate context passing (like thread ids, etc)

Please bring this pull request into the state where at least checks are passing, thank you.

@Gaganjuneja @reta
I was thinking about using SpanProcessor however it can only be tied the to Span start/end events. The reason for using TraceEventListener was to accurately measure the resource consumed by the different threads involved in the lifecycle of the span, so just start and end event of span were not sufficient, thus I associated Runnable start and end event as well.
The framework implementation has changed quite a bit and if my understanding is correct, SpanScope is something which can start and end multiple times now for a given span depending on when a thread starts/ends its execution.
Does it makes sense to associate these events with SpanScope instead of Span start/end and Runnable start/end?

@Gaganjuneja @reta I was thinking about using SpanProcessor however it can only be tied the to Span start/end events. The reason for using TraceEventListener was to accurately measure the resource consumed by the different threads involved in the lifecycle of the span, so just start and end event of span were not sufficient, thus I associated Runnable start and end event as well. The framework implementation has changed quite a bit and if my understanding is correct, SpanScope is something which can start and end multiple times now for a given span depending on when a thread starts/ends its execution. Does it makes sense to associate these events with SpanScope instead of Span start/end and Runnable start/end?

@rishabhmaurya, I think here we have 2 uber points.

How can we evaluate the metric in the scope of a span. Either through EventListener and Runnables or SpanScope.
How can we integrate the metric and traces?

As @reta also mentioned, metric and traces are two distinct components and shouldn't be coupled as in the running prod environment it both may be independently enabled/disabled or even coexist. Traces would get sampled as well but metrics we may not want to sample as these are aggregate in nature. So in case we need to integrate them together then it should be done at a framework level once both are generated.
But if I understood your use case, then you just want to aggregate a metric for all the threads being used in the scope of a span. We need to think of a way to do that like using some callbacks/logic with span state or maybe adding spanId as an attribute to the metric etc.

This PR shouldn't be confused with adding metrics support in the framework. It has following components -

Adding event listeners around spans to precisely track when a thread start/end working on a span for all threads involved for a given span (and subspans).
Thread resource recorder and observer interface to track resource utilization of threads involved in a span (aka DiagnosticSpan).
Emission of resource utilization computed using MetricEmitter interface.

If you look at the interfaces, all 3 are components are very much decoupled. MetricEmitter when implmented in OpenTelemetry uses OpenTelemetry meters which are part of another PR - https://github.com/rishabhmaurya/OpenSearch/pull/47

I think we are not digressing towards the Metrics Framework (Anyways for that I have put up an RFC #10141). I think we need the answer of following fundamental questions and then we should be proceed.

How are we going to use these metrics?
The impact on metric if spans are getting sampled?
What is the expected state when either metrics or tracing or both are disabled.
If this is so coupled with the scope of a Span then why can't we just make it an attribute to a Span?

How are we going to use these metrics?

The use of metric is open ended in the framework lib. If we use OpenTelemetry to emitMetric() then the opentelemetry meters would be used to emit these metrics and metrics would contain the span attributes which can be used to filter/aggregate metrics later on.
For instance, assuming PeerRecovery code is wrapped around spans, we can measure the resources consumed by the threads involved in PeerRecovery. We can visualize the overall impact and can drill down on the basis of attributes such as source_node, destination_node, index, shard or which phase consumed most resources (send_files, translog_recovery)as these attributes would be associated with metrics while emission.
Currently, its not adding trace-id as an attribute, however if added, consumption can be filtered on the basis of individual traces too if needed.

The impact on metric if spans are getting sampled?

As far as span context is propagated, span events would be generated and resource recording will happen and metric would be emitted.

What is the expected state when either metrics or tracing or both are disabled.

When metrics are disabled, resource usage would be computed but will not be emitted.
When tracing is completely disabled, resource usage won't be computed as there are no spans for which the resource needs to be computed and thus no metrics would be generated.
If both are disabled, nothing would be computed or emitted.

If this is so coupled with the scope of a Span then why can't we just make it an attribute to a Span?

the only reason its coupled with Span is because its recording the usage of a span and its not coupled directly to span but depends on the events of the spans.
Did you mean to say why can't we emit these metrics as part of span attribute or emit a span event?
If yes, we want them as metrics and not attributes. Metrics can be stored in a timeseries database, can be visualized, aggregated and filtered, which isn't possible with raw span attributes/events.

@rishabhmaurya let's try to get back to basics

We have two separate components: traces and metrics (see please https://github.com/opensearch-project/OpenSearch/issues/10141)
By and large, those components are independent and could function independently together or individually

Now, let's try to understand what are the relations between tracing and metrics:

tracing is about execution flow: how the individual request travels through the system
metrics are about operational state of the system expressed in terms of quantifiable measures

What are the lifetimes of tracing data and metrics in the functioning (up and running) system? Most of the metrics live as along as the system is running (I am discarding the persistent layer as it is not relevant at the moment), the system like OpenSearch may have thousands of metrics (and it really has) exposed. Most of the tracing data (like spans) live for a duration of the request (again, I am discarding the persistent layer as it is not relevant at the moment), the system like OpenSearch may have thousands of spans every single second.
Span have a mechanism to capture additional details (like attributes and events), but I would argue that spans should not mess with metrics in any way imaginable. Why is that:

each span has only limited context it is aware of, it does not have sufficient details to produce meaningful metrics
however, some metrics could be derived from spans and it is done by collectors (see please https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/README.md)

When enabled, resource consumption will be recorded between start and end of the span and will publish the net consumption. It will consider resources consumed by all threads involved in the lifecycle of a span.

First of all, we already have the tracking mechanism, see please TaskResourceTrackingService & related APIs, we definitely don't over tracking here. Secondly, could you please explain what are practical reasons to track CPU time for each and every span in the system? I would suspect the >90% of them would have 0 time spend but the overhead of calling ThreadMXBean would be huge.

Thanks @reta for the feedback. You have raised all valid concerns and its a privilege to discuss them with folks here who understand this part of system well and get their feedback. I would like to trace back my thought process on how I arrived at this solution and then we can decide if we want to move ahead with this solution or a different solution.
Its mostly derived from managed service use cases (and should be helpful for the whole community) where we don't have enough insights on system level metrics for background jobs like peer recovery, merges, index management jobs like ultrawarm-migrations/transforms/rollover etc. These job can have impact on the search/indexing latency or resiliency of the system and it becomes very hard to troubleshoot such problem both for live and past events.
My plan was to add capability to emit metrics which are available with the performance analyzer plugin - https://opensearch.org/docs/latest/monitoring-your-cluster/pa/reference/ only for background jobs and not for each search/indexing request. The number of events generated by background tasks is most of the time directly proportional to the rate of growth of data, various indexing policies and cluster events. Whereas, the number of events generated by the requests is directly proportional to the number of requests. The majority of the events of any background task are long running ( > 1s) unlike requests where majority of events are shortlived (< 10 msecs). These are important invariants in the strategy to design the impact measurement for background tasks and for this reason the proposal here cannot be used for requests.
Your concern is valid and it's crucial to consider the trade-offs, including the potential overhead, and ensure that the data collected serves a meaningful purpose in optimizing system performance or diagnosing issues. There is no one-size-fits-all answer, and this approach should be selective in nature where operator can choose for which background job this should be enabled and which metrics to emit. For instance, if we want to get insights on segment merges, we may want to enable metrics like - cpu time, heap allocation rate, IO read/write throughput, Disk metrics, page faults - major/minor.  Similarly, for peer recovery, there could be contention between threads esp while translog recovery, so we may want to monitor additionally thread block_time.
Once the impact is quantified, these metrics can be used for various purposes - (1) Improving the resiliency by detecting issues faster, understanding the bottlenecks because of a background task and devising the remediation by changing configuration at various layers - OpenSearch (Shards/Index or Cluster), JVM or by using the right hardware or scaling the machines. (2) Building automation to throttle the background tasks at most granular level like shards and nodes on which they run. This would contribute towards improving the availability of the system. (3) Improving the quality of software releases by using these metrics to create baselines of background tasks impact, for different workloads defined in the benchmark.
** Why I'm trying to club span events with metrics? **
If we have spans around these background jobs, then we don't have to write any additional code to get these system level metrics around them. It would just be some configuration to be enabled and choosing available metrics to be computed/emitted against each of these background task. Spans generated for background jobs would be very less in number compared to spans for search/indexing request, so scaling should be an issue. Also, overhead of calling ThreadMXBean or any metric computation logic would be limited too. I agree we should not enable such metrics for regular search/indexing requests.

@rishabhmaurya  -- I've read through your comment above, and I'm still not entirely sure what the purpose of this PR should be -- which I think is more on me than it is on you -> I lack sufficient context, but I would like to get it. Essentially, I would like to review your PR, but I don't know where to start.
Can you provide a tl;dr: explanation in three sentences that conveys the following:

Who needs to care about this PR -- essentially, who is the target audience?
What problem does this PR solve?
How does the PR solve the problem?

Who needs to care about this PR -- essentially, who is the target audience?

Target audience is cluster administrators to get insights into scaling requirements, schedule (if possible) background tasks  at appropriate time, insights into increased latency because of contention with background tasks.
Background job could be segment merges, snapshots, shard peer recovery, index management jobs etc.

What problem does this PR solve?

It assumes once a background job is wrapped around span, its span events can be used to trigger computation of resources consumed by the threads involved. It will compute the delta of resources consumed by threads from start to end of the span and publish it in form of metrics.
Initial plan is to support metrics which can be computed using ThreadMXBean. Ultimately support subset of metrics from https://opensearch.org/docs/latest/monitoring-your-cluster/pa/reference/ like Disk IO - readBytes, writeBytes, readSysCallRate, writeSysCallRate, page faults etc.
Admins should be able to control for which background job this diagnosis needs to be enabled dynamically. They should also have control to select which metrics needs to be enabled against a background job.

How does the PR solve the problem?

It exposes trace event listeners and runnable event listeners (Runnable wrapper to listen to event for a Runnable when it starts/stop executing a task).
It adds a DiagnosisSpan, which is a wrapper over a Span. DiagnosticsEventListener which is a TraceEventListener have access to DiagnosisSpan to persist the start reading and other attributes and use it to compute delta at end event and publish the metrics with the right attributes.
This PR adds ThreadResourceObserver and ThreadResourceRecorder. ThreadResourceRecorder starts and ends recording for a ThreadResourceObserver is observing and is invoked by DiagnosticsEventListener.
Its a pluggable architecture so one can write a plugin to add logic to observe and record usage of some other resource.

Thanks @msfroh for taking a look.

Thanks @reta for the feedback. You have raised all valid concerns and its a privilege to discuss them with folks here who understand this part of system well and get their feedback. I would like to trace back my thought process on how I arrived at this solution and then we can decide if we want to move ahead with this solution or a different solution.

Thanks for the context @rishabhmaurya, it is interesting that you have mentioned merges a few times, we have been thinking about them a lot but it seems like there should be a lot of prerequisite work to be done in order to even get this processes somehow instrumented (merge schedulers etc). Right now Apache Lucene does use internal pools and threads to spawn  merges but has to be changed.
I would like to suggest the following plan of action:

kick off https://github.com/opensearch-project/OpenSearch/issues/10141 - that will give the foundation for metrics (as the holistic APIs)
look into existing TaskResourceTrackingService, it is supposed to be tracing the most important flows in the system (which are usually executed as tasks)
specifically related to merges, look into instrumenting the Apache Lucene merge process

With respect to this particular pull request, I would suggest to put it on hold (draft mode?)  and get back to it when we a) have all pieces together b) conclude that this is the way to implement resource tracking or alike

@reta  my understanding for merge was, since we use custom OpenSearchConcurrentMergeScheduler, thus wrapping doMerge() with span should be sufficient since its a single threaded task.

@reta my understanding for merge was, since we use custom OpenSearchConcurrentMergeScheduler, thus wrapping doMerge() with span should be sufficient since its a single threaded task.

@rishabhmaurya hm ... doMerge() in called in MergeThread , whereas the merge thread is created by getMergeThread, wrapping doMerge() won't be sufficient since we have propagate the context to the MergeThread first, unless I am missing something.

@rishabhmaurya hm ... doMerge() in called in MergeThread , whereas the merge thread is created by getMergeThread, wrapping doMerge() won't be sufficient since we have to propagate the context to the MergeThread first, unless I am missing something.

@reta since merge thread is single threaded for a single merge task, do we have to care of context propagation? It sounds like a little hack but should work if we pass the tracer to OpenSearchConcurrentMergeScheduler constructor and start and end the span in doMerge().

@rishabhmaurya hm ... doMerge() in called in MergeThread , whereas the merge thread is created by getMergeThread, wrapping doMerge() won't be sufficient since we have to propagate the context to the MergeThread first, unless I am missing something.

@reta since merge thread is single threaded for a single merge task, do we have to care of context propagation? It sounds like a little hack but should work if we pass the tracer to OpenSearchConcurrentMergeScheduler constructor and start and end the span in doMerge().

@rishabhmaurya No, we need to see not only merge thread(s), which would appear as isolated spans, but how the merge was triggered, background task or force merge, other means. For this purposes we need context propagation.

@reta how about propagating context to merge thread in getMergeThread() - https://github.com/opensearch-project/OpenSearch/blob/13fc7cce5c0608afb0d3b5a5684e3c424f7b1081/server/src/main/java/org/opensearch/index/engine/OpenSearchConcurrentMergeScheduler.java#L192
so if the thread executing IndexWriter merge functions has the context, it should work?

so if the thread executing IndexWriter merge functions has the context, it should work?

Yeah, that the instrumentation that I mentioned and needed to be in place, however we also need to make sure the flow where OpenSearchConcurrentMergeScheduler is running is also instrumented.

authors: Gaganjuneja msfroh reta rishabhmaurya


text:
More logging

 Issue by lucaswin-amzn
Monday Jul 01, 2019 at 22:08 GMT
Originally opened as https://github.com/opendistro-for-elasticsearch/alerting/issues/77

We are currently missing some logging information around trigger evaluation, and in general we could improve debug capability for users with more logging.
See here for an example

Closing as resolved by #1223

source: gharchive/issue

repo: opensearch-project/alerting

license_type: permissive

license_source: github-api


text:
Updates query DSL index page

Signed-off-by: alicejw alicejw@amazon.com
Description
The query DSL topic pages needed a better overview and index page.
What's reorganized:

moved table of term-level and full-text queries to the index page
reformatted old style formatting (e.g. bold font for examples -> H4 for sample request & response sections)
added table of term-level query types with descriptions

Issues Resolved
2452
PDFs for reviewers
Query DSL - OpenSearch documentation.pdf
Term-level queries - OpenSearch documentation.pdf
Checklist

[ x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

updated pdf:
Query DSL - OpenSearch documentation.pdf

Hi @kolchfa-aws , would you please take a look at the full list of query types that I've added. Thanks

Hi @alicejw-aws - this needs to go through tech review before doc review. Can you please make sure to tag a developer on this PR? Thanks.

Hi @macohen, per Heather's request, would you please review the list of query types that we support and verify accuracy? Thanks!

Hi @hdhalter , sounds good. There were additional changes requested from the doc reviewer that now require a tech review. The original reorg of previously tech-reviewed content did not require a tech review. Thanks

Note: I've updated all query type descriptions. However, we do need a tech reviewer to approve it. Thanks!
Query DSL - OpenSearch documentation.pdf

Todo: Alice

create sub-topic pages for each query type category
move detailed descriptions to sub-topic pages
Index page - bullet list of categories  - Name -

Created necessary category sub-pages:

Geographic and shape queries
Span queries

Ready for an additional doc review.
See updated PDF:
Query DSL - OpenSearch documentation.pdf

Thanks @kolchfa-aws for your review!

Updated PDFs:
Query DSL.pdf
Geographic and shape queries.pdf
Span queries.pdf

@alicejw-aws Is this ready for editorial review?

Thanks @cwillum for your review!
Updated PDF:
Query DSL

Hi @natebower , just for clarification on the goals for this rewrite, we won't provide redundant information.
For the index page we needed a comprehensive list of all query types that we support.
Index bullet list:

If we have a topic page, then link to that page.
If the page is not documented yet, no link.

To keep the bullet list short, (doc reviewer suggestion) two sub-pages were created for those two categories, and the sub-pages contain the detailed descriptions for all query types in that category.
Thanks!

Hi @natebower , just for clarification on the goals for this rewrite, we won't provide redundant information.
For the index page we needed a comprehensive list of all query types that we support.
Index bullet list:
* If we have a topic page, then link to that page.

* If the page is not documented yet, no link.

To keep the bullet list short, (doc reviewer suggestion) two sub-pages were created for those two categories, and the sub-pages contain the detailed descriptions for all query types in that category. Thanks!

Thanks for the clarification, @alicejw-aws. I'll keep it in mind when I resume my review.

Hi @natebower, I've made the changes requested from Fanit.
Please continue your editorial review, and I'll make all of your requested changes after that.
Thanks

Hi @natebower, I've made the changes requested from Fanit.
Please continue your editorial review, and I'll make all of your requested changes after that. Thanks

@alicejw-aws Go ahead and make the changes I've submitted so far. That way I can review a clean copy following tech review. Thanks!

Hi @natebower, ok will do! Thanks

@VijayanB, can you please review the geo/maps related documentation in this PR? Much appreciated!

@VijayanB, can you please review the geo/maps related documentation in this PR? Much appreciated!

Closed in favor of #2904

source: gharchive/pull-request

authors: alicejw-aws hdhalter kolchfa-aws macohen natebower


text:
Update JDK version for 2.x distributions

Description
Update JDK version for 2.x distributions
Issues Resolved
Related to https://github.com/opensearch-project/OpenSearch/pull/16426
Version
List the OpenSearch version to which this PR applies, e.g. 2.14, 2.12--2.14, or all.
Frontend features
If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional.
Checklist

[ ] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@kolchfa-aws this is for 2.18.0, JDK distribution version update, thank you!


text:
[BUG] ML Inference Search Processors should have different model input format when input mapping has dollar symbols

What is the bug?
In ML Inference Processors,
For ingest side, we consider the use case of  getting value from nested object and getting value from object, so we support both dot path notation to get field value, and support json path notation that can get nested value in a list.
But for search processors, we support json path to read the object, the behavior is different, it always return the original data format. We want to have two ways of reading the object similar to ingest side.
How can one reproduce the bug?
In 2.14, using ml_inference ingest processor:
for nested document like this sample book index,
{
  "book": [
    {
      "chunk": {
        "text": [
          {
            "chapter": "first chapter",
            "context": "this is the first part"
          },
          {
            "chapter": "first chapter",
            "context": "this is the second part"
          }
        ]
      }
    },
    {
      "chunk": {
        "text": [
          {
            "chapter": "second chapter",
            "context": "this is the third part"
          },
          {
            "chapter": "second chapter",
            "context": "this is the fourth part"
          }
        ]
      }
    }
  ]
}

we can config the input maps as {"input": "$.book.*.chunk.text.*.context"}, which fetch the model input as
{
  "input": [
    "this is the first part",
    "this is the second part",
    "this is the third part",
    "this is the fourth part"
  ]
}

for simple object like this sample item index,

{
   "item_text":"red shoes"
}

if configuring input_map as {"input": "item.text"}
the model input will be in string representation.
{
  "input": "red shoes"
}

if configuring input_map as {"input": "$.item.text"}
the model input will be list representation.
{
  "input": ["red shoes"]
}

But in search processors:
for the same item index, if configuring input_map as {"input": "$.item.text"},
the model input will be in string representation.
{
  "input": "red shoes"
}

What is the expected behavior?
we would like the similar logic as ingest processors, when using dot path notation without dollar symbol '$', it will get the original data format, but when using  dollar symbol '$', it should return a list of value representations.
in search processors:
for the same item index, if configuring input_map as {"input": "$.item.text"},
the model input will be in list representation.
{
  "input": ["red shoes"]
}

for the same item index, if configuring input_map as {"input": "item.text"},
the model input will be in list representation.
{
  "input": "red shoes"
}

What is your host/environment?

OS: [2.17]
Plugins

will add documentations in both ingest and search pipelines.

checked on ConnectorUtils, it will return its original form of object
https://github.com/opensearch-project/ml-commons/blob/6a6cac13ee1a009ea43a3bad147059578d2adc38/ml-algorithms/src/main/java/org/opensearch/ml/engine/algorithms/remote/ConnectorUtils.java#L222

Instead of allowing different configurations when providing with/without dollar symbol.
It's better to use standard configuration across the ml-commons repo.
If users would like to use different jsonpath configurations, we should open up a new parameters to change the jsonpath configuration settings.

repo: opensearch-project/ml-commons


text:
Add jenkins release workflow, signing key and update public key

Signed-off-by: Sayali Gaikawad gaiksaya@amazon.com
Description

Adds jenkins release workflow
Adds signing key so that built artifacts are signed when tags are pushed
Updates public key

Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

There are a couple of broken links. I raised a PR but we can just add that fix to this PR if you dont mind: https://github.com/opensearch-project/opensearch-ruby/pull/117/files#diff-155d93b4eb42e4d2e1f4aaad7934dfcb7a13953497c8d8446940577be354853d

Webhook is in place now. We are ready for the release.
Thanks!

authors: gaiksaya nhtruong

repo: opensearch-project/opensearch-ruby


text:
Testing 2.7 windows build failure fix.

Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #542 (a3926f9) into 2.7 (74f3bc8) will increase coverage by 0.01%.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##              2.7     #542      +/-   ##
==========================================
+ Coverage   32.65%   32.66%   +0.01%
==========================================
  Files         135      135
  Lines        4015     4016       +1
  Branches      695      695
==========================================
+ Hits         1311     1312       +1
  Misses       2564     2564
  Partials      140      140

see 2 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: AWSHurneyt codecov-commenter


text:
[BUG] Flag log_request_body is not honored when logging compliance diff for document updates

What is the bug?
When we enable Compliance logging with diffs ( write_log_diffs ) and disable writing metadata ( write_metadata_only ) , request body is always logged even when log_request_body is disabled and it's not the first document insert.
How can one reproduce the bug?

Launch Opensearch 2.13.0 cluster locally and enable security using the default configurations.

Additionally, set plugins.security.audit.type in opensearch.yml to log4j to see the audit logs generated easily.

Update the audit logging config as follows :

curl -XPUT -kv -H 'Content-Type: application/json' https://localhost:9200/_opendistro/_security/api/audit/config -u 'admin:xx' -d '{
  "enabled": true,
  "audit": {
    "ignore_users": [],
    "ignore_requests": [],
    "disabled_rest_categories": [
      "AUTHENTICATED"
    ],
    "disabled_transport_categories": [
      "AUTHENTICATED"
    ],
    "log_request_body": false,
    "resolve_indices": false,
    "resolve_bulk_requests": false,
    "exclude_sensitive_headers": true,
    "enable_transport": false,
    "enable_rest": true
  },
  "compliance": {
    "enabled": true,
    "write_log_diffs": true,
    "read_watched_fields": {},
    "read_ignore_users": [],
    "write_watched_indices": ["index1"],
    "write_ignore_users": [],
    "read_metadata_only": true,
    "write_metadata_only": false,
    "external_config": false,
    "internal_config": true
  }
}'

Insert a document in an index

curl -XPUT -kv -H 'Content-Type: application/json' https://localhost:9200/index1/_doc/4 -u 'admin:xx' -d '{
  "name": "helloworld",
  "age": 18
}'

Audit log generated will look like
{
    "audit_compliance_operation": "UPDATE",
    "audit_cluster_name": "opensearch",
    "audit_node_name": "example.com",
    "audit_category": "COMPLIANCE_DOC_WRITE",
    "audit_request_origin": "REST",
    "audit_compliance_doc_version": 2,
    "audit_request_body": "{\n  \"name\": \"helloworld\",\n  \"age\": 18\n}",
    "audit_node_id": "oenHT55bTNW4XB358NS7Qw",
    "@timestamp": "2024-07-09T06:56:04.947+00:00",
    "audit_compliance_diff_is_noop": true,
    "audit_format_version": 4,
    "audit_request_remote_address": "::1",
    "audit_trace_doc_id": "4",
    "audit_node_host_address": "127.0.0.1",
    "audit_request_effective_user": "admin",
    "audit_trace_shard_id": 0,
    "audit_trace_indices": ["index1"],
    "audit_trace_resolved_indices": ["index1"],
    "audit_node_host_name": "127.0.0.1"
}

Since, this is the first time inserting the doc, audit_compliance_diff_is_noop is true and it makes sense for audit_request_body to be logged.

Update one field in the same document

curl -XPUT -kv -H 'Content-Type: application/json' https://localhost:9200/index1/_doc/4 -u 'admin:xx' -d '{
  "name": "helloword",
  "age": 18
}'

Audit log generated looks like
{
    "audit_compliance_operation": "UPDATE",
    "audit_cluster_name": "opensearch",
    "audit_node_name": "example.com",
    "audit_category": "COMPLIANCE_DOC_WRITE",
    "audit_request_origin": "REST",
    "audit_compliance_doc_version": 3,
    "audit_request_body": "{\n  \"name\": \"helloword\",\n  \"age\": 18\n}",
    "audit_node_id": "oenHT55bTNW4XB358NS7Qw",
    "@timestamp": "2024-07-09T06:56:35.601+00:00",
    "audit_compliance_diff_is_noop": false,
    "audit_format_version": 4,
    "audit_request_remote_address": "::1",
    "audit_trace_doc_id": "4",
    "audit_compliance_diff_content": "[{\"op\":\"replace\",\"path\":\"/name\",\"value\":\"helloword\"}]",
    "audit_node_host_address": "127.0.0.1",
    "audit_request_effective_user": "admin",
    "audit_trace_shard_id": 0,
    "audit_trace_indices": ["index1"],
    "audit_trace_resolved_indices": ["index1"],
    "audit_node_host_name": "127.0.0.1"
}

Note both audit_request_body and audit_compliance_diff_content are getting logged even if log_request_body is disabled and we are updating the same document.
What is the expected behavior?
audit_request_body should not be always logged for compliance audit logs when log_request_body is disabled and it's not the first document insert.
What is your host/environment?

OS: MacOS

Do you have any additional context?
We check that write_metadata_only == false and write_log_diffs == true here before logging audit_compliance_diff_content
Whereas we only check write_metadata_only == false before logging audit_request_body here
A simple fix is to also add log_request_body == true at the same place AND ensure that it's not the first time when document is being inserted.

[Triage] Thank you for filing this issue with detailed steps to reproduce. Marking the issue as triaged.

I believe the full request body being added is because of this line which gets executed when read_metadata_only is set to true.
I'm not familiar with the history of this file so I will see if I can dig into why the full request body is being logged here.

I created a quick POC pr on my fork here: https://github.com/cwperks/security/pull/31
Currently, there is a test for a non-security index that is configured with write_watched_indices, but another similar test should be added utilizing security APIs to create and modify security resources and ensure that the diffs are audit logged or the entire body is audit logged when a resource is first created.

Closing this issue now that https://github.com/opensearch-project/security/pull/4832 is merged. This will be released in v2.19.0.

authors: cwperks expani

repo: opensearch-project/security


text:
Add JDBC driver jenkins CI workflow

Description
Please, see team review discussion in https://github.com/Bit-Quill/opensearch-project-sql/pull/140
Jenkins file
Combined from this and this.
It calls publishing gradle workflow to generate artifacts.
Gradle workflow
New workflow (should be called from sql-jdbc dir)
./gradlew publishPublishMavenPublicationToLocalRepoRepository

Files generated by gradle
$ tree build | grep -vF .class
build
├── classes
│   └── java
│       └── main
│           └── org
│               └── opensearch
│                   └── jdbc
│                       ├── auth
│                       ├── config
│                       ├── internal
│                       │   ├── exceptions
│                       │   ├── results
│                       │   └── util
│                       ├── logging
│                       ├── protocol
│                       │   ├── exceptions
│                       │   └── http
│                       ├── transport
│                       │   └── http
│                       │       └── auth
│                       │           └── aws
│                       └── types
├── libs
│   └── opensearch-sql-jdbc-2.0.0.0.jar
├── publications
│   └── publishMaven
│       └── pom-default.xml
├── repository
│   └── org
│       └── opensearch
│           └── client
│               └── opensearch-sql-jdbc
│                   ├── 2.0.0.0
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom.md5
│                   │   └── opensearch-sql-jdbc-2.0.0.0.pom.sha1
│                   ├── maven-metadata.xml
│                   ├── maven-metadata.xml.md5
│                   └── maven-metadata.xml.sha1
├── resources
│   └── main
│       └── META-INF
│           └── services
│               └── java.sql.Driver
└── tmp
    ├── compileJava
    ├── jar
    │   └── MANIFEST.MF
    └── publishPublishMavenPublicationToLocalRepoRepository
        └── org
            └── opensearch
                └── client
                    └── opensearch-sql-jdbc
                        ├── maven-metadata-remote.xml
                        └── resolver-status.properties

42 directories, 159 files

See artifacts in artifacts.zip. The jar file has no dependencies inside.

jar content
$ tree
.
├── META-INF
│   ├── MANIFEST.MF
│   └── services
│       └── java.sql.Driver
└── org
    └── opensearch
        └── jdbc
            ├── ConnectionImpl.class
            ├── DatabaseMetaDataImpl$ColumnMetadataStatement$ColumnMetadataResultSet.class
            ├── DatabaseMetaDataImpl$ColumnMetadataStatement.class
            ├── DatabaseMetaDataImpl$ResultSetColumnDescriptor.class
            ├── DatabaseMetaDataImpl.class
            ├── Driver.class
            ├── OpenSearchConnection.class
            ├── OpenSearchDataSource.class
            ├── OpenSearchVersion.class
            ├── PreparedStatementImpl.class
            ├── ResultSetImpl.class
            ├── ResultSetMetaDataImpl.class
            ├── StatementImpl.class
            ├── auth
            │   └── AuthenticationType.class
            ├── config
            │   ├── AuthConnectionProperty.class
            │   ├── AwsCredentialsProviderProperty.class
            │   ├── BoolConnectionProperty.class
            │   ├── ConnectionConfig$1.class
            │   ├── ConnectionConfig$Builder.class
            │   ├── ConnectionConfig.class
            │   ├── ConnectionProperty.class
            │   ├── ConnectionPropertyException.class
            │   ├── FetchSizeProperty.class
            │   ├── HostConnectionProperty.class
            │   ├── HostnameVerificationConnectionProperty.class
            │   ├── IntConnectionProperty.class
            │   ├── KeyStoreLocationConnectionProperty.class
            │   ├── KeyStorePasswordConnectionProperty.class
            │   ├── KeyStoreTypeConnectionProperty.class
            │   ├── LogLevelConnectionProperty.class
            │   ├── LogOutputConnectionProperty.class
            │   ├── LoginTimeoutConnectionProperty.class
            │   ├── PasswordConnectionProperty.class
            │   ├── PathConnectionProperty.class
            │   ├── PortConnectionProperty.class
            │   ├── RegionConnectionProperty.class
            │   ├── RequestCompressionConnectionProperty.class
            │   ├── StringConnectionProperty.class
            │   ├── TrustSelfSignedConnectionProperty.class
            │   ├── TrustStoreLocationConnectionProperty.class
            │   ├── TrustStorePasswordConnectionProperty.class
            │   ├── TrustStoreTypeConnectionProperty.class
            │   ├── UseSSLConnectionProperty.class
            │   └── UserConnectionProperty.class
            ├── internal
            │   ├── JdbcWrapper.class
            │   ├── Version.class
            │   ├── exceptions
            │   │   └── ObjectClosedException.class
            │   ├── results
            │   │   ├── ColumnMetaData.class
            │   │   ├── Cursor.class
            │   │   ├── Row.class
            │   │   └── Schema.class
            │   └── util
            │       ├── AwsHostNameUtil.class
            │       ├── JavaUtil.class
            │       ├── SqlParser.class
            │       └── UrlParser.class
            ├── logging
            │   ├── FilePrintWriterLogger.class
            │   ├── Layout.class
            │   ├── LogLevel.class
            │   ├── Logger.class
            │   ├── LoggerFactory.class
            │   ├── LoggingSource$1.class
            │   ├── LoggingSource.class
            │   ├── NoOpLogger.class
            │   ├── PrintWriterLogger.class
            │   └── StandardLayout.class
            ├── protocol
            │   ├── ClusterMetadata.class
            │   ├── ColumnDescriptor.class
            │   ├── ConnectionResponse.class
            │   ├── JdbcDateTimeFormatter.class
            │   ├── JdbcQueryParam.class
            │   ├── JdbcQueryRequest.class
            │   ├── Parameter.class
            │   ├── Protocol.class
            │   ├── ProtocolFactory.class
            │   ├── QueryRequest.class
            │   ├── QueryResponse.class
            │   ├── RequestError.class
            │   ├── exceptions
            │   │   ├── InternalServerErrorException.class
            │   │   ├── MalformedResponseException.class
            │   │   └── ResponseException.class
            │   └── http
            │       ├── HttpException.class
            │       ├── HttpResponseHandler.class
            │       ├── JdbcCursorQueryRequest.class
            │       ├── JsonClusterMetadata.class
            │       ├── JsonConnectionResponse.class
            │       ├── JsonCursorHttpProtocol.class
            │       ├── JsonCursorHttpProtocolFactory.class
            │       ├── JsonCursorQueryRequest.class
            │       ├── JsonHttpProtocol.class
            │       ├── JsonHttpProtocolFactory.class
            │       ├── JsonHttpResponseHandler$JsonContentParser.class
            │       ├── JsonHttpResponseHandler.class
            │       ├── JsonOpenSearchVersion.class
            │       ├── JsonQueryRequest.class
            │       ├── JsonQueryResponse$JsonRequestError.class
            │       ├── JsonQueryResponse$SchemaEntry.class
            │       └── JsonQueryResponse.class
            ├── transport
            │   ├── Transport.class
            │   ├── TransportException.class
            │   ├── TransportFactory.class
            │   └── http
            │       ├── ApacheHttpClientConnectionFactory.class
            │       ├── ApacheHttpTransport.class
            │       ├── ApacheHttpTransportFactory.class
            │       ├── HttpParam.class
            │       ├── HttpTransport.class
            │       ├── JclLoggerAdapter.class
            │       ├── LoggingInputStream.class
            │       ├── LoggingManagedHttpClientConnection.class
            │       ├── LoggingOutputStream.class
            │       └── auth
            │           └── aws
            │               └── AWSRequestSigningApacheInterceptor.class
            └── types
                ├── BaseTypeConverter.class
                ├── BinaryType.class
                ├── BooleanType.class
                ├── ByteType.class
                ├── DateType.class
                ├── DoubleType.class
                ├── FloatType.class
                ├── IntegerType.class
                ├── LongType.class
                ├── NumberType.class
                ├── OpenSearchType.class
                ├── ShortType.class
                ├── StringType.class
                ├── TimeType.class
                ├── TimestampType.class
                ├── TypeConverter.class
                ├── TypeConverters$1.class
                ├── TypeConverters$BigIntTypeConverter.class
                ├── TypeConverters$BinaryTypeConverter.class
                ├── TypeConverters$BooleanTypeConverter.class
                ├── TypeConverters$DateTypeConverter.class
                ├── TypeConverters$DoubleTypeConverter.class
                ├── TypeConverters$FloatTypeConverter.class
                ├── TypeConverters$IntegerTypeConverter.class
                ├── TypeConverters$NullTypeConverter.class
                ├── TypeConverters$RealTypeConverter.class
                ├── TypeConverters$SmallIntTypeConverter.class
                ├── TypeConverters$TimeTypeConverter.class
                ├── TypeConverters$TimestampTypeConverter.class
                ├── TypeConverters$TinyIntTypeConverter.class
                ├── TypeConverters$VarcharTypeConverter.class
                ├── TypeConverters.class
                ├── TypeHelper.class
                └── UnrecognizedOpenSearchTypeException.class

20 directories, 146 files

Shadow copy publishing could be invoked by
./gradlew publishShadowPublicationToLocalRepoRepository

Artifacts
$ tree build | grep -vF .class
build
├── classes
│   └── java
│       └── main
│           └── org
│               └── opensearch
│                   └── jdbc
│                       ├── auth
│                       ├── config
│                       ├── internal
│                       │   ├── exceptions
│                       │   ├── results
│                       │   └── util
│                       ├── logging
│                       ├── protocol
│                       │   ├── exceptions
│                       │   └── http
│                       ├── transport
│                       │   └── http
│                       │       └── auth
│                       │           └── aws
│                       └── types
├── libs
│   ├── opensearch-sql-jdbc-2.0.0.0-javadoc.jar
│   ├── opensearch-sql-jdbc-2.0.0.0-sources.jar
│   └── opensearch-sql-jdbc-2.0.0.0.jar
├── publications
│   └── shadow
│       └── pom-default.xml
├── repository
│   └── org
│       └── opensearch
│           └── client
│               └── opensearch-sql-jdbc
│                   ├── 2.0.0.0
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-javadoc.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-javadoc.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-javadoc.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-sources.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-sources.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0-sources.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom.md5
│                   │   └── opensearch-sql-jdbc-2.0.0.0.pom.sha1
│                   ├── maven-metadata.xml
│                   ├── maven-metadata.xml.md5
│                   └── maven-metadata.xml.sha1
├── resources
│   └── main
│       └── META-INF
│           └── services
│               └── java.sql.Driver
└── tmp
    ├── compileJava
    ├── javadocJar
    │   └── MANIFEST.MF
    ├── publishShadowPublicationToLocalRepoRepository
    │   └── org
    │       └── opensearch
    │           └── client
    │               └── opensearch-sql-jdbc
    │                   ├── maven-metadata-remote.xml
    │                   └── resolver-status.properties
    ├── shadowJar
    │   └── MANIFEST.MF
    └── sourcesJar
        └── MANIFEST.MF

44 directories, 169 files

Shadow copy content
$ tree . | grep -vF .class
.
├── META-INF
│   ├── MANIFEST.MF
│   ├── services
│   │   └── java.sql.Driver
│   └── versions
│       └── 9
├── mozilla
│   └── public-suffix-list.txt
└── org
    └── opensearch
        ├── jdbc
        │   ├── auth
        │   ├── config
        │   ├── internal
        │   │   ├── exceptions
        │   │   ├── results
        │   │   └── util
        │   ├── logging
        │   ├── protocol
        │   │   ├── exceptions
        │   │   └── http
        │   ├── transport
        │   │   └── http
        │   │       └── auth
        │   │           └── aws
        │   └── types
        └── sql
            └── jdbc
                └── shadow
                    ├── com
                    │   ├── amazonaws
                    │   │   ...
                    │   └── fasterxml
                    │       └── jackson
                    │           ...
                    ├── org
                    │   ├── apache
                    │   │   ├── commons
                    │   │   │   ├── codec
                    │   │   │   │   ...
                    │   │   │   └── logging
                    │   │   │       └── impl
                    │   │   └── http
                    │   │       ...
                    │   └── joda
                    │       └── time
                    │           ...
                    └── software
                        └── amazon
                            └── ion
                                ...

223 directories, 3902 files

Obsolete references
I removed obsolete references to com.amazonaws.opensearch (replaced by org.opensearch where applicable).
Issues Resolved
Adds JDBC driver publishing workflow
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #957 (6ed5ecd) into 2.x (05d53e7) will decrease coverage by 35.18%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##                2.x     #957       +/-   ##
=============================================
- Coverage     97.94%   62.76%   -35.19%
=============================================
  Files           300       10      -290
  Lines          7746      658     -7088
  Branches        499      119      -380
=============================================
- Hits           7587      413     -7174
- Misses          158      192       +34
- Partials          1       53       +52

Flag
Coverage Δ

query-workbench
62.76% <ø> (?)

sql-engine
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...h/sql/protocol/response/format/ErrorFormatter.java

...nsearch/sql/expression/window/WindowFunctions.java

...l/planner/optimizer/rule/MergeFilterAndFilter.java

.../expression/operator/convert/TypeCastOperator.java

...nsearch/sql/expression/system/SystemFunctions.java

...search/sql/planner/logical/LogicalAggregation.java

...opensearch/executor/OpenSearchExecutionEngine.java

.../org/opensearch/sql/ppl/antlr/PPLSyntaxParser.java

.../opensearch/sql/data/model/ExprTimestampValue.java

...ression/operator/arthmetic/ArithmeticFunction.java

... and 300 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Hi Yuri

Closing - changes and discussion to be tracked in https://github.com/opensearch-project/sql-jdbc/pull/15.

repo: opensearch-project/sql


text:
Adding issue tracking document

issue_tracking.md details both the triage and developer flows when it comes to tracking and resolving issues.
The ISSUE_TEMPLATE.md in this PR is related to this repo. There will be a similar one for osc.org with an additional section for the website. The code repo issue templates will be similar to normal defect info: expected behavior, actual behavior, steps to reproduce with the additional Status section.
Notice the labels added in each repository. The document repos contain slightly different labels, e.g., document repos will not have defect/product-code labels.
This issue addresses part of #51 by creating the template.

@opensecuritycontroller/osc_triage test notification

Thanks so much Hailee.
I am really really going to miss you. Loved working with you.
All the best. Hopefully we will work again in future.
Let us know if you come to Santa Clara.
My best/Manishi
From: Hailee Thomas [mailto:notifications@github.com]
Sent: Friday, June 30, 2017 11:27 AM
To: opensecuritycontroller/community community@noreply.github.com
Cc: Rajora, Manishi manishi.rajora@intel.com; Mention mention@noreply.github.com
Subject: Re: [opensecuritycontroller/community] Adding issue tracking document (#61)
@hmthomaxhttps://github.com/hmthomax requested your review on: opensecuritycontroller/community#61https://github.com/opensecuritycontroller/community/pull/61 Adding issue tracking document.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/opensecuritycontroller/community/pull/61#event-1146440870, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AbRqpRFGZDxhlP8-tPMe3e27cgw9fZ0lks5sJT4NgaJpZM4OJ7p9.

Also, for reference, the following will be the issue_template.md for code repositories:
Expected Behavior
Actual Behavior
Steps to Reproduce
Additional Information (Optional)
Status

[ ] Reproduced
[ ] In Progress
[ ] Pull Request Out
[ ] Validated

repo: opensecuritycontroller/community


text:
Add support of Traffic-routing spec

Describe what this PR does / why we need it
Does this pull request fix one issue?
refrence to issue 16

Describe how you did it
Add xds proto so that OpenSergo can listen the route configuration
Describe how to verify it
Special notes for reviews

and the traffic-routing proto files of opensergo should be added into directory src/main/proto/
Therer is no traffic-routing proto, we just reuse the proto in xds

and the traffic-routing proto files of opensergo should be added into directory src/main/proto/

Actually we just leveraged the original "route configuration" from xDS RDS proto (and there are some extension fields inside it, which may satisfy our requirement)

cc @123liuziming @panxiaojun233 We may need to add some description for our design later.

cc @sczyh30 @123liuziming @panxiaojun233 I will merge it with squash and merge

Nice work. Thanks!

repo: opensergo/opensergo-java-sdk


text:
Inconsistencies between ServiceBindingMetadata and ServiceInstanceMetadata

What is the problem?
ServiceInstanceMetadata if fairly general: only two opaque objects.
    ServiceInstanceMetadata:
      type: object
      properties:
        labels:
          type: object
        attributes:
          type: object

but the ServiceBindingMetadata is more specific.
    ServiceBindingMetadata:
      type: object
      properties:
        expires_at:
          type: string
        renew_before:
          type: string

The parallel in names suggests they are functionally equivalent, but this inconsistency changes how brokers use the two objects.
For example, a broker may want to add a general created_at field to both resource's metadata objects, but can't for the service binding.
Or more holistically, it sends mixed messages about OSB as a prescriptive or general specification.
Who does this affect?
Broker authors and platforms.
Do you have any proposed solutions?
Replace the specific ServiceBindingMetadata fields in favor of attributes and labels, or add the two fields in addition to existing, specific ones. The latter is probably better for backwards compatibility.
    ServiceBindingMetadata:
      type: object
      properties:
        labels:
          type: object
        attributes:
          type: object

or
    ServiceBindingMetadata:
      type: object
      properties:
        expires_at:
          type: string
        renew_before:
          type: string
        labels:
          type: object
        attributes:
          type: object

I'm happy to open an PR, but collecting feedback for now.

repo: openservicebrokerapi/servicebroker

license_source: bigquery


text:
Fix Egress test flag

Enabling egress testing seems to properly enable egress on deployment (by configuration on OSM and also on bookstore infra), but seems bookbuyer/thief are expecting the wrong value:
with ENABLE_EGRESS="true" in run-osm-demo.sh:
Fetching http://github.com
Request Headers: map[]
Identity: n/a
Booksbought: n/a
Server: GitHub.com
Date: Mon, 31 Aug 2020 21:04:18 GMT
Status: 200 OK
ERROR: response code for "http://github.com" is 200;  expected 404       // Response code is correct, expected is wrong (should expect 200 with ENABLE_EGRESS`

Bug description:
Affected area (please mark with X where applicable):

Tests                      [X]

Expected behavior:
No error. expected should be 200

Don't quite understand the issue, but here is where the egress flag is set in the CI:
https://github.com/openservicemesh/osm/blob/main/demo/cmd/common/books.go#L38
While you are it, would you mind fixing this if it is not working as expected?

just synchronized to master and redeployed demo.
.env
export ENABLE_EGRESS=true

...tail-bookthief.sh | grep github
....
ERROR: response code for "https://github.com" is 200;  expected 0
Fetching http://github.com
ERROR: response code for "http://github.com" is 200;  expected 404
Fetching https://github.com
ERROR: response code for "https://github.com" is 200;  expected 0
Fetching http://github.com
ERROR: response code for "http://github.com" is 200;  expected 404
Fetching http://github.com
ERROR: response code for "http://github.com" is 200;  expected 404
Fetching https://github.com
ERROR: response code for "https://github.com" is 200;  expected 0

This is failing for you locally because when you enable egress, you also need to override the EGRESS_EXPECTED_RESPONSE_CODE in .env file as done in the CI.

repo: openservicemesh/osm


text:
MGMT-16841: Failed to add host when infraenv has networkType of DHCP [BACKPORT 2.10]

https://issues.redhat.com/browse/MGMT-16841
BP of https://github.com/openshift-assisted/assisted-installer-ui/pull/2500

/lgtm

authors: jgyselov mareklibra


text:
gpu_operator_deploy_from_operatorhub: update to support v1.9 deployment

and refresh a bit the role and entrypoint.

/test-commit

/test gpu-operator-e2e

/test gpu-operator-e2e
/test-commit

/test test-commit

/test test-commit

/test test-commit
/test gpu-operator-e2e

For reference, when deploying without the OperatorGroup resource (specifying the target namespace), we had this error in the CSV:
  version: 1.9.0-beta
status:
  cleanup: {}
  conditions:
  - lastTransitionTime: "2021-11-25T09:09:55Z"
    lastUpdateTime: "2021-11-25T09:09:55Z"
    message: AllNamespaces InstallModeType not supported, cannot configure to watch
      all namespaces
    phase: Failed
    reason: UnsupportedOperatorGroup
  lastTransitionTime: "2021-11-25T09:09:55Z"
  lastUpdateTime: "2021-11-25T09:09:55Z"
  message: AllNamespaces InstallModeType not supported, cannot configure to watch
    all namespaces
  phase: Failed
  reason: UnsupportedOperatorGroup

/test gpu-operator-e2e

/lgtm

thanks @omertuc
/approve

authors: kpouget omertuc

repo: openshift-psap/ci-artifacts


text:
WIP: RHODS: fix on-demand cluster password

[kube-password] Remove, not used

Not working anymore it seems.

cluster_create_htpasswd_user: new role to create a htpasswd user

testing: ods: clusters: use create_htpasswd_user to set the cluster's password

/test ods-jh-on-single

/test ods-jh-on-single
On Sat, Jul 16, 2022 at 12:07 AM openshift-ci[bot] @.***>
wrote:

@kpouget https://github.com/kpouget: The following test failed, say
/retest to rerun all failed tests or /retest-required to rerun all
mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/ods-jh-on-single d6fa3d0
https://github.com/openshift-psap/ci-artifacts/commit/d6fa3d0f591e90c9ac488f0b1b6d0a5429a10719
link
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/441/pull-ci-openshift-psap-ci-artifacts-master-ods-jh-on-single/1548043446986477568
true /test ods-jh-on-single
Full PR test history
https://prow.ci.openshift.org/pr-history?org=openshift-psap&repo=ci-artifacts&pr=441.
Your PR dashboard
https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:kpouget.
Instructions for interacting with me using PR comments are available here
https://git.k8s.io/community/contributors/guide/pull-requests.md. If
you have questions or suggestions related to my behavior, please file an
issue against the kubernetes/test-infra
https://github.com/kubernetes/test-infra/issues/new?title=Prow issue:
repository. I understand the commands that are listed here
https://go.k8s.io/bot-commands.
—
Reply to this email directly, view it on GitHub
https://github.com/openshift-psap/ci-artifacts/pull/441#issuecomment-1185984358,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABZVQIQAHBGRWGREROSU6SLVUHOKTANCNFSM53WA7J3A
.
You are receiving this because you were mentioned.Message ID:
@.***>

/test ods-jh-on-single

/test ods-jh-on-single


text:
disable concurrent builds in reposync

@sosiouxme @tbielawa

Also adding 4.1 to schedule per Clayton's request

/lgtm

authors: adammhaile tbielawa

repo: openshift/aos-cd-jobs


text:
Make release controller acceptance retry with input

Waiting for release acceptance can retry indefinitely
If a failure occurs, send a slack notification and request human input on how to proceed
Human can retry/skip/abort

Seems legit
/lgtm

authors: jupierce tbielawa


text:
config/v1/types_cluster_version: Add Architecture to DesiredUpdate

field of ClusterVersion spec to request target architecture per OTA-817.

/test integration

/lgtm

/label px-approved
/label qe-approved
/label docs-approved

authors: LalatenduMohanty deads2k jottofar

repo: openshift/api


text:
Adding fake cluster support - OSD-7515

Skip certificate generation for fake clusters (ClusterDeployments labeled with the hive "fake" lable) as per OSD-7515

/lgtm
@sulmer-redhat feel free to throw another change in, pulling the hive labels from the hive constants package, otherwise we can deal with it later

authors: sulmer-redhat yithian

repo: openshift/certman-operator


text:
WRKLDS-1421: Serve metrics server in https

This PR is complementary of this https://github.com/openshift/cli-manager/pull/92 as operator actually applies the resource configurations.

/test ci/prow/e2e-aws-operator

/test e2e-aws-operator

/hold

/hold cancel

no need to run e2e-aws-operator again
/override ci/prow/e2e-aws-operator

repo: openshift/cli-manager-operator


text:
OSASINFRA-3533: Bump gophercloud to v2

This is a new major version of gophercloud.
https://github.com/gophercloud/gophercloud/blob/master/docs/MIGRATING.md

/test e2e-openstack

/test pull-ci-openshift-cloud-network-config-controller-master-e2e-openstack-ovn-serial-e2e-only

/lgtm

/lgtm

/assign danwinship

/retest-required

/lgtm

/lgtm

authors: EmilienM kyrtapz mandre


text:
[OCPCLOUD-1555] Implement provider config abstraction for Azure

This PR implements Azure provider config abstraction based on the AWS one.

/hold
Found that I missed one case and some tests.

/retest-required


text:
Bug 1869613: don't ignore swift errors during container deletion

Currently when we destroy Swift image registry container with BulkDelete, all internal errors are ignored, which can lead to some obscure issues in the future.
This commit parses the response and returns an error if Swift failed to delete some objects.

/test e2e-openstack

/lgtm

/test e2e-openstack

/test e2e-openstack

/test e2e-openstack

/test e2e-openstack

/bugzilla refresh

authors: Fedosin dmage mandre


text:
HAC-731 Cypress ITests fixes

Signed-off-by: Josef Kopriva jkopriva@redhat.com
Fixes
Fixes: https://issues.redhat.com/browse/HAC-731
Description
Fixes for ITests due to last changes in project.
Type of change

[ ] Bugfix
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)

Browser conformance:

[x] Chrome
[ ] Firefox
[ ] Safari
[ ] Edge

/test all

/uncc

Tests are green

/lgtm

/assign @rohitkrai03

authors: jerolimov jkopriva jrichter1

repo: openshift/hac-dev


text:
Refactor e2e test common cleanup

What this PR does / why we need it:
Run the Ensure* validations in the generic create cluster to avoid maintining the list in each individual test.
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[x] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

cc @sjenning

@enxebre The tests in t.Cleanup currently fail because their context is cancelled

@enxebre The tests in t.Cleanup currently fail because their context is cancelled

updated.

/retest-required

authors: alvaroaleman enxebre

repo: openshift/hypershift


text:
HOSTEDCP-1994: Add Secrets Store CSI RBAC Policies for HO and CPO

What this PR does / why we need it:
This PR adds the secrets store CSI driver to the go.mod and adds the needed Secrets Store CSI RBAC policies to the HO and CPO for ARO HCP deployments.
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/retest

/lgtm

authors: bryan-cox csrwng


text:
NO-JIRA: KubeVirt CSI Security and Isolation Documentation

This adds documentation that outlines how KubeVirt CSI achieves storage security and isolation between the guest clusters and the infrastructure cluster.

/lgtm

authors: bryan-cox davidvossel


text:
dispatcher: honor heading args while processing FROM --platform

When evaluating flag arguments to FROM instructions (currently that's just --platform=), also consider the values of ARGs defined in the heading (the part of the file before the first FROM instruction), as we do when evaluating the name of the base image or stage.

LGTM
@flouthoc @giuseppe @TomSweeneyRedHat PTAL

/lgtm

authors: TomSweeneyRedHat nalind rhatdan

repo: openshift/imagebuilder


text:
Added missing unit tests for error parsing module

Description
Added missing unit tests for error parsing module
Type of change

[ ] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library
[ ] Bump-up library or tool used for development (does not change the final image)
[x] Unit tests

/lgtm

/override "Red Hat Konflux / ols-e2e-tests-rosa / lightspeed-service"
/override "Red Hat Konflux / ols-enterprise-contract / lightspeed-service"

authors: JoaoFula tisnik

repo: openshift/lightspeed-service


text:
Use UUID as conversation ID

Description
Use UUID as conversation ID
Type of change

[x] Refactor
[ ] New feature
[ ] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library

My concerns were addressed here.
/lgtm

authors: onmete tisnik


text:
WIP USHIFT-2226: Test Multus - Bridge CNI

/cc

/test ?

/test metal-periodic-test metal-periodic-test-arm

/cc @pliurh @s1061123 @zshi-redhat @dhensel-rh @jogeo

Regarding of CNI config in net-attach-def, it looks good to me.

LGTM

/lgtm

authors: ggiguash pliurh pmtk s1061123

repo: openshift/microshift


text:
Added HOME for CI

Description
Simple fix for permissions errors in CI which is in line with what some other projects do

/lgtm

/lgtm

authors: afflom jharmison-redhat jpower432

repo: openshift/oc-mirror


text:
OCPBUGS-26513: Add Type for CopyImageSchema; Skip graph for IDMS; Fix UpdateService

Description
This PR introduces a Type for CopyImageSchema.
Type is following v1alpha2.ImageType, with possible values:

TypeOCPRelease
TypeOCPReleaseContent
TypeCincinnatiGraph
TypeOperatorCatalog
TypeOperatorBundle
TypeOperatorRelatedImage
TypeGeneric

This enables us to later distinguish and exclude the graph image from the list of images passed to clusterresource Generator interface (responsible for IDMS/ITMS/UpdateService generation) when generating IDMS.
In this PR, we also switch to using the destination reference of the graph image in the updateService.yaml custom resource instead of the source reference (in oc-mirror cache).
Fixes # OCPBUGS-26513
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
With the following imagesetconfig, perform MirrorToDisk, followed by DiskToMirror, and control the IDMS file generated.
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
mirror:
  platform:
    channels:
    - name: stable-4.14
      minVersion: 4.14.2
      maxVersion: 4.14.2
    graph: true

Expected Outcome
IDMS file generated doesn't contain an entry for source: localhost:55000/openshift:
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  creationTimestamp: null
  name: idms-2024-01-09t14-10-17z
spec:
  imageDigestMirrors:
  - mirrors:
    - localhost:5000/openshift-release-dev
    source: quay.io/openshift-release-dev
status: {}

/hold

/unhold

Done pre-merge test , no issue now

/label qe-approved

/lgtm

authors: lmzuccarelli sherine-k zhouying7780


text:
odo project delete considers .odo/config.yaml over the cli args

[kind/bug]

odo considers the .odo/config.yaml's project over the one provided as an are to
odo project delete <project-name>
What versions of software are you using?

Operating System: fedora 29
Output of odo version:

odo v1.0.0-beta1 (1af22f3b)

Server: https://api.engint.openshift.com:443
OpenShift: v3.11.43
Kubernetes: v1.11.0+d4cacc0

How did you run odo exactly?
odo project delete prj

if I try to delete a project prj from a directory with .odo dir then it doesn't delete the prj project but rather tries to delete the actual project from the .odo.
and if I go to a non .odo dir then it deletes the prj project.

/kind bug
/priority medium
/milestone 1.0

On similar lines, odo project set doesn't work as expected either:
$ odo project get
The current project is: myproject

$ odo project list
ACTIVE     NAME
           helloworld
*          myproject

$ odo project set helloworld
Switched to project : helloworld

$ odo project get
The current project is: myproject

$ cd ..

$ odo project get
The current project is: helloworld

Project deletion issue mentioned in https://github.com/openshift/odo/issues/1715#issue-442604070 is not reproducible at my side. I'm trying out the steps on CentOS system with below versions of odo and oc:
$ odo version
odo v1.0.0-beta1 (e5b00a3)

Server: https://192.168.42.121:8443
Kubernetes: v1.11.0+d4cacc0

$ oc version
oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Steps tried by me:
$ cat .odo/config.yaml
kind: LocalConfig
apiversion: odo.openshift.io/v1alpha1
ComponentSettings:
  Type: python
  SourceLocation: ./
  SourceType: local
  Application: app
  Project: myproject
  Name: python-foel
  Url:
  - Name: python-foel-8080
    Port: 8080

$ odo project list
ACTIVE     NAME
*          myproject
           newproject
           newprojet

$ odo project delete newproject
? Are you sure you want to delete project newproject Yes
 ✓  Deleting project newproject
Deleted project : newproject

$ odo project list
ACTIVE     NAME
*          myproject
           newprojet

Did you want to take this up @dharmit ? If not, I can have a look at it :+1:

@cdrage yes I did but wanted to reproduce the issue first. I can self-assign it so that it doesn't show up as unassigned

/shrug

This is still not reproducible @girishramnani
$ odo config view
ENVIRONMENT VARIABLES
------------------------------------------------
NAME    VALUE
FOO     BAZ

COMPONENT SETTINGS
------------------------------------------------
PARAMETER         CURRENT_VALUE
Type              nodejs
Application       app
Project           myproject
SourceType        local
Ref
SourceLocation    ./
Ports
Name              nodejs-wmea
MinMemory
MaxMemory
Ignore
MinCPU
MaxCPU

$ odo project create newproject
 ✓  New project created and now using project : newproject

$ odo project list
ACTIVE     NAME
*          myproject
           newproject

$ odo project delete newproject
? Are you sure you want to delete project newproject Yes
 ✓  Deleting project newproject [6s]
Deleted project : newproject

$ odo project list
ACTIVE     NAME
*          myproject

Ping @girishramnani maybe we can close this.

Unable to reproduce, it works as it supposed to
▶ cat .odo/config.yaml
kind: LocalConfig
apiversion: odo.openshift.io/v1alpha1
ComponentSettings:
  Type: nodejs
  SourceLocation: ./
  SourceType: local
  Application: app
  Project: myproject
  Name: mynode
  Envs:
  - Name: FOO
    Value: BAR
  Url:
  - Name: mynode-8080
    Port: 8080

tmp/odo-examples/nodejs-ex  master ✗                                                                             266d ⚑ ◒
▶ odo project delete asdf
? Are you sure you want to delete project asdf Yes
 ✓  Deleting project asdf [6s]
Deleted project : asdf

Since we're unable to reproduce this and @girishramnani hasn't replied back, let's close this for now :+1:

authors: cdrage dharmit girishramnani kadel

repo: openshift/odo


text:
Validate config values (#1549)

What is the purpose of this change? What does it change?
Checks if CPU parameter is able to be parsed to float64
Checks if Ports parameter compiles with a regex: [0-9]{1,5}\/(TCP|UDP)
Checks if Name parameter is longer than 0
Checks Memory, MaxMemory, MinMemory values if they comply to github.com/openshift/odo/pkg/util Parse method. Memory can't be set to 0 as it would set MaxMemory to 0 which does not make sense to me.
Was the change discussed in an issue?
fixes #1549
How to test changes?
The commit also provides tests.

I would like to use this to check the MaxMemory and MinMemory parameters. Thoughts?

Added Ports check. Regex checks for one to five numbers at the beginning + dash + either TCP or UDP strings.

@drichtarik tests are failing

also if you feel that this is ready to review then please remove the work-in-progress and move it from Draft PR to actual PR @drichtarik

@drichtarik test are failing due to the mismatch value of memory in the config file. The memory value is converted to bytes (for example 100M -> 104857600 bytes) whatever the value is passed through the set command. Probably you need to look into that conversion logic.
I think the conversion is controllable/fixed thorough our odo code as the value is not coming from os Level.

@amitkrout I hope I fixed it correctly.

/ok-to-test

/retest

/retest

/hold
till GA

/hold cancel

It should not be possible to set a Type that's not listed in the output of odo catalog list components (refer the Name column):
$ odo config set Type noschtype
? Type is already set. Do you want to override it in the config Yes
Local config was successfully updated.
Run `odo push --config` command to apply changes to the cluster.

$ odo catalog list components
Odo Supported OpenShift Components:
NAME       PROJECT       TAGS
nodejs     openshift     10,8,8-RHOAR,latest
java       myproject     latest

Odo Unsupported OpenShift Components:
NAME        PROJECT       TAGS
dotnet      openshift     2.0,latest
httpd       openshift     2.4,latest
nginx       openshift     1.10,1.12,1.8,latest
nodejs      openshift     6
perl        openshift     5.24,5.26,latest
php         openshift     7.0,7.1,latest
python      openshift     2.7,3.5,3.6,latest
ruby        openshift     2.3,2.4,2.5,latest
wildfly     openshift     10.0,10.1,11.0,12.0,13.0,8.1,9.0,latest

I think we could change the error response here to indicate that the user has specified invalid value:
$ odo config set MinMemory -f 50L
 ✗  Unit parse error: invalid fields [50 50  ]

$ odo config set MaxMemory -f 50L
 ✗  Unit parse error: invalid fields [50 50  ]

$ odo config set Memory -f 50L
 ✗  Unit parse error: invalid fields [50 50

@drichtarik can you rebase please

closing this PR due to inactivity.

authors: amitkrout dharmit drichtarik girishramnani mohammedzee1000


text:
Modifies WaitForDeploymentRollout() to collect events.

What type of PR is this?
/kind flake
What does does this PR do / why we need it:
Modifies WaitForDeploymentRollout() to collect events. It also uses WaitAndGetPodWithEvents() for getting the pod in the adapter package.
Which issue(s) this PR fixes:
Fixes NA
PR acceptance criteria:

[X] Unit test

[ ] Integration test

[ ] Documentation

[ ] I have read the test guidelines

How to test changes / Special notes to the reviewer:

Tests should pass.
If failures related to https://github.com/openshift/odo/issues/3256 occur, collected events should be printed.

This is great! I was just trying to untangle the WaitAndGetPod vs WaitAndGetPodWithEvents mess.
Could we completely remove WaitAndGetPod and replace it with WaitAndGetPodWithEvents?

[odo] Push successful

1952

[odo] ✗ Waiting for build to finish [6m]

1953

[odo] ✗ Failed to create component with name uberjar-git-test. Please use `odo config view` to view settings used to create component. Error: failed to build component with args {Name:uberjar-git-test SourcePath:https://github.com/geoand/javalin-helloworld SourceRef:master SourceType:git ImageName:java:8 EnvVars:[DEBUG_PORT=5858] Ports:[8080/TCP 8443/TCP 8778/TCP] Resources:&ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},} ApplicationName:app Wait:true StorageToBeMounted:map[] StdOut:0xc0000fe008}: unable to build uberjar-git-test-app-1, error: : timeout waiting for build uberjar-git-test-app-1 to start

/retest

Error creating VPC: VpcLimitExceeded: The maximum number of VPCs has been reached."

/retest

Could we completely remove WaitAndGetPod and replace it with WaitAndGetPodWithEvents?

WaitAndGetPod() was not being used anywhere after the initial changes in the PR. I have removed the function in the latest commit.

/approve

[odo] Push successful

1952

[odo] ✗ Waiting for build to finish [6m]

1953

[odo] ✗ Failed to create component with name uberjar-git-test. Please use `odo config view` to view settings used to create component. Error: failed to build component with args {Name:uberjar-git-test SourcePath:https://github.com/geoand/javalin-helloworld SourceRef:master SourceType:git ImageName:java:8 EnvVars:[DEBUG_PORT=5858] Ports:[8080/TCP 8443/TCP 8778/TCP] Resources:&ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{},} ApplicationName:app Wait:true StorageToBeMounted:map[] StdOut:0xc0000fe008}: unable to build uberjar-git-test-app-1, error: : timeout waiting for build uberjar-git-test-app-1 to start

/retest

Should we have been able to see what's proposed in this PR when we hit this failure on CI?

Should we have been able to see what's proposed in this PR when we hit this failure on CI?

It was a build failure and is not related to https://github.com/openshift/odo/issues/3256.

authors: dharmit kadel mik-dass


text:
different used repositories leads in inability to apply update

Not sure, whether I did everything right, but with okd 4.4 (preview2) deployed I am now trying to force update to the newer image:

neither channel offers update (so far so good)
trying to apply desiredUpdate to the ClusterVersion operator:
initial deployment claims to be installed from quay.io repo

- completionTime: '2020-01-30T17:29:08Z'
      image: >-
        quay.io/openshift/okd@sha256:df6caca5d103f75909fd7cdbf294991023b9fc79c137dbb61754000196f97b8e
      startedTime: '2020-01-30T17:09:04Z'
      state: Completed

creating spec.desiredUpdate, which will refer quay.io with newer sha image from https://origin-release.svc.ci.openshift.org/releasestream/4.4.0-0.okd/release/4.4.0-0.okd-2020-01-31-123546 leads to image not found
using registry.svc.ci.openshift.org/origin/release@sha256:12739a1e37e13b8557039c6eec1e4212e3c5532deb283419e3cac9a3783e5393 works and starts update
update stucks on the last step for machine-operator, which degrades, since in crio.conf "quay.io/openshift/okd-content@sha256:2f68508143d397e98e2aa53967d28357263a10d0832d611bee1656b3a32ecbc2 != registry.svc.ci.openshift.org/origin/4.4-2020-02-03-013019@sha256:2f68508143d397e98e2aa53967d28357263a10d0832d611bee1656b3a32ecbc2"

Result: can not update to the newer release, cluster degraded

neither channel offers update (so far so good)

Upgrades via channels are out of scope for now, at least until Beta

trying to apply desiredUpdate to the ClusterVersion operator:

Does it work with oc adm upgrade --force --force --allow-explicit-upgrade --to-image=<release pull spec>? Note that CI registry may be unreliable sometimes so its recommended to mirror the image to quay / your registry - docs

not sure it will help, but I give it a try (actually this command also results in very similar spec of the ClusterConfig operator. I need now somehow to overcome downgraded machine-config operator.
I think the problem is that machine-config operator generates/uses some registry name, which does not match to what some other component, or even itself, expects/does (generated machine-config contains pause_image ref to registry.svc.co.openshift.org (what was initially given as target), while MCO further step's expectation is quay.io).
So, MCO applies update to the target state and immediately exists literally complaining about this particular inconsistency.
rendered config and applied file contain:
pause_image = "registry.svc.ci.openshift.org/origin/4.4-2020-02-03-013019@sha256:2f68508143d397e98e2aa53967d28357263a10d0832d611bee1656b3a32ecbc2"

error messages from MCD:
I0203 11:22:08.302318 2059171 daemon.go:732] Current config: rendered-worker-e0186732891c0943f9f1d3a266c90c34
I0203 11:22:08.302350 2059171 daemon.go:733] Desired config: rendered-worker-8e62143cdd82ff4cd2fd6a6293b6e669
I0203 11:22:08.308816 2059171 daemon.go:423] Detected a login session before the daemon took over on first boot
I0203 11:22:08.308840 2059171 daemon.go:424] Applying annotation: machineconfiguration.openshift.io/ssh
I0203 11:22:08.321357 2059171 daemon.go:959] Validating against current config rendered-worker-e0186732891c0943f9f1d3a266c90c34
E0203 11:22:08.322104 2059171 daemon.go:1351] content mismatch for file /etc/crio/crio.conf: # The CRI-O configuration file specifies all of the available configuration

And rolling back manually the crio.conf content on the target system brings me to https://github.com/openshift/machine-config-operator/issues/1378, which most likely is the real cause here

well yes, after reading more precisely - https://github.com/openshift/machine-config-operator/issues/1378 is exactly the cause

Lets keep this bug so it would be used as an OKD tracking bug

Tried a few updates recently and didn't hit it - is it still occurring?

authors: gtema vrutkovs

repo: openshift/okd


text:
Pass --env variables to the build container

This fixes a bug introduced in https://github.com/openshift/source-to-image/pull/501
@bparees @php-coder

Thank you for finding and fixing it! LGTM.

Fixed the error message, good catch :)

lgtm [merge]

authors: bparees mnagy php-coder

repo: openshift/source-to-image


text:
PR check jobs for saas-openshiftio running e2e smoke test

Please DO NOT merge until @joshuawilson reviews the change.

@joshuawilson does this looks good to you ?

@joshuawilson yes, this is PR check for repository saas-openshiftio, i.e. before promoting code to production.
You are right that e2e tests don't pass 100% time but they are failing because of bugs in product, we need to fix these bugs, this PR check will make them more visible

[test]

@ppitonak @joshuawilson are we OK to merge this?

OK from my side, depends only on Joshua.

@jmelis since this PR check won't be enforced, the risk of breaking something is very low, feel free to merge this PR

authors: jmelis ppitonak riuvshin

repo: openshiftio/openshiftio-cico-jobs


text:
Clarify license/copyright in README.md

We may want to add a creative commons license for the content.

Since we migrated OSCU over to medium, closing this to clear my assigned open issues

authors: trabianmatt treeme

license: mit


text:
🛑 Weblate is down

In defc5f8, Weblate (https://translate.opensourcepos.org) was down:

HTTP code: 500
Response time: 1984 ms

Resolved: Weblate is back up in beaf2e3 after 2 hours, 38 minutes.

repo: opensourcepos/upptime

license: MIT


text:
🛑 Weblate is down

In 0eccbc4, Weblate (https://translate.opensourcepos.org) was down:

HTTP code: 500
Response time: 1582 ms

Resolved: Weblate is back up in a0eb72f after 12 minutes.


text:
No documentation for Usage

Why is it that there is no documentation for this project. I think this is a very good project. I remember the previous for box had a little documentation.

What about https://github.com/openspout/openspout/tree/4.x/docs?

authors: Slamdunk otengkwame

repo: openspout/openspout