text:
[Proposal] Upgrade Testing Framework UX

NOTE: Development of the proposed Framework will be pursued in another repo; posting the proposal in this one for visibility
Objective
The purpose of this doc is to outline key user experiences for the Upgrade Testing Framework so the design is driven from a customer-first perspective.  The designs represented in the doc are speculative and open to change; think of them as proposals rather than assertions.
Terminology

Application Developer: individual that uses OpenSearch as a search/logs engine in their system, but doesn't necessarily manage it
Cluster Admin: individual responsible for the maintenance and/or support of one or more ElasticSearch/OpenSearch Clusters
Committed OpenSearch Developer: individual who contributes code to the OpenSearch Project, either its core repositories or the constellation of repositories connected to them
Casual OpenSearch Developer: individual who might be willing to contribute code to the OpenSearch Project, assuming that the process is easy
User: Either a Cluster Admin or a Committed/Casual OpenSearch Developer; an Application Developer might wear any of those hats as well
Upgrade: moving from one version of ElasticSearch/OpenSearch to another without necessarily changing the underlying hardware hosting the cluster
Migration: moving an ElasticSearch/OpenSearch cluster from one underlying hosting solution to another without necessarily changing the version of the cluster

Background
The Upgrade Testing Framework is a proposed project (https://github.com/opensearch-project/OpenSearch/issues/5167) for the OpenSearch Project.  It has the following goals:

Provide OpenSearch Developers a way to easily test the outcomes of more complex, real-world upgrades than the existing backwards compatibility (BWC) tests allow and ideally replace those existing tests.  It intends to address upgrades to the core engine and plugins, along with accompanying data, configuration and metadata, and handle N+M version upgrades instead of just N+1.
Perform upgrades during testing in the same manner as Users execute them “in the wild”.
Provide Cluster Admins mechanisms to predict the outcome of an upgrade before executing one and enable them to make a representative simulacra of their cluster on their laptop or other development machine in order to test out an upgrade in a safe environment.
Promote and encapsulate validation mechanisms that Cluster Admins can use to determine whether a upgraded candidate cluster is ready for production traffic.
Serve as a central repo to iteratively capture the community’s full understanding of differences between versions and continuously assert that each difference remains as expected.  This will enable the community to formally understand, document, and ideally resolve these differences.

Setup
The Upgrade Testing Framework will just be a Python project in GitHub.  To set it up on a laptop, developer desktop, or CI/CD fleet, just clone it to the host and execute the command-line utility in the downloaded source.  The Framework will use system-supplied copies of Python and Docker to perform its work.
Use Cases
Core Use Cases
These use-cases are core features that are necessary for any “initial release”.

UC1 - I want to test an upgrade using default data/tests
UC2 - I want to simulate an upgrade using custom data/tests
UC3 - I want to know if an upgraded cluster is ready for production traffic

Incremental Use Cases
These use-cases add value, but will be prioritized after the Core Use Cases and based on user feedback/business requirements.

UC4 - I want to run a bunch of different tests against a bunch of different clusters
UC5 - I want the framework to auto-generate test config based on my real cluster
UC6 - I want a predictive assessment of the issues I’ll encounter when upgrading my real cluster

UC1 - Use Case - I want to test an upgrade using default data/tests
Step 1: Define task configuration
Before executing a upgrade test run, the User will need to provide some configuration so the framework knows what to do. It is proposed that for the initial release, we do that via a configuration file:
{
    "cluster_def": {
        "start": {
            "dockerfile": "./path/to/Dockerfile1",
            "tags": [
                "ElasticSearch_7_10_2"
            ]
        },
        "end": {
            "dockerfile": "./path/to/Dockerfile2",
            "tags": [
                "OpenSearch_2_3"
            ]
        },
        "desired_nodes": 2
    },
    "upgrade_def": {
        "style": "snapshot-restore"
    },
    "test_def": {
        "default_tests": {
            "dataset_file": "./data/los_datos_del_diablo.json",
            "test_files_start": [
                "./tests/default_tests_start.py",
            ],
            "test_files_end": [
                "./tests/default_tests_end.py",
            ]
        },
        "default_cluster_comparison_tests": {
            "test_files_end": [
                "./tests/default_cluster_comparison_tests.py",
            ]
        }
    }
}

A few notes:

The Framework will ship with a library of test_config.json files and their associated Dockerfiles, data files, and test scripts that encompass the current understanding of the differences between versions
Users can easily supply whatever Dockerfiles they want to represent the nodes in their cluster
We will likely want the cluster setup, such as number and type of nodes, to be independently configurable between the starting and ending clusters, but for the initial cut we’ll avoid that complexity
The “tags” are probably unnecessary, but they are intended to highlight the fact we’ll want to down-select which expectations/data/tests to apply to a given cluster based on its configuration later on.  We’ll likely solve this problem by directly interrogating the clusters after they are stood up.
The “test_def” section is where the framework will learn about the data to load into the cluster and the tests to execute, and when to perform those actions.  It seems like data must necessarily be paired with tests (what’s the point of adding data if you don’t have tests that use it?); tests do not necessarily require new data.

Step 2: Invoke the framework
The User then invokes the framework CLI to kick off the upgrade.  The CLI gives the user feedback throughout the upgrade process:
> ./utf test --file test_config.json
[bootstrap] Building Dockerfile "./path/to/Dockerfile1"...
[bootstrap] Building Dockerfile "./path/to/Dockerfile2"...
[bootstrap] Checking for conflicting Docker resources from previous runs...
[bootstrap] Found Docker resources from previous run: 6 containers, 1 network
[bootstrap] Do you wish to remove the conflicting resources and continue? (y/n)
> y
[bootstrap] Removing conflicting resources...
[initialize_start] Spinning up (2) nodes for the starting cluster...
[initialize_start] Starting Node "start.1"
[initialize_start] Starting Node "start.2"
[initialize_start] Cluster stabilized, Node "start.2" elected master
[test_start] Beginning upload of test data...
[test_start] Upload of test index "los_datos_del_diablo" started
[test_start] Upload of test index "los_datos_del_diablo" completed
[test_start] Uploaded all test data
[test_start] Beginning analysis of cluster's starting state...
[test_start] Running tests "./tests/default_tests_start.py"...
[test_start] Cluster met all expectations for starting state
[execute_upgrade] Beginning upgrade of cluster...
[execute_upgrade] Upgrade style: Snapshot-Restore
[execute_upgrade] Performing snapshot on source cluster...
[execute_upgrade] Snapshot "test_snapshot_2022_11_21" stored at /home/blzbub/snapshots
[execute_upgrade] Starting Node "end.1"
[execute_upgrade] Starting Node "end.2"
[execute_upgrade] Target cluster stabilized, Node "end.1" elected master
[execute_upgrade] Restoring snapshot into target cluster...
[execute_upgrade] Snapshot restored
[execute_upgrade] Finished upgrade of cluster
[test_end] Beginning analysis of target cluster's ending state...
[test_end] Running tests "./tests/default_tests_end.py"...
[test_end] Running tests "./tests/default_cluster_comparison_tests.py"...
[test_end] Target cluster did NOT meet all expectations for ending state; see final report for details

==== FINAL RESULTS ====
Source cluster did meet all starting expections
Target cluster did NOT meet all ending expections, see final report

==== LOGS AND REPORTS ====
Final report: ./reports/report.2022_11_11_08_10_31.txt
Full Logs: ./logs/logs.2022_11_11_08_10_31.txt
State File: ./state_file

==== CLUSTER DETAILS ====
Source cluster is STILL RUNNING (127.0.0.1:9200)
Target cluster is STILL RUNNING (127.0.0.1:9201)

Some notes:

The Cluster is left running at the end of the test run by default so that the user can poke at it if they want to, so this behavior should be controlled via command-line flags and the default can easily be flipped.
The CLI should present only the most important, highest-level details to the user via stdout unless something goes wrong.  The full logs of every under-the-hood operation are dumped to a log file the user can explore if interested or they have a need.
The overall flow of the testing is broken up into stages marked with prefixed brackets ([test_start]). These should ideally be idempotent break-points where the application can be stopped/started. This is made possible by saving the application state to a state-file so that it can pick up where it stopped. This enables the User to do things like stop the application immediately before the upgrade begins and poke around the running containers for the starting cluster for debugging purposes. It’s pretty easy to set up and maintain this, and it adds substantial value in comparison to the additional complexity.
It is expected that the “Test” steps ([test_start], [test_end]) will be to invoke separate Python executables to run unit tests against the starting/ending cluster(s).  Separating the test/analysis layer from the CLI orchestration layer makes it possible to do things like have users run the final validation steps against “real” clusters instead of just ones on their laptop, etc.
The report will be something that outlines specifically all the tests performed and their results.  Ideally, we’d present it in a user-friendly way like an HTML form, but that’s probably an optimization for later as long as we have something functional initially.  However, the initial cut will need to be consumable by non-Developers, so we’ll want to do something more sophisticated than just dump raw unit test results.

Step 3: Review the Results
For the initial cut of the framework, the results will likely just be the output of a standard code testing framework (e.g. pytest).  If this doesn’t have the clarity we desire, then we can assemble our own report w/ custom code.
In terms of what types of things the tests are expected to cover and the report is expected to surface, consider the following scenarios:

Things we expect to work/be true (e.g. same number of documents before/after the upgrade)
Updates in OpenSearch core or plugin code resulting in translations between the source and target versions (configs, data, metadata “morph” during the upgrade)
Changes in compatibility of the system under test - features removed, changed, bugs introduced or fixed, etc
Bugs in the validation framework’s tests themselves

UC2 - Use Case - I want to test an upgrade using custom data/tests
Step 1: Update task configuration
Users can supply custom data (e.g. data that is unique to their setup and a part of the public repo) using the configuration file.  Custom data necessarily implies custom tests, so the two will need to be configured together.
As long as the custom data and tests conform to the same interface as the “default” ones, the framework will ingest and execute them without distinction between what’s “custom” or not.  The default data/tests serve as an example to base custom versions on, in addition to the documentation we write.
# test_config_custom.json
{
    "cluster_def": {
        "start": {
            "dockerfile": "./path/to/Dockerfile1",
            "tags": [
                "ElasticSearch_7_10_2"
            ]
        },
        "end": {
            "dockerfile": "./path/to/Dockerfile2",
            "tags": [
                "OpenSearch_2_3"
            ]
        },
        "desired_nodes": 2
    },
    "upgrade_def": {
        "style": "snapshot-restore"
    },
    "test_def": {
        "default_tests": {
            "dataset_file": "./data/los_datos_del_diablo.json",
            "test_files_start": [
                "./tests/default_tests_start.py",
            ],
            "test_files_end": [
                "./tests/default_tests_end.py",
            ]
        },
        "default_cluster_comparison_tests": {
            "test_files_end": [
                "./tests/default_cluster_comparison_tests.py",
            ]
        },
        "custom_tests": {
            "dataset_file": "./my_custom_data.json",
            "test_files_start": [
                "./my_custom_tests_1.py",
            ],
            "test_files_end": [
                "./my_custom_tests_2.py",
            ]
        }
    }
}

An existing tool, Elasticdump (see here), provides a way to export indices from a cluster.  It seems reasonable to either base our data format around its format or at the very least accept it.

Step 2: Invoke the framework
As normal, the User would just invoke the Framework with their configuration file and the Framework will do the rest:
> ./usf test --file test_config_custom.json
[bootstrap] Building Dockerfile "./path/to/Dockerfile1"...
.
. # Intermediate steps skipped for clarity
.
[test_start] Beginning upload of test data...
[test_start] Upload of test index "los_datos_del_diablo" started
[test_start] Upload of test index "los_datos_del_diablo" completed
[test_start] Upload of test index "my_custom_data"
[test_start] Upload of test index "my_custom_data" completed
[test_start] Uploaded all test data
[test_start] Beginning analysis of cluster's starting state...
[test_start] Running tests "./tests/default_tests_start.py"...
[test_start] Running tests "./my_custom_tests_1.py"
[test_start] Cluster met all expectations for starting state
[execute_upgrade] Beginning upgrade of cluster...
.
. # Intermediate steps skipped for clarity
.
[test_end] Beginning analysis of cluster's ending state...
[test_end] Running tests "./test/default_tests_end.py"
[test_end] Running tests "./test/default_cluster_comparison_tests.py"
[test_end] Running tests "./my_custom_tests_2.py"
[test_end] Cluster did NOT meet all expectations for ending state; see final report for details
.
. # And so on
.

UC3 - Use Case - I want to know if an upgraded cluster is ready for production traffic
While the Upgrade Testing Framework, as a whole, is focused on providing Users a way to test the full upgrade process on their laptop, developer desktop, etc against a Dockerized test cluster, the validation tests themselves should ideally not care whether they are being pointed at a “test” cluster or a “real” cluster, and be equally applicable to either.  For that reason, we will structure the validation tests as separate executable scripts that the Framework will invoke.  This enables the same validation tests to be used by a Cluster Admin to help determine if an upgraded cluster is ready for production traffic.
Presumably, a Cluster Admin would not need to run the validation tests against an upgraded cluster that is currently serving production traffic, as they could use their existing alarms/metrics for that.  This is an assumption though and open to re-evaluation.
Step 1: Duplicate production cluster
The Cluster Admin would create a duplicate of their production cluster, perhaps using cross-cluster replication.
Step 2: Perform manual upgrade of duplicate
The Cluster Admin would use the OpenSearch documentation to perform an upgrade of the duplicate cluster, perhaps using snapshot/restore.
Step 3: Execute the validation tests against the clusters
Behind the scenes of the [test_end] step, all the Framework is doing is making shell invocations of executable Python scripts that contain tests to be performed against a pair of (ip address, port) tuples.  As long as there’s a network path to the source/target clusters, the tests should run the same.
> ./test/default_cluster_comparison_tests.py --source_cluster 10.0.0.2:9200 --target_cluster 10.0.1.2:9200 --auth root:i_am_g_root
Running tests against source (10.0.0.2:9200) and target (10.0.1.2:9200)
TEST 1: Same number of documents in index...
TEST 1: Passed
.
.
.

This is obviously easiest for groups of tests that can directly compare the contents of two clusters without requiring a setup step of first uploading specific data that the tests are intrinsically tied to.  As a result, there is a need to carefully separate groups of tests based on their setup requirements (e.g. avoid adverse “mingling”).
However, in principal there’s no reason why user couldn’t upload test data to their duplicate (pre-upgrade) cluster, perform the upgrade on the duplicate, and then run the validation tests tied with the test data... but would they really want to?  Currently unclear.

UC4 - Use Case - I want to run a bunch of different tests against a bunch of different clusters
The above discussion has so far assumed that all the tests we’d want to run could be performed using a single cluster/upgrade process.  However, it’s likely we’ll have sets of tests that we want to run that are mutually incompatible, or at the very least inconvenient to combine.  The proposed Framework should be able to handle that simply, but to understand how we need to go into implementation details.
As has been presented thus far, the Framework is built around the idea of a top level run-loop (a Runner) which executes a sequence of steps (Framework Steps) in order.  The Runner will be a self-contained object which handles its own setup/teardown, and accepts a list of Framework Steps to perform as well as a test_config.json as some of its constructor arguments.  The Runner is agnostic to the specific Framework Steps, and the specific test_config.json.    This means that a full library of test_config.json’s, along with accompanying test data and validation tests, can be created and executed with a Runner in series.  One trivial way to do this would be to use a Python unit testing framework (unittest, Pytest, etc) to create individual tests that each represent the invocation of the runner on a specific test_config.json.  The tests will likely need to be executed in series due to host resource constraints if only a single host is available, but if multiple hosts are available they can be executed in parallel using built-in mechanisms for running specific tests.
UC5 - Use Case - I want the framework to auto-generate test config based on my real cluster
In this scenario, the user has an existing “real” cluster that they want to convert into a test_config.json, likely alongside representative test data and other configuration.  The user would then be able to perform a test upgrade against a high-fidelity simulacra of their cluster from the comfort of a developer laptop/desktop.
While this use-case needs further design consideration, at a high level it seems possible to write a tool that can be pointed at a “real” cluster and extract the relevant details using the REST API (data types, plugin configuration, cluster configuration, etc).  This can then be converted into a test configuration that feeds into the Upgrade Testing Framework like any other.
As with creating useful test and data sets for the other use-cases, community involvement will be crucial for determining what specific items to look for and capture when auto-generating test config.
NOTE - The mechanism for scanning a cluster and classifying its configuration appears shared w/ the ability to make predictive assessments
UC6 - Use Case - I want a predictive assessment of the issues I’ll encounter when upgrading my real cluster
In this scenario, the user wants an assessment of what would happen if they were to perform an upgrade on an extant, “real” cluster to some proposed version of ElasticSearch/OpenSearch - WITHOUT having to actually execute a real or test upgrade.
This use-case also needs further design consideration, but it seems possible to write a tool that connects to a “real” cluster, uses the REST API to classify its relevant details, then uses the knowledge base of “expectations” captured in the validation tests to make a prediction of the types of issues likely to be encountered.  At that point, a report could be generated and supplied to the user.
The complexity here appears to lie in how to model the “expectations” used by those validation tests in a way that they can be consumed by the assessment tool.
NOTE - The mechanism for scanning a cluster and classifying its configuration appears shared w/ auto-generating test config

Removed section on the "I want to make a test cluster without performing an upgrade".  Still planning on fulfilling it, but highlighting it here in this doc just distracts from other, more important use-cases


text:
[BUG] During rolling upgrade, OpenSearch node cannot handling the exception right.

Describe the bug
We found this issue during rolling upgrade test of IM plugin from opendistro cluster to OpenSearch cluster. We have security enabled.
The issue happens when it's a mixed cluster state, for example, 2 opendistro nodes node-1 and node-2, one OpenSearch node node-3.
When you hit node-3 with request like curl "https://localhost:9202/nonexist" -u admin:admin --insecure (simple get request to a non existent index), this request is not returned. And with exception in the log:

Exception, click to see
[2021-06-28T16:23:51,104][ERROR][o.o.r.a.RestResponseListener] [node-3] failed to send failure response
java.lang.StringIndexOutOfBoundsException: String index out of range: -3
	at java.lang.String.substring(String.java:1850) ~[?:?]
	at org.opensearch.OpenSearchException.innerToXContent(OpenSearchException.java:351) ~[opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.OpenSearchException.toXContent(OpenSearchException.java:339) ~[opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.OpenSearchException.generateFailureXContent(OpenSearchException.java:593) ~[opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.rest.BytesRestResponse.build(BytesRestResponse.java:162) ~[opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:123) ~[opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:106) ~[opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.rest.action.RestActionListener.onFailure(RestActionListener.java:71) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:111) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.handleException(TransportMasterNodeAction.java:201) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.TransportService$6.handleException(TransportService.java:662) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:294) [opensearch-security-1.0.0.0-rc1.jar:1.0.0.0-rc1]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1203) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:290) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:239) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.InboundHandler.handleException(InboundHandler.java:288) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:280) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:144) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:713) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:155) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:130) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:95) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:87) [transport-netty4-client-1.0.0-rc1.jar:1.0.0-rc1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1533) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]
	Suppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder
		at org.opensearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1030) ~[opensearch-x-content-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:122) ~[opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:106) ~[opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.rest.action.RestActionListener.onFailure(RestActionListener.java:71) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:111) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$1.handleException(TransportMasterNodeAction.java:201) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.TransportService$6.handleException(TransportService.java:662) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:294) [opensearch-security-1.0.0.0-rc1.jar:1.0.0.0-rc1]
		at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1203) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:290) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:239) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.InboundHandler.handleException(InboundHandler.java:288) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:280) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:144) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:102) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:713) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:155) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:130) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:95) [opensearch-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:87) [transport-netty4-client-1.0.0-rc1.jar:1.0.0-rc1]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1533) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
		at java.lang.Thread.run(Thread.java:835) [?:?]
	Caused by: java.io.IOException: Unclosed object or array found
		at org.opensearch.common.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:469) ~[opensearch-x-content-1.0.0-rc1.jar:1.0.0-rc1]
		at org.opensearch.common.xcontent.XContentBuilder.close(XContentBuilder.java:1028) ~[opensearch-x-content-1.0.0-rc1.jar:1.0.0-rc1]
		... 47 more
	Suppressed: org.opensearch.transport.RemoteTransportException: [node-1][127.0.0.1:9300][indices:admin/get]
	Caused by: org.opensearch.index.IndexNotFoundException: no such index [example1]
		at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:945) ~[?:?]
		at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:882) ~[?:?]
		at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:838) ~[?:?]
		at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:217) ~[?:?]
		at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:197) ~[?:?]
		at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:90) ~[?:?]
		at org.elasticsearch.action.support.master.info.TransportClusterInfoAction.checkBlock(TransportClusterInfoAction.java:52) ~[?:?]
		at org.elasticsearch.action.support.master.info.TransportClusterInfoAction.checkBlock(TransportClusterInfoAction.java:34) ~[?:?]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:138) ~[?:?]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:115) ~[?:?]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:59) ~[?:?]
		at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[?:?]
		at com.amazon.opendistroforelasticsearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:124) ~[?:?]
		at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[?:?]
		at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityFilter.apply0(OpenDistroSecurityFilter.java:311) ~[?:?]
		at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityFilter.apply(OpenDistroSecurityFilter.java:151) ~[?:?]
		at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[?:?]
		at com.amazon.opendistro.elasticsearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77) ~[?:?]
		at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[?:?]
		at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) ~[?:?]
		at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64) ~[?:?]
		at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:60) ~[?:?]
		at com.amazon.opendistro.elasticsearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:49) ~[?:?]
		at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceivedDecorate(OpenDistroSecuritySSLRequestHandler.java:182) ~[?:?]
		at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityRequestHandler.messageReceivedDecorate(OpenDistroSecurityRequestHandler.java:293) ~[?:?]
		at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceived(OpenDistroSecuritySSLRequestHandler.java:142) ~[?:?]
		at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin$7$1.messageReceived(OpenDistroSecurityPlugin.java:639) ~[?:?]
		at com.amazon.opendistroforelasticsearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:124) ~[?:?]
		at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72) ~[?:?]
		at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:207) ~[?:?]
		at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[?:?]
		at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[?:?]
		at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) ~[?:?]
		at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) ~[?:?]
		at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) ~[?:?]
		at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) ~[?:?]
		at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[?:?]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
		at java.lang.Thread.run(Thread.java:835) [?:?]

At first, we think the problem is with IM plugin or security plugin, so we did different combination tests, results showing in the table. Based on this, the problem is likely in the security plugin.

w/o IM plugin
w/ IM plugin

w/o security plugin
not tested
good

w/ security plugin
has problem
has problem

This problem probably can happen for any request that throw exception we believe.
Expected behavior
The request got back response like
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [example1]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"example1","index":"example1"}],"type":"index_not_found_exception","reason":"no such index [example1]","index_uuid":"_na_","resource.type":"index_or_alias","resource.id":"example1","index":"example1"},"status":404}

Host/Environment (please complete the following information):

OS: Linux
Version [e.g. 22]

Additional context
Add any other context about the problem here.

If the problem is in the security plugin we should move this issue to the security repository

Also, if this is a 1.0.0 GA release blocker please make sure it's labeled as such.

source: gharchive/issue

authors: bowenlan-amzn nknize

repo: opensearch-project/OpenSearch

license_type: permissive

license_source: github-api


text:
Fix Flaky CloseIndexIT.testCloseWhileDeletingIndices

Description
The test was errorring out due to timeout.

java.lang.AssertionError: Unexpected exception: ProcessClusterEventTimeoutException[failed to process cluster event (add-block-index-to-close [[vquumsvuoq/KiZV1g8bRsWoscd3K-G7Tg]]) within 30s]

Increased the timeout to 60s. 5000 iterations passed locally.
Screenshot of 1000 iterations passing:

Related Issues
Resolves #11610
Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[x] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Increased the timeout to 60s. 5000 iterations passed locally.

@sarthakaggarwal97 Could you reliably reproduce this error within 5000 iterations in your local testing before increasing the timeout?

@andrross, yes able to reproduce it currently in the 5000 iterations. Attaching screenshot for reference.

source: gharchive/pull-request


text:
Added missed API visibility annotations for public / experimental APIs

Description
Added missed API visibility annotations for public / experimental APIs
Related Issues
Fixing warnings from OpenSearch API annotation processor:
Note: The element org.opensearch.http.HttpServerTransport is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.plugins.SecureTransportSettingsProvider)
Note: The element org.opensearch.transport.TcpTransport is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.plugins.SecureTransportSettingsProvider)
Note: The element org.opensearch.semver.SemverRange is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.plugins.PluginInfo)
Note: The element org.opensearch.rest.RestHandler.Route is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.rest.action.admin.indices.RestViewAction.CreateViewHandler)
Note: The element org.opensearch.rest.BaseRestHandler.RestChannelConsumer is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.rest.action.admin.indices.RestViewAction.SearchViewHandler)
Note: The element org.opensearch.common.cache.RemovalListener is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.common.cache.store.config.CacheConfig)
Note: The element org.opensearch.common.cache.serializer.Serializer is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.common.cache.store.config.CacheConfig)
Note: The element org.opensearch.common.cache.policy.CachedQueryResult.PolicyValues is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.common.cache.store.config.CacheConfig)
Note: The element org.opensearch.common.lucene.index.OpenSearchDirectoryReader.DelegatingCacheHelper is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.common.lucene.index.OpenSearchDirectoryReader)
Note: The element org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory is part of the public APIs but is not marked as @PublicApi, @ExperimentalApi or @DeprecatedApi (referenced by org.opensearch.index.store.RemoteSegmentStoreDirectory)

Check List

[X] New functionality includes testing.

[X] All tests pass

[X] New functionality has been documented.

[X] New functionality has javadoc added

[X] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
[X] Commits are signed per the DCO using --signoff
[X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[X] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

If these builds are now clean, what do you think about enforcing the annotation process' checks and failing the build?

Thanks @peternied , so there was one change that held me back [1] but I am not sure anymore we should wait for it, I will work on that  to have clean pass (and make it mandatory)
[1] https://github.com/opensearch-project/OpenSearch/pull/10921


text:
[Remote Segment Store] Combine metadata and snapshot files

Description

Currently, we upload 2 files in addition to segment files per refresh (we upload these files even if there are no segments to upload - that will be fixed by https://github.com/opensearch-project/OpenSearch/issues/7443)
These 2 files are metadata file and SegmentInfosSnapshot file. In order to avoid explosion of number of files in the remote store, we use the same name of metadata and snapshot file per commit.
But the assumed invariant is: there is 1:1 mapping between metadata and SegmentInfosSnapshot file.
This does not work if while reading we read metadata file and by the time we read snapshot file, remote store has new snapshot file with the same name.
To avoid this race condition, this change combines these 2 files. metadata file will have one more field: segmentInfosBytes and in the restore flow, instance of SegmentInfos will be created by this field in the metadata.

Related Issues

https://github.com/opensearch-project/OpenSearch/issues/7772

Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

org.opensearch.remotestore.ReplicaToPrimaryPromotionIT.testPromoteReplicaToPrimary is failing intermittently. Looking into it.

Precommit is failing on windows. Re-triggering the build.


text:
Backport 1122, 1302, 1355, 1435 and 1441  to 2.7

Issue #, if available:
Backport #1122, #1302 and #1355, #1435 and #1441 to 2.7
Description of changes:
CheckList:

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Backports for 2.7 were previously completed. Closing this PR

authors: engechas goyamegh

repo: opensearch-project/alerting


text:
[BUG] Infinite redirect on root

What is the bug?
The root of playground.opensearch.org produces an infinite redirect for bots. Works for a browser.
Redirects / to /auth/anonymous? which redirects back to /.
How can one reproduce the bug?
$ curl -L "https://playground.opensearch.org/"
curl: (47) Maximum (50) redirects followed

What is the expected behavior?
No infinite redirect loop.
Do you have any additional context?
Found via https://github.com/dblock/ruby-link-checker/issues/1

@dblock
the curl need to provide cookie for anonymous to avoid infinity loop
have a try below
curl --dump-header headers https://playground.opensearch.org/
curl -b -v -L headers https://playground.opensearch.org/

authors: dblock seraphjiang

repo: opensearch-project/dashboards-anywhere


text:
Kibana read only access not working

With read_only permission role in kibana, users seem to be able to edit the dashboard which is not expected.
Issue
We have an Elasticsearch cluster set up with fine-grain access control enabled, and authentication to Kibana is handled by Cognito. We have Cognito set up to assign an IAM role to the user based on a claim passed from our SAML idp. At the moment we have two IAM roles:

Admin (Full access to Kibana, this is also specified as the elasticsearch Master User ARN)
Read-Only (Should only have read-only access to kibana dashboards/visualizations/indices)

The admin user/role is working OK, but I cannot get the read-only role configured such that they have read-only access. They are assigned the IAM role fine, and I can map the read-only backend IAM role to kibana roles. I was following this guide: https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/users-roles/#set-up-a-read-only-user-in-kibana. I created the read_only_index role w/ indices, and mapped the read-only IAM role to this and the "kibana_user"/"kibana_read_only" roles per the guide. When a read-only user logs in I can see they get assigned these kibana roles, but they still have write access to dashboards.
What is the correct combination of kibana roles/setup such that I can provide kibana users read-only access?
Observations:
I have also tried replicating the issue at my end with the help of the document https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/users-roles/#set-up-a-read-only-user-in-kibana with the internal users instead of backend role.
After making the read-only setup, when I tried logging in with the new user having read only permission, in the kibana page they are getting only the Dashboard options and all other options has been greyed out and not displayed. On the dashboard I was able to load and edit the dashboard by adding visualisation or removing the existing visualisation from the dashboard.
Goal:
Need to understand why in readonly permission, the user is able to edit or remove visualization from dashborad.

I want to say we can close it? It was an issue that I could never replicate.


text:
Removing outdated limitations in Searchable Snapshots describing KNN is not compatible.

Description
Since 2.18, now snapshot supports KNN index. Hence removing the comments saying KNN index is not supported.
Issues Resolved
https://github.com/opensearch-project/k-NN/issues/2033
Version
2.18
Frontend features
N/A
Checklist

[X] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@0ctopus13prime Is this PR ready for doc review?

@0ctopus13prime Is this PR ready for doc review?

yes it is

@kolchfa-aws
Updated accordingly! Thank you for the suggestion


text:
Add Windows Support

Coming from https://github.com/opensearch-project/opensearch-plugins/issues/95, add Windows support.

[x] Passing CI on Windows
[x] Documentation

Build on Window for main branch does not work with following error. The issue could be related with OpenSearch 3.0.0-SNAPSHOT as a build with OpenSearch 2.4.0-SNAPSHOT works.
�    ? errors and warnings from D:\a\geospatial\geospatial\build\testclusters\integTest-0\logs\opensearch.stdout.log ?
=== Standard error of node `node{::integTest-0}` ===
� ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [integTest-0] uncaught exception in thread [main]
�  org.opensearch.bootstrap.StartupException: java.lang.RuntimeException: can not run opensearch as root
�  	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  Caused by: java.lang.RuntimeException: can not run opensearch as root
�  	at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:124) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:[404](https://github.com/heemin32/geospatial/actions/runs/3253777954/jobs/5341360413#step:4:405)) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
�  	... 6 more
�   ? last 40 non error or warning messages from D:\a\geospatial\geospatial\build\testclusters\integTest-0\logs\opensearch.stdout.log ?

https://github.com/heemin32/geospatial/actions/runs/3253361449/jobs/5340559789#step:4:462

Are integration/cypress tests also running? If not please reopen this issue.

integration tests are running in all platforms.

authors: dblock heemin32 peterzhuamazon

repo: opensearch-project/geospatial


text:
[BUG] Missing cancel button in Change policy page

What is the bug?
In the 'Change policy' page, users don't have a clear path to leave the page if they decided not to change policies.

How can one reproduce the bug?
Steps to reproduce the behavior:

Go to 'Index Management'
Go to 'Policy managed indexes'
Click on 'Change policy' button

What is the expected behavior?
Next to the 'Change' button, there should be a button called Cancel that will take users back to the 'Policy managed indexes' page.
Use OuiButtonEmpty, Primary for the Cancel button:

Hi @bowenlan-amzn
Please could. you assign this to me?

Thank you @kohinoor98 !

authors: bowenlan-amzn kamingleung kohinoor98


text:
Use the Lucene Distance Calculation Function in Script Scoring for doing exact search

Description
Continuation of https://github.com/opensearch-project/k-NN/pull/1287
This PR changes the implementation of our script scoring calculations to utilize Lucene's VectorUtil class.
Issues Resolved
https://github.com/opensearch-project/k-NN/issues/1032
Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Lucene VectorUtil CosineSimilOptimized microbenchmarks
No changes:
Benchmark                                               (dimension)  (vectorsPerTransfer)  Mode  Cnt  Score    Error  Units
TransferVectorsBenchmarks.transferVectors_withCapacity          128                100000    ss    3  0.659 ±  0.102   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128                500000    ss    3  0.682 ±  0.054   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128               1000000    ss    3  0.675 ±  0.048   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                100000    ss    3  1.008 ±  0.219   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                500000    ss    3  1.012 ±  0.120   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256               1000000    ss    3  1.015 ±  0.360   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                100000    ss    3  1.331 ±  0.383   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                500000    ss    3  1.346 ±  0.112   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384               1000000    ss    3  1.333 ±  0.226   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                100000    ss    3  1.693 ±  0.507   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                500000    ss    3  1.715 ±  1.113   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512               1000000    ss    3  1.677 ±  0.273   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                100000    ss    3  2.943 ±  1.095   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                500000    ss    3  3.132 ±  4.335   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960               1000000    ss    3  2.987 ±  3.950   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                100000    ss    3  3.572 ± 10.261   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                500000    ss    3  3.198 ±  1.984   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024               1000000    ss    3  3.379 ±  0.930   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                100000    ss    3  5.695 ±  6.103   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                500000    ss    3  4.757 ±  8.886   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536               1000000    ss    3  5.080 ±  6.601   s/op

Using VectorUtil:
Benchmark                                               (dimension)  (vectorsPerTransfer)  Mode  Cnt  Score    Error  Units
TransferVectorsBenchmarks.transferVectors_withCapacity          128                100000    ss    3  0.654 ±  0.033   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128                500000    ss    3  0.663 ±  0.043   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128               1000000    ss    3  0.662 ±  0.048   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                100000    ss    3  1.000 ±  0.210   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                500000    ss    3  0.996 ±  0.094   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256               1000000    ss    3  0.995 ±  0.191   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                100000    ss    3  1.363 ±  0.669   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                500000    ss    3  1.346 ±  0.132   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384               1000000    ss    3  1.356 ±  0.778   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                100000    ss    3  1.782 ±  3.863   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                500000    ss    3  1.699 ±  0.641   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512               1000000    ss    3  1.664 ±  0.183   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                100000    ss    3  3.032 ±  3.735   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                500000    ss    3  3.107 ±  5.165   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960               1000000    ss    3  3.209 ±  4.675   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                100000    ss    3  3.280 ±  5.766   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                500000    ss    3  3.310 ±  4.152   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024               1000000    ss    3  3.400 ± 14.473   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                100000    ss    3  5.076 ± 20.049   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                500000    ss    3  4.744 ±  8.573   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536               1000000    ss    3  4.974 ±  8.913   s/op

Lucene VectorUtil CosineSimilOptimized microbenchmarks
No changes:
Block (23 lines)
Benchmark                                               (dimension)  (vectorsPerTransfer)  Mode  Cnt  Score    Error  Units
TransferVectorsBenchmarks.transferVectors_withCapacity          128                100000    ss    3  0.659 ±  0.102   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128                500000    ss    3  0.682 ±  0.054   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128               1000000    ss    3  0.675 ±  0.048   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                100000    ss    3  1.008 ±  0.219   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                500000    ss    3  1.012 ±  0.120   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256               1000000    ss    3  1.015 ±  0.360   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                100000    ss    3  1.331 ±  0.383   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                500000    ss    3  1.346 ±  0.112   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384               1000000    ss    3  1.333 ±  0.226   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                100000    ss    3  1.693 ±  0.507   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                500000    ss    3  1.715 ±  1.113   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512               1000000    ss    3  1.677 ±  0.273   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                100000    ss    3  2.943 ±  1.095   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                500000    ss    3  3.132 ±  4.335   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960               1000000    ss    3  2.987 ±  3.950   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                100000    ss    3  3.572 ± 10.261   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                500000    ss    3  3.198 ±  1.984   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024               1000000    ss    3  3.379 ±  0.930   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                100000    ss    3  5.695 ±  6.103   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                500000    ss    3  4.757 ±  8.886   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536               1000000    ss    3  5.080 ±  6.601   s/op

Using VectorUtil:
Block (23 lines)
Benchmark                                               (dimension)  (vectorsPerTransfer)  Mode  Cnt  Score    Error  Units
TransferVectorsBenchmarks.transferVectors_withCapacity          128                100000    ss    3  0.654 ±  0.033   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128                500000    ss    3  0.663 ±  0.043   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128               1000000    ss    3  0.662 ±  0.048   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                100000    ss    3  1.000 ±  0.210   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                500000    ss    3  0.996 ±  0.094   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256               1000000    ss    3  0.995 ±  0.191   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                100000    ss    3  1.363 ±  0.669   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                500000    ss    3  1.346 ±  0.132   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384               1000000    ss    3  1.356 ±  0.778   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                100000    ss    3  1.782 ±  3.863   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                500000    ss    3  1.699 ±  0.641   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512               1000000    ss    3  1.664 ±  0.183   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                100000    ss    3  3.032 ±  3.735   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                500000    ss    3  3.107 ±  5.165   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960               1000000    ss    3  3.209 ±  4.675   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                100000    ss    3  3.280 ±  5.766   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                500000    ss    3  3.310 ±  4.152   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024               1000000    ss    3  3.400 ± 14.473   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                100000    ss    3  5.076 ± 20.049   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                500000    ss    3  4.744 ±  8.573   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536               1000000    ss    3  4.974 ±  8.913   s/op

@ryanbogan so what the conclusion? and I am seeing you are using function with name: transferVectors_withCapacity is that a typo that you didn't change the name of the function while running benchmarks?

Lucene VectorUtil CosineSimilOptimized microbenchmarks
No changes:
Block (23 lines)
Benchmark                                               (dimension)  (vectorsPerTransfer)  Mode  Cnt  Score    Error  Units
TransferVectorsBenchmarks.transferVectors_withCapacity          128                100000    ss    3  0.659 ±  0.102   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128                500000    ss    3  0.682 ±  0.054   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128               1000000    ss    3  0.675 ±  0.048   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                100000    ss    3  1.008 ±  0.219   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                500000    ss    3  1.012 ±  0.120   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256               1000000    ss    3  1.015 ±  0.360   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                100000    ss    3  1.331 ±  0.383   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                500000    ss    3  1.346 ±  0.112   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384               1000000    ss    3  1.333 ±  0.226   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                100000    ss    3  1.693 ±  0.507   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                500000    ss    3  1.715 ±  1.113   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512               1000000    ss    3  1.677 ±  0.273   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                100000    ss    3  2.943 ±  1.095   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                500000    ss    3  3.132 ±  4.335   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960               1000000    ss    3  2.987 ±  3.950   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                100000    ss    3  3.572 ± 10.261   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                500000    ss    3  3.198 ±  1.984   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024               1000000    ss    3  3.379 ±  0.930   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                100000    ss    3  5.695 ±  6.103   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                500000    ss    3  4.757 ±  8.886   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536               1000000    ss    3  5.080 ±  6.601   s/op

Using VectorUtil:
Block (23 lines)
Benchmark                                               (dimension)  (vectorsPerTransfer)  Mode  Cnt  Score    Error  Units
TransferVectorsBenchmarks.transferVectors_withCapacity          128                100000    ss    3  0.654 ±  0.033   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128                500000    ss    3  0.663 ±  0.043   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          128               1000000    ss    3  0.662 ±  0.048   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                100000    ss    3  1.000 ±  0.210   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256                500000    ss    3  0.996 ±  0.094   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          256               1000000    ss    3  0.995 ±  0.191   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                100000    ss    3  1.363 ±  0.669   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384                500000    ss    3  1.346 ±  0.132   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          384               1000000    ss    3  1.356 ±  0.778   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                100000    ss    3  1.782 ±  3.863   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512                500000    ss    3  1.699 ±  0.641   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          512               1000000    ss    3  1.664 ±  0.183   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                100000    ss    3  3.032 ±  3.735   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960                500000    ss    3  3.107 ±  5.165   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity          960               1000000    ss    3  3.209 ±  4.675   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                100000    ss    3  3.280 ±  5.766   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024                500000    ss    3  3.310 ±  4.152   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1024               1000000    ss    3  3.400 ± 14.473   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                100000    ss    3  5.076 ± 20.049   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536                500000    ss    3  4.744 ±  8.573   s/op
TransferVectorsBenchmarks.transferVectors_withCapacity         1536               1000000    ss    3  4.974 ±  8.913   s/op

@ryanbogan so what the conclusion? and I am seeing you are using function with name: transferVectors_withCapacity is that a typo that you didn't change the name of the function while running benchmarks?

I just ran the base version of the microbenchmarks, which should run everything right?

@ryanbogan No, the cosinesimil is not covered by those.
Lets just leave cosinesimilOptimized untouched for now. For users really concerned about performance, they should normalize vectors during ingestion and then use dotProduct.

repo: opensearch-project/k-NN


text:
[FEATURE] Re-enable tests with mockStatic in MLEngineTest to increase test coverage

Is your feature request related to a problem?
Some tests in MLEngineTest are marked as @Ignore because they contain mockStatic which would throw exception due to the reason described here: https://github.com/opensearch-project/OpenSearch/issues/14420
What solution would you like?
Explicitly set inline mock maker when use mockStatic to mock a static function, then remove @Ignore and TODO to reenable those tests.
What alternatives have you considered?
A fix from OpenSearch core like described here https://github.com/opensearch-project/OpenSearch/issues/14420 could work for all OpenSearch repos which depend on OpenSearch core, but core maintainer proposed a refactoring of the static class which can be risky for the code here.
Do you have any additional context?
Other tests which try to mock static functions can do the same, but try to think about refactor code whenever possible to avoid mocking static functions.

close as it's completed

repo: opensearch-project/ml-commons


text:
final live tail fixes

Signed-off-by: Kavitha Conjeevaram Mohan mohakavi@amazon.com
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[x] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #558 (3e9f32c) into main (29527d6) will increase coverage by 18.83%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main     #558       +/-   ##
=============================================
+ Coverage     53.49%   72.33%   +18.83%
  Complexity      271      271
=============================================
  Files           236       38      -198
  Lines          7748     2136     -5612
  Branches       1603      236     -1367
=============================================
- Hits           4145     1545     -2600
+ Misses         3407      429     -2978
+ Partials        196      162       -34

Flag
Coverage Δ

dashboards-observability
?

opensearch-observability
72.33% <ø> (-0.21%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../org/opensearch/observability/model/Application.kt
4.76% <0.00%> (-0.51%)
:arrow_down:

...ability/public/components/common/search/search.tsx

...onfig_editor/config_controls/config_thresholds.tsx

...isualization_container/visualization_container.tsx

...lic/components/visualizations/assets/chart_pie.tsx

.../components/visualizations/visualization_chart.tsx

...lic/components/visualizations/assets/chart_bar.tsx

...ts/visualizations/charts/data_table/data_table.tsx

...ts/explorer/visualizations/drag_drop/drag_drop.tsx

...e_analytics/components/traces/flyout_list_item.tsx

... and 189 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 29527d6...3e9f32c. Read the comment docs.

LGTM- can you add in the description what the bugs being fixed are and what the cypress tests are testing for?

yeah okay

authors: codecov-commenter kavithacm

repo: opensearch-project/observability


text:
Upgrade groovy library for Validation Workflow

Description
Pass each argument from jenkins separately to Validation groovy library
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This will also need a version bump, right?

This will also need a version bump, right?

Yes

This will also need a version bump, right?

Yes

You need to update the build.gradle file for that, right?

This will also need a version bump, right?

Yes

You need to update the build.gradle file for that, right?

Waiting for this PR to get merged,

This will also need a version bump, right?

Yes

You need to update the build.gradle file for that, right?

Yes Rishabh

Please bump version.

authors: Divyaasm peterzhuamazon rishabh6788


text:
[Bug]: 1.3.0 OpenSearch staging Docker image failing to come up

Describe the bug
It looks as though the 1.3.0 staging Docker image might be failing to come up due to the InitialBootClassLoaderMetaspaceSize VM option used by the PA plugin.
The container logs outputted just showed the following:
Enabling OpenSearch Security Plugin
Killing opensearch process 130
OpenSearch exited with code 143
Performance analyzer exited with code 1

Logging onto the container itself and running the entry point script manually produced the following PA logs (no other logs under the /logs directory):
sh-4.2$ cat performance-analyzer.log
Unrecognized VM option 'InitialBootClassLoaderMetaspaceSize=30720'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

When searching older issues I came across #504 which had "OpenJDK 64-Bit Server VM warning: Option InitialBootClassLoaderMetaspaceSize was deprecated in version 15.0 and will likely be removed in a future release." in one of the log entries as well which supports the claim of its removal.
To reproduce

Pull 1.3.0 staging docker image: docker pull opensearchstaging/opensearch:1.3.0
Run it: docker run -p 9200:9200 -d -p 9600:9600 -e "discovery.type=single-node" opensearchstaging/opensearch:1.3.0

Expected behavior
Container coming up successfully.
Screenshots
If applicable, add screenshots to help explain your problem.
Host / Environment
No response
Additional context
No response
Relevant log output
No response

@peterzhuamazon Do you know if we are pulling in the correct java version?
Java version on OpenSearch 1.2.0 docker image
[opensearch@58df0f69ef24 ~]$ ~/jdk/bin/java --version
openjdk 15.0.1 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)
[opensearch@58df0f69ef24 bin]$

Java version used in the manifest:
https://github.com/opensearch-project/opensearch-build/blob/cf6314bfa74dff15cb5e1bb3ae9c814790b7cb95/manifests/1.2.0/opensearch-1.2.0.yml#L6
Java version on OpenSearch 1.3.0 docker image
[opensearch@fc5630473f60 ~]$ ~/jdk/bin/java --version
openjdk 17.0.1 2021-10-19
OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12)
OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode, sharing)

Java version used in the manifest:
https://github.com/opensearch-project/opensearch-build/blob/cf6314bfa74dff15cb5e1bb3ae9c814790b7cb95/manifests/1.3.0/opensearch-1.3.0.yml#L6

The java version in the manifest is the version for ci-runner to build under, not the one within opensearch core.
The opensearch core is defining java version somewhere else.

Moving this issue to the OpenSearch repository

authors: peternied peterzhuamazon qreshi

repo: opensearch-project/opensearch-build


text:
[META] Documentation website enhancements w/ engineering support

Is your feature request related to a problem? Please describe
Yes. The Docs as Code process can be improved by adding sign offs to development PRs. This would ensure documentation is created and managed alongside the code.
Describe the solution you'd like
Add a task list item (Documentation) through GitHub Actions.
Describe alternatives you've considered
No response
Additional context
No response

Issues:

opensearch-project/documentation-website#6400

Specifically, we're proposing the following changes:

Adding a mechanism so that a developer can indicate if documentation is required for the feature.
If documentation is required, preventing the dev PR from being merged until a documentation PR is in place in the doc repo.
If the feature adds new APIs, for each new API, the documentation PR must contain a filled out API Template, that provides all parameters, whether they are required or optional, default values, descriptions, and types.

Needs more discussion on this specific request.
Will add more acceptance criteria later.

Have some discussion with Heather offline, and seems like this requires some more testing. There are generally two models as in opt-in vs. opt-out.
If we do the opt-in model where developers need to check a box or add a label, it does not guarantee the issue creation if the label is missed. If we do the opt-out model then the timing of creating such issues and avoid unnecessary noise is key.
Still need some thoughts and discussion or even a one pager / poc before proceed with implementation.

Some ideas to consider: https://about.gitlab.com/blog/2022/10/12/five-fast-facts-about-docs-as-code-at-gitlab/

Working on GitHubApp POC and will send one pager design later.

Seems like core already have a basic workflow for creating the docs issues:
https://github.com/opensearch-project/OpenSearch/blob/main/.github/workflows/create-documentation-issue.yml

We have decided on this workflow:

Use core/plugin issue template to allow contributors to select whether they need to add a doc for their feature with needs-documentation label.
The label will trigger a issue creation in documentation-website repo.
The doc issue will inherit core/plugin issue on assigning users, add release branch, and more.

We will also add other checks later such as block PR merge, link backports to original opened issues, and more later, to help the docs team and core/plugin team avoid missing documentations, and start working on those early on.
Thanks.

authors: kolchfa-aws peterzhuamazon vagimeli


text:
[OUI] Revamp developer guide

Is your feature request related to a problem? Please describe.
Currently, the OUI Developer Guide simply just links to some EUI developer guides, which outline consuming EUI, rather than working on it.
Describe the solution you'd like
I think a developer guide like OpenSearch Dashboards' would be ideal. We need a few sections about things like:

Common commands used during development

yarn start
yarn lint
yarn test-unit
yarn build

General project-related things, like file names
Writing HTML in OUI and some nuances with that
Using Typescript vs Javascript

Prefer Typescript over Javascript
Prefer modern Typescript
Interop between Typescript and Javascript

Writing SCSS in OUI and some nuances with that

Variable system
Themes

Writing good components

We definitely need to do some research into all that should go into the developer guide, but I just wanted to kick of the discussion for now, and start tracking the issue.
Describe alternatives you've considered
We could write a new developer guide, similar to EUI's (which will need to happen anyway, as we're removing all EUI references), where we explain how to use OUI from a client's perspective, however we already have the Consuming wiki page, that outlines how to use OUI.

This is tracked in the developer experience project.

I would like to propose that we take this a level higher and use this thread to discuss what the developer experience should be, and not just about the markdown file. @BSFishy  @kaddy645  @bandinib-amzn  @ashwin-pc  @joshuarrrr - Do any of you have examples of "great" developer experiences? I have seen sample projects be added to these kinds of documents, is that something we would want to pursue eventually?

The developer guide for a design system is quite different from other developer guides imo. It need not be as complicated because it meant for users who want to contribute back to the design system which is usually rare and targeted. A markdown file(s) in the project should be enough here since the information the contributor is usually looking for is one of:

How to setup the repo and run it
How to add a change
How to write a test
Show to add it to the demo catalog
How to write examples
etc

Its not for example as complex as the developer guide for dashboards which is meant for a wider audience of users who want to extend the functionality of Dashboards using its extension points and API.
To make the dev guide more accessible and readable, we can add a page to the OUI doc site if necessary, but I wouldnt complicate much more than that.

authors: BSFishy KrooshalUX ashwin-pc

repo: opensearch-project/oui


text:
upgrade spotbugs to fix commons-text cve

Signed-off-by: Kaushal Kumar kshkmr@amazon.com
Is your feature request related to a problem? Please provide an existing Issue # , or describe.
This change is to address the commons-text CVE
Describe the solution you are proposing
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Check List

[ ] New functionality includes testing.

[x] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #316 (cf13063) into main (fa938c2) will decrease coverage by 0.15%.
The diff coverage is n/a.

:exclamation: Current head cf13063 differs from pull request most recent head 40cc51c. Consider uploading reports for the commit 40cc51c to get more accurate results

@@             Coverage Diff              @@
##               main     #316      +/-   ##
============================================
- Coverage     71.84%   71.68%   -0.16%
+ Complexity      368      367       -1
============================================
  Files            44       44
  Lines          2522     2522
  Branches        171      171
============================================
- Hits           1812     1808       -4
- Misses          598      601       +3
- Partials        112      113       +1

Impacted Files
Coverage Δ

...ormanceanalyzer/writer/EventLogQueueProcessor.java
55.95% <0.00%> (-4.77%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

repo: opensearch-project/performance-analyzer


text:
[BUG] Transient settings not taken into effect

What is the bug?
When pulling the settings: https://github.com/opensearch-project/query-insights-dashboards/blob/main/public/pages/TopNQueries/TopNQueries.tsx#L185, only persistent settings are taken into account. What happens when transient settings are present? Does it require these settings as persistent?
How can one reproduce the bug?
Steps to reproduce the behavior.
What is the expected behavior?
A clear and concise description of what you expected to happen.
What is your host/environment?
Operating system, version.
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.

[Catch All Triage - 1, 2, 3]

authors: dblock derek-ho


text:
[Backport 2.x] Resolve logback-classic to 1.2.13

Description
Resolve logback-classic to 1.2.13
Backport of: https://github.com/opensearch-project/security/pull/3823
Issues Resolved
[List any issues this PR will resolve]
Is this a backport? If so, please add backport PR # and/or commits #
Testing
[Please provide details of testing done: unit testing, integration testing and manual testing]
Check List

[ ] New functionality includes testing
[ ] New functionality has been documented
[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@derek-ho Could you please resolve the conflicts in the build.gradle and create/update flaky test failures from this build?

authors: derek-ho peternied

repo: opensearch-project/security