text:
开发文档需求

希望能够有一份开发文档，整体架构，依赖关系和本地运行的文档

希望能够有一份开发文档，整体架构，依赖关系和本地运行的文档

@git2017-jpg  Yes.  The documents are coming soon!

@git2017-jpg 看看这个文档呢，https://openkruise.io/rollouts/introduction

@git2017-jpg 看看这个文档呢，https://openkruise.io/rollouts/introduction

如果有一份dev的运行文档就好了，和kruise的类似https://github.com/openkruise/kruise/tree/master/docs/contributing，对开发者友好的帮助文档

如果我没有理解错，你是想在本地/测试环境运行 rollouts ，但不知道如何入手，所以希望有一份开发文档帮助你解决这个问题？
@git2017-jpg

https://github.com/openkruise/rollouts/blob/master/docs/contributing/debug.md#third-step-deploy-your-kruise-rollout-with-a-deployment
@git2017-jpg  @MrSumeng

@MrSumeng @veophi 是的，已经本地运行成功

/close

authors: MrSumeng git2017-jpg veophi zmberg

repo: openkruise/rollouts


text:
Update repos.gradle

Update repo URL for spring.

Minor repo change in buildscript not waiting on build.

source: gharchive/pull-request

repo: openlattice/openlattice

license_type: permissive

license_source: github-api


text:
Release does not contain CHANGES.md

I believe that the release zipfile should contain the CHANGES.md, but as of v1.1 no such file is in the archive.

Thank you @vog for pointing this, you are absolutely right.
I committed a patch for having it in next release.
For this release, please see the one at https://github.com/openlayers/ol3-cesium/blob/v1.1/CHANGES.md.

source: gharchive/issue

authors: gberaudo vog

license_source: bigquery


text:
Interpolation of TileLayer does not work in Firefox after adding VectorImage layer

Describe the bug
This only happens in Firefox, Chrome seems to be OK.
To Reproduce
Steps to reproduce the behavior:

Open Firefox (I used 95.0.1 under Linux Mint)
Go to https://jsfiddle.net/oxdLycwn/1/
See the map interpolated nicely ("blur effect")
Click on the button to add VectorImage layer
Move the map a little by dragging the map
The map is not interpolated now and you can see individual pixels

Expected behavior
A clear and concise description of what you expected to happen.

Chrome is also affected for me.  I think #13221 should also have been applied to Vector source (or interpolation made an option for Vector source where it is used in a VectorImage layer).
This also demonstrates there is an additional problem if two layers with different interpolation settings share a 2d canvas, as interpolation will then be disabled for both layers.

This also demonstrates there is an additional problem if two layers with different interpolation settings share a 2d canvas, as interpolation will then be disabled for both layers.

I think interpolation should always be set per tile, then this won't be a problem. Then it would also be consistent to always use the interpolate tile option, instead of the source option, which we could deprecate entirely.

What about Image sources?  This has been an unnoticed issue that goes back to v6.4.0. https://openlayers.org/en/v6.4.0/examples/reprojection-image.html  Ensure browser zoom is off so the pixel ratio is one then zoom in to where the edge of the image is close to some text and turn off interpolation.  Both layers change.  Switching interpolation back on does not undo either layer - in this case the image layer will appear interpolated but that is due to interpolation during scaling and reprojection, not the final render.

The underlying problem seems to be that when imageSmoothingEnabled is set false at the start of the Tile and Image layer renderers' renderFrame methods it is never reset, on the assumption that would never be necessary. https://github.com/openlayers/openlayers/blob/89f632fa7e5a06efe35e47eaaace7cfa9a10ac0f/src/ol/renderer/canvas/TileLayer.js#L321-L323 https://github.com/openlayers/openlayers/blob/89f632fa7e5a06efe35e47eaaace7cfa9a10ac0f/src/ol/renderer/canvas/ImageLayer.js#L188-L190
That would be easy to fix.

authors: ahocevar lysek mike-000

repo: openlayers/openlayers


text:
minZoom on 'ol/layer/Vector' with 'declutter' set causes error

Describe the bug
When set minZoom property and declutter to true,
As of v9.0 set minZoom on a VectorLayer when declutter is enabled  and listen 'postrender' throw error
To Reproduce
Steps to reproduce the behavior:

'Example'

Thanks for reporting this, I created #15630 to fix this.


text:
When setting global is true, point feature is also repeated.

When I set the global is true,the map can be repeated,but the point on the map repeated too,I want the points on the map do not repeat.How can I do it?

var iconFeature = new ol.Feature({
                        geometry: new ol.geom.Point([longitude,latitude]),
                    });
DataSource.addFeature(iconFeature);

Use wrapX: false on the vector source

Thank you very much! I solved the problem with the method you provided.
I still have a problem:I designed a function that pops up a message box after I click on the point.When the point is close to the map boundary,part of the message box is outside the map,is there a way to make the message box all inside the map?

I used overLay to display the message box.

if you are using ````ol.Overlay```, try setting autoPan to true

I tried setting autoPan to true,and I found that only when position is not within the current map extent will take effect,but the position of the overlay already within the extent of the map,Is there any other solution?

Original issue resolved with wrapX


text:
Does translate interaction is available in Openlayers 4 ?

Actually I'm searching for a interaction like Transform in version 3 that can scale , rotate and move vector feature on ol4 but I didn't find any working example.
I also need to combine it with modify interaction.
So please if it is steal available in offer me an example that can does thoese or If it's not guide me how to do that.
Thanks

You would have to write your own based on Pointer, like Translate is.
See https://github.com/openlayers/openlayers/blob/master/src/ol/interaction/Translate.js

@raiyni I did it before but the modify don't works any more! you can find it here:
https://gist.github.com/kingofnull/0ad644be69d8dd43c05721022ca5c3a5

Thanks! I test again and wow the translate works like a charm. And about combining it with modify it need to add before modify interaction and it works without conflicting the modify interaction. here if my final code:
var map = new ol.Map({...});
var featureDrager = new ol.interaction.Translate();
var modify = new ol.interaction.Modify({...});
map.addInteraction(featureDrager);
map.addInteraction(modify);

authors: kingofnull raiyni


text:
WebGLRenderingContext problem

Hello, I have found a webgl error. To reproduce the problem, just move the map until the image disappears from the view. Error occurs only with custom projection (proj4) of the static image.
Video:
http://lowcysniezyc.pl/wp-content/uploads/2018/09/screencast-codesandbox.io-2018.09.30-00-01-14-1.webm
and Codesandbox:
https://codesandbox.io/s/ol06x3m1nz
Thank you and please help me.

I think, i have found a solution. If image is outside the window, the image parameter returns null, so webgl can't create a texture. I don't know that this try is the right direction, but if i have replaced Your code in a Context.js by the following lines, error disappears. (i just create transparent image
instead of returning null in the image parameter)
export function createTexture(gl, image, opt_wrapS, opt_wrapT) {   var texture = createTextureInternal(gl, opt_wrapS, opt_wrapT);   if(image){     gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, image);     return texture;   }   else{     var img = new Uint8Array([ 255, 0, 0, 0 ]);     gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 1, 1, 0, gl.RGBA, gl.UNSIGNED_BYTE, img);   }   return texture; }

Can You fix this error in the new library version?

Hi,
The next version will not include a webgl renderer for image layers, so the issue will not be present anymore. Depending on your needs, you will have the possibility to use webgl rendering for edge cases and/or custom layer types.

authors: Shekenix jahow


text:
FAQ update regarding supported Operating systems

Since PDS had only Linux distros, this page had these. Need to add SDT's sources and change these to z/os, Debian and ClefOS sources
"Supported Distros" need to be changed to "Supported OS" everywhere in the code
And this:

@pleia2 can you assign this to me


text:
As a developer, I need to hash pieces of the staged ledger reconstruction parts as soon as they arrive, to speed up it's verification

This can be incredibly slow: https://github.com/openmina/openmina/blob/5810a039962c9e09c0e556939b427478a667fe26/node/src/transition_frontier/sync/ledger/staged/mod.rs#L51
Since we receive that struct, piece by piece in webnode, I want to start hashing them as soon as possible. This way we'll spread out computation, while we are waiting for the rest of the pieces from P2P.

[ ] Add validation logic for each received piece in p2p. This validation logic should calculate hash of the piece and cache it. These cached hashes must be used later as inputs are provided to the ledger service.
[ ] [Less Important] In the initial response of staged ledger parts, hashes of all individual trees must be included, so that we can validate received data as soon as possible, to detect and disconnect malicious peer as soon as possible.

After #734 pr, this issue is no longer relevant, since validation now is extremely fast, so no need parallelize it with fetching.

repo: openmina/openmina


text:
Cache verifier index

In this commit I've implemented very basic caching of indices. If that would work fine, we should also include SRS hash to the file name (in case input for verifier index is changed). and probably also move it directly to ledger (unless we don't want to have OS ops in that crate)

BTW, serialized indices are quite small:
$ ls -l ~/.cache/openmina/
total 8
-rw-rw-r-- 1 alexander alexander 2268 Jan 26 11:32 block_verifier_index.bin
-rw-rw-r-- 1 alexander alexander 2268 Jan 26 11:33 transaction_verifier_index.bin

BTW, serialized indices are quite small:

@akoptelov They are small because VerifierIndex is serialized by skipping lots of data, and those data are slow to recompute:
https://github.com/o1-labs/proof-systems/blob/0863042ca383c6578564e166724e9dbc66da19bf/kimchi/src/verifier_index.rs#L134-L151
Instead of using serde_cbor to de/serialize the VerifierIndex, we should use the existing verifier_index_{from,to}_bytes :
https://github.com/openmina/openmina/blob/8e68037aafddd43842a54c8439baeafee4c6e1eb/ledger/src/proofs/caching.rs#L352
https://github.com/openmina/openmina/blob/8e68037aafddd43842a54c8439baeafee4c6e1eb/ledger/src/proofs/caching.rs#L362
Those 2 methods do not skip any data

@sebastiencs now they are much bigger:
$ ls -l ~/.cache/openmina/*
-rw-rw-r-- 1 alexander alexander 5622552 Jan 26 18:44 /home/alexander/.cache/openmina/block_verifier_index.bin
-rw-rw-r-- 1 alexander alexander 5622552 Jan 26 18:46 /home/alexander/.cache/openmina/transaction_verifier_index.bin

@akoptelov For webnode we do have that implementation, so it might be useful. It includes hash verification.
Dumping key + srs: https://github.com/openmina/openmina/blob/webnode/cli/src/commands/snark/precalculate_block_verifier_index_and_srs.rs
Loading: https://github.com/openmina/openmina/blob/6a2dcb43af04fea43b348c05f161e919326ccc7a/node/wasm/src/lib.rs#L261

@sebastiencs @binier What should happen to SRS (and verifier index) to change? Hard fork? I'm thinking would it be enough to just have 1 file per index kind. That should be enough for a common user, but for developers this might be essential to be able to work with different versions...

@akoptelov The Verifier index will change whenever the circuits change.
rampup8 already contain changes to circuits, I don't know if more changes are planned.
Note that the verifier index is created/derived from a Prover.
To create a Prover we need its list of gates, I have pushed them all to this repository:
https://github.com/openmina/circuit-blobs/tree/main/rampup4

All *_gates.json are the list of gates, for each circuit
*_internal_vars.bin and *_rows_rev.bin are other data we need for witness generation: link
All others *.bin are snark works (ExternalSnarkWorkerRequest) used for testings
.txt files are witnesses vectors generated from OCaml, they are used for testings too

So in the future, we will need to use the *_gates.json files, to create the provers, and create the verifier index from them.
block_verifier_index.bin and transaction_verifier_index.bin probably won't be necessary anymore.
Though the advantage of having them is that we can create the verifier index quickly, without having to create their provers (which is quite slow).

for developers this might be essential to be able to work with different versions

Probably not, we will work only with the original circuits

OK, than it seems enough for now to have just these two cached files, each containing also a hashsums, for both srs and the index itself.

@sebastiencs I've moved the caching to the ledger crate. Are there any tests that these indices are involved in, so I can check that the loading works fine?

@akoptelov There is no such test for now

authors: akoptelov binier sebastiencs


text:
Errors on add an item on main page

Hi,
Not sure if I should mention this issue here...
It looks like there is an issue on main page.

Go to http://demo.openpim.org/ and login with demo/demo.
In section with Data - Product Data, etc., click on ➕

Actual
Nothing happens. In console:
Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'en')

Expected
Some right action without errors.

Hi,
Thanks for submitting this issue we will work on this.
But in common you are not able to add something with the demo account. You should receive the error: "Impossible to create children for ..." because "demo" user can not edit or create something.
So even if we fix the issue you will not be able to make something at http://demo.openpim.org/.
But you can run demo locally as described at our site and then you will be able to make changes with the user "admin" with password "admin". By the way, we have more recent version for local demo, so this issue can be already solved there.
Thanks,
Slava Pedak

Hi @vpedak,
Understood. Thanks for the quick answer.

fixed

authors: andkirby openpim

repo: openpim/web


text:
Related process refactor

This change is

Pull Request Test Coverage Report for Build 2907

4 of 19 (21.05%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.3%) to 72.411%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/openprocurement/api/plugins/related_processes/views/related_processes.py
3
18
16.67%

Totals

Change from base Build 2906:
-0.3%

Covered Lines:
2042

Relevant Lines:
2820

💛  - Coveralls

authors: bdmbdsm coveralls


text:
Upload and patch award documents

This change is

Coverage increased (+0.2%) to 72.286% when pulling c2d75dfbe555c7845d8b001f3e48cdeaadaa3015 on kosaniak:master into e540a7fc960715f63fdcccc5d8bc46be8e241f5c on openprocurement:master.

authors: coveralls kosaniak


text:
OrderImports with removeUnused=true removes types+annotations used in record components

I've created this config
---
type: specs.openrewrite.org/v1beta/recipe
name: com.fprochazka.Experiments
recipeList:
  - org.openrewrite.java.OrderImports:
      removeUnused: true

and then executed that over my project that uses a lot of records. It removed every import, that was used within a file only in record components.

@fprochazka Thanks for the detailed report! A fix has been integrated and will be available in the next release (and the next SNAPSHOT build).

authors: fprochazka knutwannheden

repo: openrewrite/rewrite


text:
Inputs containing spaces can not be passed in the CLI

Values containing space(s) can not be passed into an input when triggering a flow in the CLI.
Multiple combinations using both " and ' have been tried without success.
CLI version: 0.1.11-SNAPSHOT
Please see attached screenshot:

pending #120 -replacing CLI FW

authors: SSgtMendel oritstone

repo: openscore/score-language


text:
Enable doc_values based searching across different field types

Is your feature request related to a problem? Please describe.
While working on #11097 i realized that MultiTerm queries over Boolean fields in OpenSearch don't support doc_values. I also noticed that the documentation is incorrect and that booleans have doc_values enabled by default.
This is prevalent across multiple field types, there is a fix going out in #11144 to fix this for IP fields, I would also like to do this for other fields:

Date field -- https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/mapper/DateFieldMapper.java#L467
Geo Point - https://github.com/opensearch-project/OpenSearch/blob/10d34beb85ebee0a6bf36afece16e4ed991de641/server/src/main/java/org/opensearch/index/mapper/GeoPointFieldMapper.java#L239C9-L239C9
Geo Shape - https://github.com/opensearch-project/OpenSearch/blob/10d34beb85ebee0a6bf36afece16e4ed991de641/server/src/main/java/org/opensearch/index/query/VectorGeoShapeQueryProcessor.java#L84

Describe the solution you'd like
Since boolean fields type extends TermBasedFieldType we should probably override the termQuery and termsQuery functions and add support for doc_values queries. While we're at it, we should also add IndexOrDocValuesQuery to the range query. This will help use-cases where we have a range on a boolean field matching a lot of values.
Describe alternatives you've considered
Alternative is to do nothing. This is obviously bad since users will miss out on doc_values based queries that can improve performance.
Additional context
N/A

Hi, are we on track for this to be released in 2.12 ?

Note that for Boolean fields, we might not see much benefit from a performance standpoint for indexed fields (since we have an efficient advance implementation for term queries and multiterm query expansion isn't going to expand beyond two terms).
That said, for consistency, now that we're supporting DV-only queries on many field types (including keyword, where Boolean is essentially a very restricted subcase of a keyword field), I like the idea that any field with doc values can support a (maybe slow) DV-only query.

authors: harshavamsi kiranprakash154 msfroh

repo: opensearch-project/OpenSearch


text:
[Feature Request] Make remote snapshot local file_cache block size configurable

Is your feature request related to a problem? Please describe
To perform a search on a remote snapshot we download only the specific blocks of the snapshot needed to complete the search. These blocks have a set 8MB size and are stored on disk in a local reference counted file cache. While there are benefits to pulling down large blocks to take advantage of spatial locality and reduce the overhead of accessing our remote store, we also risk over populating our cache with un-needed data.
The large block size is particularly noticeable when initializing a remote snapshot. For each segment Lucene opens and holds onto file references to metadata. Lucene never closes these file references so the blocks must remain downloaded and present in our cache for the lifetime of the program. Particularly in the case of 'metadata' blocks 8MB is a lot and so the baseline disk usage of our caches can be drastically reduced with a more conservative block size.
Describe the solution you'd like
Can this block size be a configurable setting for a remote snapshot repository?
Related component
Search:Searchable Snapshots
Describe alternatives you've considered
Alternatively could a smaller default still improve performance? How was 8MB selected?
Additional context
Some short tests with 13GB of OSB Big5 data restored from a remote snapshot local to that cluster. This does mean very little overhead for accessing the remote snapshot and a more robust test should use an actual remote store to get a better idea of how the overhead of more frequent block downloads impacts performance.
file_cache capacity is 10MB so that we can easily populate our cache fully.
OSB query-string-on-message workload chosen due to the large number of block downloads required. Something less expensive might never access any doc fields.
Block size: 2^23 bytes
File cache baseline: 275 MB
Snapshot restore time: 843 ms
    // |                                                 Min Throughput | query-string-on-message |    1.99 |  ops/s |
    // |                                                Mean Throughput | query-string-on-message |    1.99 |  ops/s |
    // |                                              Median Throughput | query-string-on-message |    1.99 |  ops/s |
    // |                                                 Max Throughput | query-string-on-message |    1.99 |  ops/s |
    // |                                        50th percentile latency | query-string-on-message | 315.597 |     ms |
    // |                                        90th percentile latency | query-string-on-message |  320.52 |     ms |
    // |                                        99th percentile latency | query-string-on-message | 432.611 |     ms |
    // |                                       100th percentile latency | query-string-on-message | 508.895 |     ms |
    // |                                   50th percentile service time | query-string-on-message | 314.269 |     ms |
    // |                                   90th percentile service time | query-string-on-message | 318.254 |     ms |
    // |                                   99th percentile service time | query-string-on-message |  431.92 |     ms |
    // |                                  100th percentile service time | query-string-on-message | 507.844 |     ms |
    // |                                                     error rate | query-string-on-message |       0 |      % |

Block size: 2^21 bytes
File cache baseline: 109 MB
Snapshot restore time: 342 ms
    // |                                                 Min Throughput | query-string-on-message |       2 |  ops/s |
    // |                                                Mean Throughput | query-string-on-message |       2 |  ops/s |
    // |                                              Median Throughput | query-string-on-message |       2 |  ops/s |
    // |                                                 Max Throughput | query-string-on-message |       2 |  ops/s |
    // |                                        50th percentile latency | query-string-on-message | 295.624 |     ms |
    // |                                        90th percentile latency | query-string-on-message | 299.426 |     ms |
    // |                                        99th percentile latency | query-string-on-message | 343.345 |     ms |
    // |                                       100th percentile latency | query-string-on-message | 358.427 |     ms |
    // |                                   50th percentile service time | query-string-on-message |  294.41 |     ms |
    // |                                   90th percentile service time | query-string-on-message | 297.427 |     ms |
    // |                                   99th percentile service time | query-string-on-message |  342.32 |     ms |
    // |                                  100th percentile service time | query-string-on-message | 356.455 |     ms |
    // |                                                     error rate | query-string-on-message |       0 |      % |

Block size: 2^19 bytes
File cache baseline: 38 MB
Snapshot restore time: 235 ms
    // |                                                 Min Throughput | query-string-on-message |       2 |  ops/s |
    // |                                                Mean Throughput | query-string-on-message |       2 |  ops/s |
    // |                                              Median Throughput | query-string-on-message |       2 |  ops/s |
    // |                                                 Max Throughput | query-string-on-message |       2 |  ops/s |
    // |                                        50th percentile latency | query-string-on-message | 327.288 |     ms |
    // |                                        90th percentile latency | query-string-on-message |  338.82 |     ms |
    // |                                        99th percentile latency | query-string-on-message | 377.653 |     ms |
    // |                                       100th percentile latency | query-string-on-message | 387.042 |     ms |
    // |                                   50th percentile service time | query-string-on-message | 326.218 |     ms |
    // |                                   90th percentile service time | query-string-on-message | 337.726 |     ms |
    // |                                   99th percentile service time | query-string-on-message | 376.914 |     ms |
    // |                                  100th percentile service time | query-string-on-message | 385.839 |     ms |
    // |                                                     error rate | query-string-on-message |       0 |      % |

One question, if we change the block size of an existing file_cache, how do we handle the old blocks with a different block size? clear them all and repopulate the cache or split/combine the old blocks into new blocks?

@finnegancarroll are we proposing this setting to be static or dynamic?

Thanks @finnegancarroll this is an interesting feature request. I'm curious about this part:

Particularly in the case of 'metadata' blocks 8MB is a lot and so the baseline disk usage of our caches can be drastically reduced with a more conservative block size.

Have you done any measurements on how much baseline cache usage can be reduced with different block sizes? And on smaller block sizes would there be additional data that has to be redownloaded each time?
Coming at it from a different perspective, if the problem we are trying to solve is reducing the baseline cache usage then is it viable to introduce some custom logic for handling the metadata blocks instead?

Hi @finnegancarroll,
I have a noob question if you don't mind, do you think you can elaborate a bit more on the following?

For each segment Lucene opens and holds onto file references to metadata. Lucene never closes these file references so the blocks must remain downloaded and present in our cache for the lifetime of the program.

I am interested in learning more details about this part.


text:
org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout

Below test failure identified in https://github.com/opensearch-project/OpenSearch/pull/2238
**REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout" -Dtests.seed=36921F5835FAE3A1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-IN -Dtests.timezone=Etc/GMT+5 -Druntime.java=17

org.opensearch.search.SearchCancellationIT > testMSearchChildRequestCancellationWithClusterLevelTimeout FAILED
    java.lang.AssertionError: Actual child request with cancellation failure is different that expected expected:<[0, 1]> but was:<[]>
        at __randomizedtesting.SeedInfo.seed([36921F5835FAE3A1:882FD4BDA4EC03D6]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:120)
        at org.opensearch.search.SearchCancellationIT.ensureMSearchWasCancelled(SearchCancellationIT.java:194)
        at org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout(SearchCancellationIT.java:537)

❌   Gradle Check failure https://github.com/opensearch-project/OpenSearch/commit/9e413b424e9832118cb552e9e2b6faba216540a6
Log 2777
Reports 2777

The test is not reproducible when tried with provided reproduce gradle command.

Another one: https://github.com/opensearch-project/OpenSearch/pull/2541#issuecomment-1074479459

https://github.com/opensearch-project/OpenSearch/pull/2541#issuecomment-1074454204

Ran this test in isolation for 400 times and wasn't able to reproduce it.
 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.SearchCancellationIT.testMSearchChildRequestCancellationWithClusterLevelTimeout" -Dtests.seed=36921F5835FAE3A1 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-IN -Dtests.timezone=Etc/GMT+5 -Druntime.java=17 -Dtests.iters=400
Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details

> Configure project :qa:os
Cannot add task 'destructiveDistroTest.docker' as a task with that name already exists.
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.6
  OS Info               : Linux 5.11.0-1020-aws (amd64)
  Runtime JDK Version   : 17 (Oracle JDK)
  Runtime java.home     : /usr/lib/jvm/java-17/jdk-17.0.1
  Gradle JDK Version    : 17 (Oracle JDK)
  Gradle java.home      : /usr/lib/jvm/java-17/jdk-17.0.1
  Random Testing Seed   : 36921F5835FAE3A1
  In FIPS 140 mode      : false
=======================================

> Task :server:internalClusterTest
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/home/ubuntu/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/home/ubuntu/.gradle/wrapper/dists/gradle-7.6-all/9f832ih6bniajn45pbmqhk2cw/gradle-7.6/lib/plugins/gradle-testing-base-7.6.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

BUILD SUCCESSFUL in 20m 3s
43 actionable tasks: 1 executed, 42 up-to-date


text:
[BUG] org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT > testPendingTasksWithClusterNotRecoveredBlock FAILED

org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT > testPendingTasksWithClusterNotRecoveredBlock FAILED
    java.lang.AssertionError: timed out waiting for green state
        at __randomizedtesting.SeedInfo.seed([F19D942E1E4110F4:5F2076E19A98425]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureColor(OpenSearchIntegTestCase.java:979)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:918)
        at org.opensearch.test.OpenSearchIntegTestCase.ensureGreen(OpenSearchIntegTestCase.java:907)
        at org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT.testPendingTasksWithClusterNotRecoveredBlock(PendingTasksBlocksIT.java:111)

https://github.com/opensearch-project/OpenSearch/pull/3259#issuecomment-1121460572

This is one of a few different tests where I've seen an unreadable (truncated with large length) deprecation log message in the failure trace:
[2022-12-29T09:26:34,355][INFO ][o.o.a.a.c.t.PendingTasksBlocksIT] [testPendingTasksWithClusterNotRecoveredBlock] [PendingTasksBlocksIT#testPendingTasksWithClusterNotRecoveredBlock]: setting up test
[2022-12-29T09:26:34,536][DEPRECATION][o.o.d.c.s.Settings       ] [node_s0] [index.force_memory_term_dictionary] setting was dep
...[truncated 1376351 chars]...
eS8W-VsYrMCZy5A}{Ur513a1IRy24HkBPUMqLPw}{127.0.0.1}{127.0.0.1:42445}{imr}{shard_indexing_pressure_enabled=true}}, term: 4, version: 14, reason: ApplyCommitRequest{term=4, version=14, sourceNode={node_s0}

The setting comes from Store.java:
https://github.com/opensearch-project/OpenSearch/blob/57d44859bdf26d381688f37ec55afe50b46c98df/server/src/main/java/org/opensearch/index/store/Store.java#L151-L162
Flakiness is introduced as part of setRandomIndexSettings() in OpenSearchIntegTestCase:
https://github.com/opensearch-project/OpenSearch/blob/57d44859bdf26d381688f37ec55afe50b46c98df/test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java#L474-L476
But similar messages are on completely different log messages on completely unrelated tests:
[2022-11-23T10:30:18,307][INFO ][o.
...[truncated 1171668 chars]...
ansport.TransportService.sendRequest(TransportService.java:718) [main/:?]
	at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.handleWakeUp(FollowersChecker.java:348) [main/:?]
	at org.opensearch.cluster.coordination.FollowersChecker$FollowerChecker.start(FollowersChecker.java:336) [main/:?]
	at org.opensearch.cluster.coordination.FollowersChecker.lambda$setCurrentNodes$2(FollowersChecker.java:178) [main/:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) [?:?]

<and more...>

Test hasn't been reported flaky in over 2 years I guess, closing this out. Will be automatically opened if the issue persists.

authors: dblock dbwiddis sandeshkr419


text:
[Searchable Snapshot] Finalize behavior of settings API

Currently, searchable snapshots disallow all settings updates. I think we’ll need to be smarter here and allow some settings to be writable whereas others not. For example, most (all?) search-related settings (e.g. “index.max_result_window”) should be configurable, whereas all write-related settings should not be allowed to change (e.g. “index.refresh_interval”). This logic applies to both changing settings of an existing searchable snapshot index (e.g. PUT myindex/_settings) as well as when specifying the “settings” parameter when creating a searchable snapshot index via the restore API.

@andrross Do we have a list of settings which we would want to be updatable? Not sure if there is documentation on such (search/indexing) categorization?

There's no list. Part of this task is to figure it out :)
Looking at what AWS UltraWarm allows to be updated is probably a good place to start.

authors: adnapibar andrross


text:
IndicesClusterStateService blocking ClusterApplier thread and causing node drop.

Describe bug
IndicesClusterStateService is one of the cluster state applier. IndicesClusterStateService is responsible for other activity as well like handling of peer recovery failure or shard failure. All these three methods applyClusterState , handleRecoveryFailure and  FailedShardHandler are syncronized method, hence different threads can block each other while performing these operations.
We have seen cases where clusterApplierService thread gets blocked on IndicesClusterStateService because other generic thread has already took the lock on service. It can be either due to recovery failure or shard failure. If those operation takes long time to complete, it will block Cluster Applier thread for long time and cause frequent node drops. Node will follow the cycle of node-join and left events until the applier thread is not unblocked. Similarly this behavior can block other generic threads as well.
For e.g. in one case we have seen cluster applier thread was blocked for long time due to failed recovery of KNN index, where handleFailedRecovery took very long time and caused frequent node drops in cluster.
Implication:
Too many node-join/left events will occur in cluster.
Thread dumps supporting problem
Blocked Cluster Applier thread for IndicesClusterStateService
opensearch[59a08cdd7fbe4cdbecca8f5917fd78d9][clusterApplierService#updateTask][T#1]
Stack Trace is:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:256)
- waiting to lock <0x000000100bd24c78> (a org.opensearch.indices.cluster.IndicesClusterStateService)
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:606)
at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:593)
at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:561)
at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484)
at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:756)
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

Generic thread holding to lock due to recovery failure
opensearch[59a08cdd7fbe4cdbecca8f5917fd78d9][generic][T#3]
Stack Trace is:
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for  <0x00000010230dac70> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.18/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.18/AbstractQueuedSynchronizer.java:917)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.18/AbstractQueuedSynchronizer.java:1240)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.18/ReentrantReadWriteLock.java:959)
at org.opensearch.common.util.concurrent.ReleasableLock.acquire(ReleasableLock.java:69)
at org.opensearch.index.engine.Engine.close(Engine.java:1959)
at org.opensearch.core.internal.io.IOUtils.close(IOUtils.java:87)
at org.opensearch.core.internal.io.IOUtils.close(IOUtils.java:129)
at org.opensearch.core.internal.io.IOUtils.close(IOUtils.java:79)
at org.opensearch.index.shard.IndexShard.close(IndexShard.java:1691)
- locked <0x00000010230d2368> (a java.lang.Object)
at org.opensearch.index.IndexService.closeShard(IndexService.java:602)
at org.opensearch.index.IndexService.removeShard(IndexService.java:582)
- locked <0x00000010230a2a70> (a org.opensearch.index.IndexService)
at org.opensearch.indices.cluster.IndicesClusterStateService.failAndRemoveShard(IndicesClusterStateService.java:795)
at org.opensearch.indices.cluster.IndicesClusterStateService.handleRecoveryFailure(IndicesClusterStateService.java:775)
- locked <0x000000100bd24c78> (a org.opensearch.indices.cluster.IndicesClusterStateService)
at org.opensearch.indices.recovery.RecoveryListener.onFailure(RecoveryListener.java:53)
at org.opensearch.indices.recovery.RecoveryTarget.notifyListener(RecoveryTarget.java:139)
at org.opensearch.indices.replication.common.ReplicationTarget.fail(ReplicationTarget.java:176)
at org.opensearch.indices.replication.common.ReplicationCollection.fail(ReplicationCollection.java:194)
at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.onException(PeerRecoveryTargetService.java:710)
at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.handleException(PeerRecoveryTargetService.java:636)
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:316)
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1379)
at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420)
at org.opensearch.transport.InboundHandler$$Lambda$4982/0x00000017a264bc40.run(Unknown Source)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:756)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

Other blocked generic thread.
opensearch[59a08cdd7fbe4cdbecca8f5917fd78d9][generic][T#6]
Stack Trace is:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.opensearch.indices.cluster.IndicesClusterStateService.handleRecoveryFailure(IndicesClusterStateService.java:775)
- waiting to lock <0x000000100bd24c78> (a org.opensearch.indices.cluster.IndicesClusterStateService)
at org.opensearch.indices.recovery.RecoveryListener.onFailure(RecoveryListener.java:53)
at org.opensearch.indices.recovery.RecoveryTarget.notifyListener(RecoveryTarget.java:139)
at org.opensearch.indices.replication.common.ReplicationTarget.fail(ReplicationTarget.java:176)
at org.opensearch.indices.replication.common.ReplicationCollection.fail(ReplicationCollection.java:194)
at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.onException(PeerRecoveryTargetService.java:710)
at org.opensearch.indices.recovery.PeerRecoveryTargetService$RecoveryResponseHandler.handleException(PeerRecoveryTargetService.java:636)
at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:316)
at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1379)
at org.opensearch.transport.InboundHandler.lambda$handleException$3(InboundHandler.java:420)
at org.opensearch.transport.InboundHandler$$Lambda$4982/0x00000017a264bc40.run(Unknown Source)
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:756)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

I have tried exploring couple of approach to see if it is feasible to decouple both logic from synchronized behavior, but it seems like we need to keep both under synchronized block.
Cluster applier thread calls applyClusterState and Shard/Recovery failure calls the failAndRemoveShard method. I have analyzed the responsibility of both method to asses if those can run simultaneously or not.
Responsibility of applyClusterState : It will apply new cluster state and perform relvent operations like create/update/delete shards or index.
Responsibility of failAndRemoveShard: It removes the shard files and sends the shard-failed tasks to cluster manager based on method parameter. It is being used from handleRecoveryFailure and FailedShardHandler
Approach 1) Remove synchronized behavior with current responsibility
Both methods might acts on same shard, so simultaneous execution can introduce multiple race conditions and issues. I have listed few race condition based on high level analysis, but there can be more race conditions.

failedShardsCache is maintained between those two method to prevent repeated recovery of these shards on each cluster state update. This can break and it can start repeated recovery on each state update.

If applyClusterState deletes the shard as part of delete index and at same time failAndRemoveShard also goes ahead and remove the shard, it can cause issue accessing underlying node environment. This operations might not be using the same lock in IndexService and might cause issues. This can be one of the race conditions, there can be more.

Approach 2) Change responsibility of failAndRemoveShard to just fail the shard to cluster manager node
failAndRemoveShard method, removes the local shard files and send shard-failed task to cluster manager node based on method parameter. we can change this method's responsibility to just send the shard failure to master node and then rely on cluster state applier to remove the shard files.  After this we may remove synchronization between these methods.
There are some issue with this as well, like:

In some cases like source node cancels the recovery/source shard is closed/etc, where target data node don't send shard-failed to cluster manager node and directly removes the shard files. We will need to handle such scenario separately.

If recovery fails due to disk space, where space on target node gets full, data node might not be able to apply the new cluster state itself and wont be able to delete those shard files and release space. For this case, we will need behaviour where we first remove the files and then send shard-failed to cluster manager node.

After analysing above two approach, looks like removing the synchronize behavior between applyClusterState , handleRecoveryFailure and FailedShardHandler is not feasible.
I am open to hear more suggestion/approach on decoupling it.

Closing this, since it doesn't seem feasible to decouple it, we will need to keep it in synchronized block.

@dhwanilpatel

Closing this, since it doesn't seem feasible to decouple it, we will need to keep it in synchronized block.

What are next action items for this issue?

Reopening this issue, as this is impacting cluster stability everytime applier threads are stuck and causes node drops.

[Catch All Triage - 1, 2]

authors: dblock dhwanilpatel romiras shwetathareja


text:
[Backport 2.x] Backports 6558 and 6537

Description

Backports #6357 and #6558 to 2.x

Issues Resolved

NA

Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #6584 (a343c15) into 2.x (59f38d2) will decrease coverage by 0.05%.
The diff coverage is 37.50%.

:exclamation: Current head a343c15 differs from pull request most recent head aae57b1. Consider uploading reports for the commit aae57b1 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##                2.x    #6584      +/-   ##
============================================
- Coverage     70.50%   70.45%   -0.05%
+ Complexity    59328    59310      -18
============================================
  Files          4800     4800
  Lines        284591   284605      +14
  Branches      41347    41353       +6
============================================
- Hits         200640   200532     -108
- Misses        67190    67320     +130
+ Partials      16761    16753       -8

Impacted Files
Coverage Δ

...mote/directory/RemoteSnapshotDirectoryFactory.java
21.42% <0.00%> (-0.80%)
:arrow_down:

...ain/java/org/opensearch/search/DocValueFormat.java
61.93% <11.11%> (-4.51%)
:arrow_down:

...a/org/opensearch/repositories/fs/FsRepository.java
70.73% <100.00%> (+3.16%)
:arrow_up:

...main/java/org/opensearch/node/NodeMocksPlugin.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...g/opensearch/index/analysis/CharFilterFactory.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/opensearch/search/dfs/AggregatedDfs.java
51.61% <0.00%> (-45.17%)
:arrow_down:

...ava/org/opensearch/search/dfs/DfsSearchResult.java
43.26% <0.00%> (-35.58%)
:arrow_down:

...earch/client/indices/GetIndexTemplatesRequest.java
50.00% <0.00%> (-34.62%)
:arrow_down:

...ion/admin/cluster/node/info/PluginsAndModules.java
53.12% <0.00%> (-34.38%)
:arrow_down:

...nsearch/index/shard/IndexShardClosedException.java
66.66% <0.00%> (-33.34%)
:arrow_down:

... and 529 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter kotwanikunal


text:
Refactor metric names

Description
Refactor metric names
Issues Resolved
https://github.com/opensearch-project/documentation-website/issues/3570
Test
UT
dashboards-search-relevance % yarn test ./server/metrics
yarn run v1.22.19
$ ../../node_modules/.bin/jest --config ./test/jest.config.js ./server/metrics
 PASS  server/metrics/metrics_service.test.ts
  MetricsService
    test addMetric and getStats
      ✓ should add metrics to the correct interval and ignore metrics in the future (7 ms)
      ✓ should add metrics to the correct component (1 ms)
      ✓ should add metrics to the correct action (1 ms)
      ✓ should add metrics to the correct status code (2 ms)
    resetMetrics
      ✓ should clear all metrics data (2 ms)
    trim
      ✓ should remove old metrics data (2 ms)

Test Suites: 1 passed, 1 total
Tests:       6 passed, 6 total
Snapshots:   0 total
Time:        1.948 s, estimated 3 s
Ran all test suites matching /.\/server\/metrics/i.
✨  Done in 3.23s.

IT
dashboards-search-relevance % curl -XGET http://localhost:5603/wbd/api/relevancy/stats
{
  "data": {
    "search_relevance": {
      "fetch_index": {
        "200": {
          "response_time_total": 28.79286289215088,
          "count": 1
        }
      },
      "single_search": {
        "200": {
          "response_time_total": 29.817723274230957,
          "count": 1
        }
      },
      "comparison_search": {
        "200": {
          "response_time_total": 13.265346050262451,
          "count": 2
        }
      }
    }
  },
  "overall": {
    "response_time_avg": 17.968983054161072,
    "requests_per_second": 0.06666666666666667
  },
  "counts_by_component": {
    "search_relevance": 4
  },
  "counts_by_status_code": {
    "200": 4
  }
}%

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

export enum METRIC_NAME {
  RELEVANT_SEARCH = 'search_relevance',
}

If you're renaming things in this PR, can you please rename this constant?

sure updated


text:
Receive log data from S3 as a Source

Use-Case
Many users have external systems which write their logs to Amazon S3. These users want to use OpenSearch to analyze these logs. Data Prepper is an ingestion tool which can aid teams in extracting these logs for S3 and sending them to OpenSearch or elsewhere.
This proposal is to receive events from S3 notifications, read the object from S3, and create log lines for these.
Basic Configuration
This plugin will be a single source plugin which:

Polls a configured SQS standard queue which should hold S3 Event messages.
Reads S3 objects which the message indicates as created.
Uses a configured codec to parse the S3 object into Log Events.
Writes the Log Events into the Data Prepper buffer.

The following example shows what a basic configure would look like.
source:
  s3:
    notification_type: sqs
    sqs:
      queue_url: "https://sqs.us-east-2.amazonaws.com/123456789012/MyS3EventQueue"
    codec:
      single-line:
  processor:
    grok:
      match:
        message:  [ "%{COMMONAPACHELOG}" ]

Detailed Process
The S3 Source will start a new thread for reading from S3. (The number of threads can be configured).
This thread will perform the following steps repeatedly until shutdown

Use the SQS ReceiveMessage API to receive messages from SQS.
For each Message from SQS, it will:
a. Parse the Message as an S3Event.
b. Download the S3 Object which the S3Event indicates was created.
c. Decompress the object if configured to do so.
d. Parse the decompressed file using the configured codec into a list of Log Event objects.
e. Write the Log objects into the Data Prepper buffer.
Perform a DeleteMessageBatch with all of the messages which were successfully processed.
Repeat

Error Handling
The S3 Source will suppress exceptions which occur during processing. Any Message which is not processed correctly will not be included in the DeleteMessageBatch request. Thus, the message will appear in the SQS again. Data Prepper expects that the SQS queue is correctly configured with a DLQ or MessageRetentionPeriod to prevent the SQS queue from filling up with invalid messages.
Codecs
The S3 Source will use configurable codecs to support multiple data formats in the S3 objects. Initially, two codecs are planned:

single-line - This is used for logs which should be separated by a newline.
json - A codec for parsing JSON logs

Single Line
The single-line codec has no configuration items.
Below is an example S3 object.
POST /search
POST /index
PUT /document/12345

With single-line, the S3 source will produce 3 Events, each with the following structure.
"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.log",
"message" : "POST /search"

"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.log",
"message" : "POST /index"

"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.log",
"message" : "PUT /document/12345"

JSON
The json codec supports reading a JSON file and will create Events for each JSON object in an array. This S3 plugin is starting with the expectation that the incoming JSON is formed as a large JSON array of JSON objects. Each JSON object in that array is an Event. Thus, this codec will find the first JSON array in the JSON. It will output the objects within that array as Events from the JSON.
Future iterations of this plugin could allow for more customization. One possibility is to use JSON Pointer. However, the first iteration should meet many use-cases and allows for streaming the JSON to support parsing large JSON objects.
Below is an example configuration. This configures the S3 Sink to read a JSON array from the items key.
s3:
  codec:
    json:

Given the following S3 Object:
{
  "http_requests" : [
    { "status" : 200, "path" : "/search", "method" : "POST" },
    { "status" : 200, "path" : "/index", "method" : "POST" },
    { "status" : 200, "path" : "/document/12345", "method" : "PUT" }
  ]
}

The S3 source will output 3 Log events:
"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.json",
"message" : { "status" : 200, "path" : "/index", "method" : "POST" }

"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.json",
"message" : { "status" : 200, "path" : "/search", "method" : "POST" }

"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.json",
"message" : { "status" : 200, "path" : "/document/12345", "method" : "PUT" }

Compression
The S3 Source will support three configurations for compression.

none - The object will be treated as uncompressed.
gzip - The object will be decompressed using the gzip decompression algorithm
automatic - The S3 Source will example the object key to guess if it is compressed or not. If the key ends with .gz the S3 Source will attempt to decompress it using gzip. It can support other heuristics to determine if the file is compressed in future iterations.

Full Configuration Options

Option
Type
Required
Description

notification_type
Enum: sqs
Yes
Only SQS is supported. SNS may be a future option

compression
Enum: none, gzip, automatic
No
Default is none

codec
Codec
Yes
See Codecs section above.

sqs.queue_url
String - URL
Yes
The queue URL of the SQS queue.

sqs.maximum_messages
Integer
No
Directly related to SQS input. Default is 10.

sqs.visibility_timeout
Duration
No
Directly related to SQS input. Default is TBD.

sqs.wait_time
Duration
No
Directly related to SQS input. Default is TBD.

sqs.poll_delay
Duration
No
An optional delay between iterations of the process. Default is 0 seconds.

sqs.thread_count
Integer
No
Number of threads polling S3. Default is 1.

region
String
Yes
The AWS Region. TBD.

sts_role_arn
String
No
Role used for accessing S3 and SQS

access_key_id
String
No
Static access to S3 and SQS

secret_key_id
String
No
Static access to S3 and SQS

S3 Events
The S3 Source will parse all SQS Messages according to the S3 Event message structure.
The S3 Source will also parse out any event types which are not s3:ObjectCreated:*. These events will be silently ignored. That is, the S3 Source will remove them from the SQS Queue, and will not create an Events for them.
AWS Permissions Needed
The S3 Source will require the following permissions:

Action
Resource

s3:GetObject
The S3 bucket and key path for any object needed

sqs:ReceiveMessage
The ARN of the SQS queue specified by sqs.queue_url

sqs: DeleteMessageBatch
The ARN of the SQS queue specified by sqs.queue_url

Possible Future Enhancements
Direct SNS Notification
The notification_type currently only supports SQS. Some teams may want Data Prepper to receive notifications directly from SNS and thus remove the need for an SQS queue.
The notification_type could support an sns value in the future.
Additional Codecs
As needed, Data Prepper can support other codecs. Some possible candidates to consider are:

Multi-line
JSON List

Metrics

messagesReceived (Counter)
messagesDeleted (Counter)
messagesFailed (Counter)
eventsCreated (Counter)
requestsDuration (Timer)

Not Included

This proposal is focused only reading S3 objects starting with a notification. Thus any use-case for replay is not part of this scope. Also, use-cases for reading existing logs are not covered. These use-cases can have their own issue.
Updated S3 objects are not part of the scope. This work will only support use-cases when a log file is written once.
Configuration of SQS queue to receive SNS topics should be done externally. Data Prepper will not manage this.

Tasks

[x] #1423
[x] #1424
[ ] #1425
[ ] #1433
[ ] #1434
[ ] #1435

There are at least two use-cases for this feature:

Playing back which Data Prepper originally created via the S3 Sink (to be created in #1048)
Reading data which was created in S3 by other sources

+1

I have created an initial draft for the S3 Source. Please see the description for details.

Does it make sense to require users to use the json processor for json logs? Why not just drop the message key and make each json object it's own Event?

@graytaylor0 , That's a good question. At first, I was thinking that it may be valuable to use the json processor since it could possibly have more advanced features. But, as it is currently proposed, there isn't much there. Also, the way Jackson works, Data Prepper may have to re-serialize the JSON for the current proposal.
So, I am for changing this to output Event objects which fit the JSON model.
I'm also interested in including bucket and key data in the S3 object. So perhaps the JSON should go into a message or object key.
As an example, here is a possible input S3 object:
{
  "fieldA" : "abc",
  "fieldB" : "xyz",
  "http_requests" : [
    { "status" : 200, "path" : "/search", "method" : "POST" },
    { "status" : 200, "path" : "/index", "method" : "POST" },
    { "status" : 200, "path" : "/document/12345", "method" : "PUT" }
  ]
}

The output would be three Events:
"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.log",
"message" : { "status" : 200, "path" : "/index", "method" : "POST" }

"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.log",
"message" : { "status" : 200, "path" : "/search", "method" : "POST" }

"bucket" : "my-bucket",
"key" : "application1/instance200/2022-05-11.log",
"message" : { "status" : 200, "path" : "/document/12345", "method" : "PUT" }

Thoughts?

I have a few points to raise:

I think we have some potential configuration naming inconsistencies issues around credentials. The OpenSearch sink supports aws_region, aws_sts_role_arn, etc.
Would it be possible to make this solution a little more generic to support reading from a source based on a notification? I am thinking we have plugin support for notifications (AWS SQS, Azure Event Grid) and source (AWS S3, Azure Blob Storage, etc).
What metrics will be available with this new source plugin?

Some of the additional SQS properties to consider for polling:
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

ReceiveMessage AttributeName: attributes that need to be returned along with each message. This can be used to generate client side metrics.
MaxNumberOfMessages: To handle batch request, this can be configured to read up to 10 messages.

Are we planning to use long polling to receive the message ?

public class S3Configuration {
}

I think we have some potential configuration naming inconsistencies issues around credentials. The OpenSearch sink supports aws_region, aws_sts_role_arn, etc.
Would it be possible to make this solution a little more generic to support reading from a source based on a notification? I am thinking we have plugin support for notifications (AWS SQS, Azure Event Grid) and source (AWS S3, Azure Blob Storage, etc).
What metrics will be available with this new source plugin?

Thanks @cmanning09  for bringing up these points.

I addressed the naming inconsistencies in a PR which aligns with the OpenSearch sink naming.
We can create a separate section in the configuration for different notifications as seen here in current configuration for AWS.
Metrics will be added to issue description.

I made a change to the proposal for how JSON is loaded. The new proposal is to choose the first JSON array found. The expectation for Data Prepper is to receive a JSON object which is a single array of many different events. This should work with systems such as AWS CloudTrail.
The original proposal was the use a JSON Pointer to select the array. The disadvantage to that approach is that it is not compatible with Jackson's streaming API. Using the new proposal will allow Data Prepper to use Jackson's streaming API. This will allow Data Prepper to load only parts of the JSON into memory at a time. And it could allow for retrying large files.

I'd like to propose adding a few other metrics:

eventsFailed - A count for the number of Record<Event> objects which the S3Source created, but was unable to place in the buffer before the buffer write times out.
s3ObjectsFailed - Counter for the number of S3 objects which the S3 Source was unable to either load or parse. This could happen if the permissions are insufficient, or if the format is not correct (e.g. the source is configured for JSON, but it is a text file).
sqsApproximateNumberOfMessages - Counter for the ApproximateNumberOfMessages value provided by GetQueueAttributes. This will require that the S3 Source also poll the the GetQueueAttributes endpoint to provide this metric. Including this metric should be optional since not all Data Prepper nodes will be given permissions to call GetQueueAttributes.

Taking another look, I recommend that we remove the eventsCreated and eventsFailed metrics. These are redundant with the buffer's metrics - recordsWritten and writeTimeouts. Also, the Trace and HTTP sources do not have similar metrics.

authors: asifsmohammed cmanning09 dinujoh dlvenable graytaylor0 laneholloway stijnvanrenterghem

repo: opensearch-project/data-prepper


text:
Logging for Transform Job

Issue: #1133
Description of changes:
Improved logging around transform job
CheckList:

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@r1walz @vikasvb90 it would be great if you can take a look at this!

repo: opensearch-project/index-management


text:
Add support for build version qualifiers.

Coming from https://github.com/opensearch-project/opensearch-build/issues/1632, add support for -Dbuild.version_qualifier=.
The distribution build process now supports passing additional optional qualifiers other than the 3 digit versions, e.g. 2.0.0-alpha1. Add the ability to pass additional qualifier to build scripts, and generate artifacts with a version that includes the qualifier.

@dblock what exactly does this mean for the plugin? We can use qualifier to refer base OpenSearch artifact, in such case effectively it's going to be part of the "opensearch.version". Or it can be used to build plugin artifact as well, so the build command in plugin produces something like "opensearch-knn-2.0.0.0-preview-SNAPSHOT.zip".

authors: dblock martin-gaievski

repo: opensearch-project/k-NN


text:
Add a wrapper library to validate artifacts

Description
This library will be responsible for running validation.sh script from the workspace. It will hard exit if validation.sh is not found. The args are variable so its generic enough to be used by project
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #173 (6e34e5d) into main (513fb3c) will increase coverage by 0.08%.
The diff coverage is 100.00%.

:exclamation: Current head 6e34e5d differs from pull request most recent head 7198319. Consider uploading reports for the commit 7198319 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main     #173      +/-   ##
============================================
+ Coverage     83.67%   83.75%   +0.08%
  Complexity       25       25
============================================
  Files            71       72       +1
  Lines           196      197       +1
  Branches         11       11
============================================
+ Hits            164      165       +1
  Misses           24       24
  Partials          8        8

Impacted Files
Coverage Δ

tests/jenkins/jobs/ValidateArtifacts_Jenkinsfile
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter gaiksaya


text:
[Backport 1.x] Add library to publish artifacts to PyPi  (#36)

Backport 75b24efe07a15a30772c9cd660477b8d3d304499 from #36

Codecov Report

:exclamation: No coverage uploaded for pull request base (1.x@9db6880). Click here to learn what that means.
The diff coverage is n/a.

:exclamation: Current head 75b24ef differs from pull request most recent head 0dea738. Consider uploading reports for the commit 0dea738 to get more accurate results

@@          Coverage Diff           @@
##             1.x      #37   +/-   ##
======================================
  Coverage       ?   83.95%
  Complexity     ?       24
======================================
  Files          ?       64
  Lines          ?      187
  Branches       ?       10
======================================
  Hits           ?      157
  Misses         ?       24
  Partials       ?        6

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


text:
k-NN library x64 build should use generic march

In ODFE, we had run into issues with library incompatibilities due to the k-NN library being compiled on a machine with support for SIMD instruction extensions and being run on a machine without support for SIMD instruction extensions. We added a small fix in ODFE CD to resolve this.
ARM is not impacted.
We should have a similar fix in OpenSearch.

Jack has provided the libs we can manually replace it.
We can also try to run it on centos docker, if this PR merged very soon #588

Issues coming as CentOS7 does not like -march=native during the build:
https://gcc.gnu.org/onlinedocs/gcc/AArch64-Options.html
Native Error Log on CentOS7 make

+ cmake .
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Build type: Release
-- Header files: /tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/cmd_options.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/distcomp.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/distcomp_edist.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/eval_metrics.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/eval_results.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/experimentconf.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/experiments.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/falconn_heap_mod.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/flexbuff.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/floatdiff.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/global.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/gold_standard.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/idtype.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/incremental_quick_select.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/index.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/init.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/inmem_inv_index.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/knnquery.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/knnqueue.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/logging.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/memory.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/meta_analysis.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/methodfactory.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/my_isnan_isinf.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/object.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/params.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/params_cmdline.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/params_def.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/permutation_type.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/permutation_utils.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_align.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_intrinsics.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_popcount.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_prefetch.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/portable_simd.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/ported_boost_progress.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/pow.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/query.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/query_creator.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/rangequery.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/read_data.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/report.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/report_intr_dim.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/searchoracle.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/simddebug.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/sort_arr_bi.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/spacefactory.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/thread_pool.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/tune.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/utils.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/ztimer.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/dummy.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/hnsw.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/hnsw_distfunc_opt_impl_inline.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/permutation_prefix_index.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/pivot_neighb_common.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/pivot_neighb_invindx.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/seqsearch.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/simple_inverted_index.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/small_world_rand.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/vptree.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/method/vptree_utils.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_ab_diverg.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_hamming.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_jaccard.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_bit_vector.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_bregman.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_dummy.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_js.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_l2sqr_sift.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_leven.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_lp.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_rank_correl.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_renyi_diverg.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_scalar.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_bin_common.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_dense_fusion.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_jaccard.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_lp.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar_bin_fast.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_scalar_fast.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_vector.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sparse_vector_inter.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_sqfd.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_string.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_vector.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_vector_gen.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/space/space_word_embed.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/factory/init_methods.h;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/include/factory/init_spaces.h
-- Source files: /tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_bregman.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_diverg.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_edist.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_js.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_l2sqr_sift.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_lp.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_overlap.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_rankcorr.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_scalar.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_sparse_scalar_fast.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/experimentconf.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/global.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/init.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/knnquery.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/logging.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/memory.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/params.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/params_cmdline.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/query.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/rangequery.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/searchoracle.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/utils.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_ab_diverg.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_bregman.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_dummy.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_js.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_l2sqr_sift.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_lp.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_renyi_diverg.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_scalar.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_dense_fusion.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_jaccard.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_lp.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_scalar_bin_fast.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_scalar_fast.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_vector.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_sparse_vector_inter.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_string.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_vector.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/space/space_word_embed.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/dummy.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/hnsw.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/hnsw_distfunc_opt.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/pivot_neighb_invindx.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/seqsearch.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/simple_inverted_index.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/small_world_rand.cc;/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/method/vptree.cc
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found Threads.
-- OS:                Linux
-- Compiler:          GNU 4.8.5
-- CXX flags release: -Wall -Wunreachable-code -Wcast-align -Ofast -lm -DNDEBUG -std=c++11 -pthread  -DHAVE_CXX0X  -march=native -Wl,--no-as-needed -fpic
-- CXX flags debug:   -Wall -Wunreachable-code -Wcast-align -ggdb  -lm -DNDEBUG -std=c++11 -pthread  -DHAVE_CXX0X  -march=native -Wl,--no-as-needed -fpic
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/tmp_8_16mqn/k-NN/jni
+ make
Scanning dependencies of target NonMetricSpaceLib
[  2%] Building CXX object external/nmslib/similarity_search/src/CMakeFiles/NonMetricSpaceLib.dir/distcomp_bregman.cc.o
/tmp/tmp_8_16mqn/k-NN/jni/external/nmslib/similarity_search/src/distcomp_bregman.cc:1:0: error: unknown value 'native' for -march
 /**
 ^
make[2]: *** [external/nmslib/similarity_search/src/CMakeFiles/NonMetricSpaceLib.dir/distcomp_bregman.cc.o] Error 1
make[1]: *** [external/nmslib/similarity_search/src/CMakeFiles/NonMetricSpaceLib.dir/all] Error 2
make: *** [all] Error 2

We have resolved this by changing native to armv8-a as native is only introduced in CentOS8:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/conside[…]lers-and-development-tools_considerations-in-adopting-rhel-8
@jmazanec15 will raise a PR to fix this in the build workflows.
Thanks.

Close this as it is working now in build workflow.

repo: opensearch-project/opensearch-build


text:
Include integration testing against unreleased version

Description
Include integration testing against unreleased version
Issues Resolved
#47
Check List

[ ] New functionality includes testing.

[x] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has documentation added

[x] Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #53 (6beb227) into main (4e33131) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main      #53   +/-   ##
=======================================
  Coverage   67.26%   67.26%
=======================================
  Files          31       31
  Lines        1848     1848
=======================================
  Hits         1243     1243
  Misses        499      499
  Partials      106      106

Flag
Coverage Δ

opensearch-cli
67.26% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

authors: VijayanB codecov-commenter

repo: opensearch-project/opensearch-cli


text:
Copy of Sigv4excision (PR #257), and add SigV4Signer (Not yet used)

Description
This PR copies what PR #257 (by @mikaylathompson ) does + adds the SigV4 signer class (Not yet used).
"
A task to add SigV4 signing to the replayer is underway.
In the meantime, testing against a cluster that does not require SigV4 revealed that if the original clients are signing the requests, the signature-related headers will be passed along to the target cluster. This signature will almost certainly fail to be validated because it includes the Host header, which we modify.
This PR, as a pre-req, to full SigV4 signing, simply strips out the signature-related headers: Authorization and X-Amz-Security-Token.
"
TODO: Enable SigV4 support
Issues Resolved
Testing
Copied from PR #257
Unit tests are included.
Additionally, evidence is available from deploying this in-situ for WhiteFir.
Pre-deployment, 100% of _bulk requests (which were signed by the client) failed with a 403 error. Requests to /, which came overwhelmingly from ALB health checks and were not signed, succeeded (200) 100% of the time.
Post-deployment, all requests--signed or not--received a 200 response.
Check List

[X] New functionality includes testing

[X] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented
[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Be careful rolling out your own copy of Sigv4 signer. There's tons of bugs in those that are hard to find and fix. If you don't mind a dependency, use https://github.com/acm19/aws-request-signing-apache-interceptor.

authors: dblock okhasawn


text:
[BUG] Tests fail on Windows

What is the bug?
Several unit tests fail on Windows.
2023-05-15T20:39:04.2655572Z test_opensearchpy/test_transport.py::TestTransport::test_failed_connection_will_be_marked_as_dead FAILED [ 17%]

2023-05-15T20:39:04.3356121Z test_opensearchpy/test_transport.py::TestTransport::test_resurrected_connection_will_be_marked_as_live_on_success FAILED [ 18%]

2023-05-15T20:39:05.7608976Z test_opensearchpy/test_async/test_connection.py::TestAIOHttpConnection::test_uses_given_ca_certs FAILED [ 22%]
2023-05-15T20:39:05.9891902Z test_opensearchpy/test_async/test_connection.py::TestAIOHttpConnection::test_uses_default_ca_certs FAILED [ 22%]
2023-05-15T20:39:06.2217161Z test_opensearchpy/test_async/test_connection.py::TestAIOHttpConnection::test_uses_no_ca_certs FAILED [ 23%]

How can one reproduce the bug?
Run tests on Windows.

Dup of #347

repo: opensearch-project/opensearch-py


text:
[Feature/extensions] Implement createDetector request in AD Extension

Create a Transport API for creating Index for AD extension

[x] Initial discussion of the deliverables
[x] Figure out if an API already exist in clients/OpenSearch
[x] Create the API if not
[x] Register custom POST route for create detector
[x] Integrate the createIndex API with AD plugin to test if SDKClient is working as expected
[ ] Integrate the createDetector workflow

As we are looking into creating a SDKClient and call the REST APIs associated with it to create the AD index
Listing out the occurrences of client for creating a detector:

https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/transport/IndexAnomalyDetectorTransportAction.java#L95
https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/transport/IndexAnomalyDetectorTransportAction.java#L190
https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/rest/handler/AbstractAnomalyDetectorActionHandler.java#L279
https://github.com/opensearch-project/anomaly-detection/blob/main/src/main/java/org/opensearch/ad/indices/AnomalyDetectionIndices.java#L538

1 and 3 will be handled as part of #22 (or are already handled in #23).
2 looks like a dependency on the "user" stored with the detector which in our case will already be "null" so probably requires no action but might need some documentation.
4 is the AdminClient that I think we get for free with the AbstractClient inheritance.

Opened up #63 to track item 2 above.  The rest should be no problem.

Keeping this on hold until #24 is finished.

Hi @dbwiddis! To replicate prepareRequest for creating a detector. This issue requires seqNo, primaryTerm, refreshPolicy, detectorId which in turns comes from paramAsLong, hasParam, param and contentParser. I see we have issue already for the same https://github.com/opensearch-project/opensearch-sdk-java/issues/111.

]This issue requires seqNo, primaryTerm, refreshPolicy, detectorId which in turns comes from paramAsLong, hasParam, param and contentParser. I see we have issue already for the same #111.

#111 (and my PR #163) has implemented the param parts, but content parser is a bigger issue.  It's the XContent parser in the existing RestRequest class and the parser itself is a functional interface which is ugly to send over transport.
I see #91 in draft but I don't think that directly solves the problem anyway.
One possible solution is to entirely recreate the (JSON, at least) XContent parser on the SDK side of things.   We can probably get smart by inheriting classes from OpenSearch but these should be instantiated on the SDK side so they are not passed as part of a RestRequest.
I think this should be part of @ryanbogan CreateComponents piece, or at least a follow-on to that piece.

One possible solution is to entirely recreate the (JSON, at least) XContent parser on the SDK side of things. We can probably get smart by inheriting classes from OpenSearch but these should be instantiated on the SDK side so they are not passed as part of a RestRequest.

This solution would require a few things. Digging into RestRequest::contentParser() here, you can see here that in order to generate this Xcontent parser on the SDK side, we would first need to transport the consolidated namedXcontentRegistry from Node.java to the SDK, or at least the namedXcontents from all the required modules so that we could also include the AnomalyDetection namedXcontent entries and generate the NamedXContentRegistry on the extension for use in creating a parser, since the underlying data structure within the registry is an unmodifiable map.

n order to generate this Xcontent parser on the SDK side, we would first need to transport the consolidated namedXcontentRegistry ...

This is what I'm saying is ugly.
I'm suggesting we generate the whole registry fresh on the SDK side.
There are a fixed number of registries, see here:
https://github.com/opensearch-project/OpenSearch/tree/main/libs/x-content/src/main/java/org/opensearch/common/xcontent
The registry is initialized in Node.java by iterating known getters:
NamedXContentRegistry xContentRegistry = new NamedXContentRegistry(
    Stream.of(
        NetworkModule.getNamedXContents().stream(),
        IndicesModule.getNamedXContents().stream(),
        searchModule.getNamedXContents().stream(),
        pluginsService.filterPlugins(Plugin.class).stream().flatMap(p -> p.getNamedXContent().stream()),
        ClusterModule.getNamedXWriteables().stream()
    ).flatMap(Function.identity()).collect(toList())
);

The ones starting from uppercase class names (Network, Indices, Cluster) are static and can be called from the SDK side exactly as they are here.

OK, I'm overcomplicating this with the registries. Let's get back to the parsers.  I think I can completely handle this in #163.
The REST request will have an enum XContentType with only 4 possible values: JSON, SMILE, YAML, and CBOR.
This enum can just be sent over transport, and then the content parser can be instantiated on the receiving side.
So there is no crisis, I just need to add the params, content type enum, and content, and handle it.
You may now return to your regular Friday Eve programming.

Closing the loop here:  #163 will send the XContentType to the extension.   Because the ExtensionRestRequest class is in OpenSearch side of the API, it doesn't make sense to put the contentParser() getter on it.   So I'll go ahead and put #163 back in review.
Part of integrating things (#132) will be finding a new place (probably the BaseExtension) to create the parser where it will have access to the extension's XContentRegistry.

#163 is now merged, so in an extension when processing an ExtensionRestRequest we will have a getXContentType() getter, returning either null, or one of the XContentType enums.  The equivalent of contentParser() is:
// must null-check request.getXContentType()
request.getXContentType().xContent().createParser(xContentRegistry, LoggingDeprecationHandler.INSTANCE, content.streamInput())

To create the xContentRegistry parameter in the above, note the AD Plugin has a getNamedXContent() method already: https://github.com/opensearch-project/anomaly-detection/blob/08fdbdde4c2f97c27f63d3a4ba31f8acd6dbf570/src/main/java/org/opensearch/ad/AnomalyDetectorPlugin.java#L938-L946
Add that to an empty registry as follows
protected NamedXContentRegistry xContentRegistry() {
    SearchModule searchModule = new SearchModule(Settings.EMPTY, Collections.emptyList());
    List<NamedXContentRegistry.Entry> entries = searchModule.getNamedXContents();
    entries
        .addAll(
            getNamedXContent()
        );
    return new NamedXContentRegistry(entries);
}

Creating the registry like the above will probably end up being a function of createComponents(), with the ExtensionsRunner class reading the getNamedXContent() from the extension (default empty list).  cc: @owaiskazi19 @joshpalis @ryanbogan

The current workflow of AD is:

Create a metedata index named .opendistro-anomaly-detectors with mapping associated with body of the request.
Add values for the keys present in the request body

Since we are using SDKClient which is based on OpenSearch java client, it requires mapping in the form of TypeMapping. To convert the json body of the request to TypeMapping, we are currently exploring the below approaches:

(Hacky way) Create properties objects for the json and parse it as TypeMapping - @owaiskazi19
Figure out a way for creating TypeMapping using XContentParser - @dbwiddis

Created AD Index with mapping using JsonParser and JsonPMapper provided by java client
JsonpMapper mapper = sdkClient._transport().jsonpMapper();
        JsonParser parser = null;
        try {
            parser = mapper
                .jsonProvider()
                .createParser(new ByteArrayInputStream(getAnomalyDetectorMappings().getBytes(StandardCharsets.UTF_8)));
        } catch (Exception e) {
            e.printStackTrace();
        }

        CreateIndexRequest request = null;
        try {
            request = new CreateIndexRequest.Builder()
                .index(AnomalyDetector.ANOMALY_DETECTORS_INDEX)
                .mappings(TypeMapping._DESERIALIZER.deserialize(parser, mapper))
                .build();

INFO  org.opensearch.sdk.handlers.ExtensionsIndicesModuleRequestHandler - Registering Indices Module Request received from OpenSearch
12:01:12.716 [opensearch[ad-extension][generic][T#3]] INFO  org.opensearch.sdk.handlers.ExtensionsIndicesModuleRequestHandler - Registering Indices Module Request received from OpenSearch
Oct 25, 2022 12:01:13 PM org.opensearch.client.RestClient logResponse
WARNING: request [PUT http://localhost:9200/.opendistro-anomaly-detectors] returned 1 warnings: [299 OpenSearch-3.0.0-SNAPSHOT-401c21054bae7a716e5f8394fce425d102c6ee00 "index name [.opendistro-anomaly-detectors] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices"]

{
    ".opendistro-anomaly-detectors": {
        "aliases": {},
        "mappings": {
            "_meta": {
                "schema_version": 5
            },
            "properties": {

The next steps are:

Get the Synchronous response from the CreateIndexRequest - @owaiskazi19
Change NodeClient to SDKClient for making request calls - @dbwiddis
Integrate the complete workflow of creating the detector

Was able to create a detector https://github.com/opensearch-project/anomaly-detection/pull/692
{
    "id": "1",
    "version": 1,
    "seqNo": 0,
    "primaryTerm": 1,
    "detector": {
        "name": "test-detector",
        "description": "Test detector",
        "time_field": "timestamp",
        "indices": [
            "server_log*"
        ],
        "filter_query": {
            "bool": {
                "filter": [
                    {
                        "range": {
                            "value": {
                                "from": 1,
                                "to": null,
                                "include_lower": false,
                                "include_upper": true,
                                "boost": 1.0
                            }
                        }
                    }
                ],
                "adjust_pure_negative": true,
                "boost": 1.0
            }
        },
        "detection_interval": {
            "period": {
                "interval": 1,
                "unit": "Minutes"
            }
        },
        "window_delay": {
            "period": {
                "interval": 1,
                "unit": "Minutes"
            }
        },
        "shingle_size": 8,
        "schema_version": 0,
        "feature_attributes": [
            {
                "feature_id": "qfrlRIQBANUbFEg-UqnN",
                "feature_name": "test",
                "feature_enabled": true,
                "aggregation_query": {
                    "test": {
                        "sum": {
                            "field": "value"
                        }
                    }
                }
            }
        ],
        "detector_type": "SINGLE_ENTITY",
        "result_index": "opensearch-ad-plugin-result-test"
    },
    "status": "CREATED"
}

authors: dbwiddis joshpalis owaiskazi19 ryanbogan

repo: opensearch-project/opensearch-sdk-java