text:
Review California Stops Data

California has a state law (RIPA) that requires departments to report traffic stop data. Originally, the state appeared to be hosting data from all the departments so when individual departments hosted this data, was not included in our source table as it was viewed as redundant. In some cases, we include pre-RIPA data in our source table instead.
At some point, California to the RIPA data down. It is now back up but only for a limited number of departments and it's in a form factor (zip file contain Excel files for each department) that is less user-friendly than a lot of the data being shared by individual departments. It would be better to now include data in our source table for individual departments. Need to re-survey California departments for RIPA data and identify departments where we are sharing pre-RIPA data instead of RIPA. We currently have no ability to share multiple datasets for the same type of data where both datasets span multiple years (Will be addressed in #131)

FYI, here is the data in the current RIPA files published by the state, with the count of stops for each bucket. First, is the list grouped by agency, showing all years for each agency. Second is the list grouped by years, showing all agencies for each year.
Are the cities putting up data for different date ranges? Is there a catalog of sources? (Sorry, thinking with my mouth, I can probably find that last.)
 +-------------------------------------+----------+
 | agency_yrs                          | count(0) |
 +-------------------------------------+----------+
 | ALAMEDA CO SO - 2021                |    15505 |
 | ANAHEIM PD - 2021                   |    26568 |
 | ARCATA PD - 2021                    |     1428 |
 | BAKERSFIELD PD - 2020               |    12170 |
 | BAKERSFIELD PD - 2021               |    11948 |
 | BELMONT PD - 2021                   |     1475 |
 | BERKELEY PD - 2021                  |     5469 |
 | CAPITOLA PD - 2021                  |      606 |
 | CARLSBAD PD - 2021                  |     5321 |
 | CHP-HQ A/C, STAFF - 2018            |  1036484 |
 | CHP-HQ A/C, STAFF - 2019            |  2175618 |
 | CHP-HQ A/C, STAFF - 2020            |  1696390 |
 | CHP-HQ A/C, STAFF - 2021            |  1749421 |
 | CONTRA COSTA CO SO - 2021           |     3131 |
 | COTATI PD - 2021                    |     1728 |
 | CSU CHICO - 2021                    |      303 |
 | CULVER CITY PD - 2021               |     9454 |
 | DAVIS PD - 2020                     |     2626 |
 | DAVIS PD - 2021                     |     4545 |
 | EMERYVILLE PD - 2021                |     1665 |
 | EUREKA PD - 2021                    |     2906 |
 | FRESNO CO SO - 2021                 |    19282 |
 | FRESNO PD - 2019                    |    51849 |
 | FRESNO PD - 2020                    |    14738 |
 | FRESNO PD - 2021                    |    10816 |
 | HILLSBOROUGH PD - 2021              |      646 |
 | KERN CO SO - 2021                   |    12277 |
 | LA DEPT OF AIRPORT - 2021           |     4672 |
 | LAPD - 2018                         |   336681 |
 | LAPD - 2019                         |   712807 |
 | LAPD - 2020                         |   521290 |
 | LAPD - 2021                         |   429242 |
 | LAUSD PD - 2020                     |     1150 |
 | LAUSD PD-GRAND - 2021               |      100 |
 | LIVERMORE PD - 2021                 |     4552 |
 | LONG BEACH PD - 2019                |    40524 |
 | LONG BEACH PD - 2020                |    17197 |
 | LONG BEACH PD - 2021                |    11986 |
 | LOS ALTOS PD - 2021                 |      987 |
 | LOS ANGELES CO SD - 2018            |   136635 |
 | LOS ANGELES CO SD - 2019            |   196850 |
 | LOS ANGELES CO SD - 2020            |   104238 |
 | LOS ANGELES CO SD - 2021            |   179888 |
 | MILL VALLEY PD - 2021               |      808 |
 | OAKLAND PD - 2019                   |    24395 |
 | OAKLAND PD - 2020                   |    21076 |
 | OAKLAND PD - 2021                   |    13782 |
 | ORANGE CO SO - 2019                 |    50396 |
 | ORANGE CO SO - 2020                 |    39794 |
 | ORANGE CO SO - 2021                 |    46283 |
 | PETALUMA PD - 2021                  |     3882 |
 | PIEDMONT PD - 2021                  |      639 |
 | POMONA PD - 2021                    |     4594 |
 | RIVERSIDE CO SO - 2018              |    44505 |
 | RIVERSIDE CO SO - 2019              |    58379 |
 | RIVERSIDE CO SO - 2020              |    56257 |
 | RIVERSIDE CO SO - 2021              |    75779 |
 | RIVERSIDE PD - 2021                 |    19262 |
 | ROHNERT PARK PD - 2021              |     2360 |
 | SACRAMENTO CO SHERIFF'S DEPT - 2019 |    60944 |
 | SACRAMENTO CO SHERIFF'S DEPT - 2020 |    43852 |
 | SACRAMENTO CO SHERIFF'S DEPT - 2021 |    33018 |
 | SACRAMENTO PD - 2019                |    68012 |
 | SACRAMENTO PD - 2020                |    51391 |
 | SACRAMENTO PD - 2021                |    46671 |
 | SAN BERNARDINO CO SO - 2018         |    62433 |
 | SAN BERNARDINO CO SO - 2019         |   157715 |
 | SAN BERNARDINO CO SO - 2020         |   108979 |
 | SAN BERNARDINO CO SO - 2021         |    98576 |
 | SAN DIEGO CO SO - 2018              |    40515 |
 | SAN DIEGO CO SO - 2019              |    65029 |
 | SAN DIEGO CO SO - 2020              |    38800 |
 | SAN DIEGO CO SO - 2021              |    21925 |
 | SAN DIEGO PD - 2018                 |    89455 |
 | SAN DIEGO PD - 2019                 |   187231 |
 | SAN DIEGO PD - 2020                 |   150575 |
 | SAN DIEGO PD - 2021                 |   130112 |
 | SAN FRANCISCO CO SO - 2021          |      628 |
 | SAN FRANCISCO PD - 2018             |    56409 |
 | SAN FRANCISCO PD - 2019             |   101614 |
 | SAN FRANCISCO PD - 2020             |    38582 |
 | SAN FRANCISCO PD - DOC - 2021       |    27420 |
 | SAN JOSE PD - 2019                  |    44306 |
 | SAN JOSE PD - 2020                  |    17978 |
 | SAN JOSE PD - 2021                  |    17167 |
 | SANTA ANA PD - 2021                 |    21910 |
 | SANTA BARBARA PD - 2021             |     4343 |
 | SANTA CLARA CO SO - 2021            |    14469 |
 | SANTA ROSA PD - 2021                |     6697 |
 | SONOMA CO JR CLG DIST PD - 2021     |      549 |
 | SONOMA PD - 2021                    |      248 |
 | SONOMA SHERIFF - 2021               |     2569 |
 | SONOMA STATE UNIVERSITY PD - 2021   |      272 |
 | ST UNIV PD-STANISLAUS - 2021        |      202 |
 | STOCKTON PD - 2021                  |    23937 |
 | UC-IRVINE PD - 2021                 |      785 |
 | UC-SAN FRANCISCO PD - 2021          |      535 |
 | VENTURA CO SO - 2021                |    47268 |
 | WINDSOR POLICE DEPARTMENT - 2021    |      582 |
 +-------------------------------------+----------+

 +-------------------------------------+----------+
 | agency_yrs                          | count(0) |
 +-------------------------------------+----------+
 | 2018 - CHP-HQ A/C, STAFF            |  1036484 |
 | 2018 - LAPD                         |   336681 |
 | 2018 - LOS ANGELES CO SD            |   136635 |
 | 2018 - RIVERSIDE CO SO              |    44505 |
 | 2018 - SAN BERNARDINO CO SO         |    62433 |
 | 2018 - SAN DIEGO CO SO              |    40515 |
 | 2018 - SAN DIEGO PD                 |    89455 |
 | 2018 - SAN FRANCISCO PD             |    56409 |
 | 2019 - CHP-HQ A/C, STAFF            |  2175618 |
 | 2019 - FRESNO PD                    |    51849 |
 | 2019 - LAPD                         |   712807 |
 | 2019 - LONG BEACH PD                |    40524 |
 | 2019 - LOS ANGELES CO SD            |   196850 |
 | 2019 - OAKLAND PD                   |    24395 |
 | 2019 - ORANGE CO SO                 |    50396 |
 | 2019 - RIVERSIDE CO SO              |    58379 |
 | 2019 - SACRAMENTO CO SHERIFF'S DEPT |    60944 |
 | 2019 - SACRAMENTO PD                |    68012 |
 | 2019 - SAN BERNARDINO CO SO         |   157715 |
 | 2019 - SAN DIEGO CO SO              |    65029 |
 | 2019 - SAN DIEGO PD                 |   187231 |
 | 2019 - SAN FRANCISCO PD             |   101614 |
 | 2019 - SAN JOSE PD                  |    44306 |
 | 2020 - BAKERSFIELD PD               |    12170 |
 | 2020 - CHP-HQ A/C, STAFF            |  1696390 |
 | 2020 - DAVIS PD                     |     2626 |
 | 2020 - FRESNO PD                    |    14738 |
 | 2020 - LAPD                         |   521290 |
 | 2020 - LAUSD PD                     |     1150 |
 | 2020 - LONG BEACH PD                |    17197 |
 | 2020 - LOS ANGELES CO SD            |   104238 |
 | 2020 - OAKLAND PD                   |    21076 |
 | 2020 - ORANGE CO SO                 |    39794 |
 | 2020 - RIVERSIDE CO SO              |    56257 |
 | 2020 - SACRAMENTO CO SHERIFF'S DEPT |    43852 |
 | 2020 - SACRAMENTO PD                |    51391 |
 | 2020 - SAN BERNARDINO CO SO         |   108979 |
 | 2020 - SAN DIEGO CO SO              |    38800 |
 | 2020 - SAN DIEGO PD                 |   150575 |
 | 2020 - SAN FRANCISCO PD             |    38582 |
 | 2020 - SAN JOSE PD                  |    17978 |
 | 2021 - ALAMEDA CO SO                |    15505 |
 | 2021 - ANAHEIM PD                   |    26568 |
 | 2021 - ARCATA PD                    |     1428 |
 | 2021 - BAKERSFIELD PD               |    11948 |
 | 2021 - BELMONT PD                   |     1475 |
 | 2021 - BERKELEY PD                  |     5469 |
 | 2021 - CAPITOLA PD                  |      606 |
 | 2021 - CARLSBAD PD                  |     5321 |
 | 2021 - CHP-HQ A/C, STAFF            |  1749421 |
 | 2021 - CONTRA COSTA CO SO           |     3131 |
 | 2021 - COTATI PD                    |     1728 |
 | 2021 - CSU CHICO                    |      303 |
 | 2021 - CULVER CITY PD               |     9454 |
 | 2021 - DAVIS PD                     |     4545 |
 | 2021 - EMERYVILLE PD                |     1665 |
 | 2021 - EUREKA PD                    |     2906 |
 | 2021 - FRESNO CO SO                 |    19282 |
 | 2021 - FRESNO PD                    |    10816 |
 | 2021 - HILLSBOROUGH PD              |      646 |
 | 2021 - KERN CO SO                   |    12277 |
 | 2021 - LA DEPT OF AIRPORT           |     4672 |
 | 2021 - LAPD                         |   429242 |
 | 2021 - LAUSD PD-GRAND               |      100 |
 | 2021 - LIVERMORE PD                 |     4552 |
 | 2021 - LONG BEACH PD                |    11986 |
 | 2021 - LOS ALTOS PD                 |      987 |
 | 2021 - LOS ANGELES CO SD            |   179888 |
 | 2021 - MILL VALLEY PD               |      808 |
 | 2021 - OAKLAND PD                   |    13782 |
 | 2021 - ORANGE CO SO                 |    46283 |
 | 2021 - PETALUMA PD                  |     3882 |
 | 2021 - PIEDMONT PD                  |      639 |
 | 2021 - POMONA PD                    |     4594 |
 | 2021 - RIVERSIDE CO SO              |    75779 |
 | 2021 - RIVERSIDE PD                 |    19262 |
 | 2021 - ROHNERT PARK PD              |     2360 |
 | 2021 - SACRAMENTO CO SHERIFF'S DEPT |    33018 |
 | 2021 - SACRAMENTO PD                |    46671 |
 | 2021 - SAN BERNARDINO CO SO         |    98576 |
 | 2021 - SAN DIEGO CO SO              |    21925 |
 | 2021 - SAN DIEGO PD                 |   130112 |
 | 2021 - SAN FRANCISCO CO SO          |      628 |
 | 2021 - SAN FRANCISCO PD - DOC       |    27420 |
 | 2021 - SAN JOSE PD                  |    17167 |
 | 2021 - SANTA ANA PD                 |    21910 |
 | 2021 - SANTA BARBARA PD             |     4343 |
 | 2021 - SANTA CLARA CO SO            |    14469 |
 | 2021 - SANTA ROSA PD                |     6697 |
 | 2021 - SONOMA CO JR CLG DIST PD     |      549 |
 | 2021 - SONOMA PD                    |      248 |
 | 2021 - SONOMA SHERIFF               |     2569 |
 | 2021 - SONOMA STATE UNIVERSITY PD   |      272 |
 | 2021 - ST UNIV PD-STANISLAUS        |      202 |
 | 2021 - STOCKTON PD                  |    23937 |
 | 2021 - UC-IRVINE PD                 |      785 |
 | 2021 - UC-SAN FRANCISCO PD          |      535 |
 | 2021 - VENTURA CO SO                |    47268 |
 | 2021 - WINDSOR POLICE DEPARTMENT    |      582 |
 +-------------------------------------+----------+

@rkiddy Thanks, this is a big help on where to look for additional California stops data.
The difference in date ranges is because the RIPA law was phased in. The largest agencies had to start reported first and more agencies report each year.
Are you asking about our catalog of sources? That is here.
If you'd like to provide further help on this issue, there are a few other items (which help on any of them would be valuable):

We currently are not able to load the state RIPA data into our tool. The data is stored as multiple Excel or CSV files inside a zip file. We need a new data loader that can extract data from this case so that we can add the RIPA data to our source list.
Some police departments post data on their own sites in addition to it being posted by the state. These datasets tend to be better because (A) they don't require downloading the entire state's data to access and (B) sometimes they add additional fields to the data. We haven't looked through all the locations that you have listed to see if their data can be accessed from their own website. You can see the locations that we have already checked here.

By the way, there was an error regarding a small detail in the original issue statement. I have corrected that.

I knew about the RIPA law phase-in.
As far as a source for the CA RIPA data, I have put the data into mysql tables. Can that be used as a data source? I have an app sitting right next to the data which is not doing much. It can, though, provide REST services for, for example, an agency, or an agency-year.
The code to read the spreadsheet and put it into mysql was not very much. It was just a bother to set up and it takes time to run. See https://github.com/rkiddy/ca_ripa for the code.
I would be interested to see what LEOs in CA are publishing themselves. I suspect that, yes, the data gets put up quicker and may be in an easier-to-use format, but for the long run, it will be better to use the RIPA data. RIPA data includes all data, even the data that the LEO does not want to put up. And it is a standard that will represent the data of all 58 counties in California.

FYI, here are the places in CA that we have data from more from more than one source. All of the others (above) are RIPA-only.
 ANAHEIM PD - STOPS - 2021-01-01 - 2021-12-31 - CA RIPA
 Anaheim PD - TRAFFIC STOPS - 01/01/2012 - 03/14/2017 - Stanford

 BAKERSFIELD PD - STOPS - 2020-01-01 - 2021-12-31 - CA RIPA
 Bakersfield PD - TRAFFIC STOPS - 03/09/2008 - 03/09/2018 - Stanford

 CHP-HQ A/C, STAFF - STOPS - 2018-07-01 - 2021-12-31 - CA RIPA
 California HP - TRAFFIC STOPS - 07/01/2009 - 06/30/2016 - Stanford

 LONG BEACH PD - STOPS - 2019-01-01 - 2021-12-31 - CA RIPA
 Long Beach PD - STOPS - 01/01/2008 - 12/31/2017 - Stanford

 OAKLAND PD - STOPS - 2019-01-01 - 2021-12-31 - CA RIPA
 Oakland PD - STOPS - 04/01/2013 - 12/31/2017 - Stanford

 SAN DIEGO PD - STOPS - 2018-07-01 - 2021-12-31 - CA RIPA
 San Diego PD - TRAFFIC STOPS - 01/01/2014 - 03/31/2017 - Stanford

 SAN FRANCISCO PD - STOPS - 2018-07-01 - 2020-12-31 - CA RIPA
 San Francisco PD - TRAFFIC STOPS - 01/01/2007 - 06/30/2016 - Stanford

 SAN JOSE PD - STOPS - 2019-01-01 - 2021-12-31 - CA RIPA
 San Jose PD - STOPS - 09/01/2013 - 03/31/2018 - Stanford

 SANTA ANA PD - STOPS - 2021-01-01 - 2021-12-31 - CA RIPA
 Santa Ana PD - STOPS - 06/11/2014 - 04/13/2018 - Stanford

 STOCKTON PD - STOPS - 2021-01-01 - 2021-12-31 - CA RIPA
 Stockton PD - TRAFFIC STOPS - 01/01/2012 - 12/31/2016 - Stanford

Your points about the RIPA data being a better data source make sense. Perhaps if we are putting up additional stops data for CA agencies, there should be a good reason:

It's released more quickly
It provides additional time coverage (like those Stanford datasets)
It provides more information (data not part of RIPA...unlikely because RIPA has a lot of requirements)

As far as a source for the CA RIPA data, I have put the data into mysql tables. Can that be used as a data source? I have an app sitting right next to the data which is not doing much. It can, though, provide REST services for, for example, an agency, or an agency-year.

We do really need to have the RIPA data available. We have discussed alternative ways for hosting data before for some cases where PDs took data down. This could be a good trial run since the data on Open Justice isn't very user-friendly.
I would say that we can make most data that is available online in a machine-readable format into a data source. We would just need to write a new class to handle that case, which is not a big issue. How to put data online is really outside my area of expertise. When you say that you have an app right now that can provide REST services, does that mean that someone could already make API requests from it or that you have a tool that would provide REST services but would need to be hosted somewhere (or something else, I really don't know a lot about this part)?
Adding @potto216 to see if he has any thoughts/comments.

@rkiddy Just following up. I was hoping to hear more about your tool.

@sowdm Thanks. I run several flask applications at opencalaccess.org but I am having problems at the moment. My existing apps are working but new deployments are not. I am putting some TLC into my server this weekend.
I have a bog simple REST app for the RIPA data. It has parameters of {year} and {agency} and it returns a big bunch of JSON data and that is it. I have a few other ideas for searchers but will do those after I have the basic things up.

@rkiddy Hey, just following up to see if you ever got your flask application for the CA stops data back up and running

authors: rkiddy sowdm

repo: openpolicedata/openpolicedata


text:
ADD: Add initial schedule

Add an initial schedule

@kmuehlbauer - I addressed those suggestions in the latest version 👍

Thanks @mgrover1!

source: gharchive/pull-request

license_type: permissive

license_source: github-api


text:
Rename to information-sharing-agreement register

From data-sharing-agreement register.  Per custodian's request.

Good catch.  I think it's worth doing now so it can be shared, and that takes the pressure off finalising the descriptions.  We'll probably get some more feedback on them anyway.

I've updated the descriptions @arnau

repo: openregister/registry-data

license: mit

license_source: bigquery


text:
Add yaml file for StashCache 1.0.2 tests and remove StashCache 1.0.0 run

Test run in progress...

http://vdt.cs.wisc.edu/tests/20190325-1801/packages.html

repo: opensciencegrid/vm-test-runs


text:
Unable to run on up-to-date Mac

On up-to-date OSX it doesn't seem possible to run pymagicc--- is this correct?
The problem is that 32-bit applications (including wine) are no longer supported (see: https://superuser.com/questions/1513877/bad-cpu-type-in-executable-wine-macos-catalina).
If this is the case, could you update the README to make this apparent up front? Otherwise, are there any workarounds?

Does it work with the MAGICC7 binary available here? https://magicc.org/download/magicc7
See further down in the README on how to use a different binary.

That does seem to be the correct workaround. We weren't able to get it to work, though for seemingly unrelated reasons (libgfortran wasn't linked properly and we gave up). I would request updating the README for Macs since using the provided binary will be necessary for all Macs going forward.

Having the same issue, both libgfortran is not linked and struggling with setting the environment variable to use 'magicc-darwin-arm64' in MAGICC7. An updated readme would be really helpful, thank you!

@Aq0amancer and @grahamas, not sure if you're still experiencing the issue with libgfortran, but there's the steps I recently took to get it to work on macOS Sequoia:
# ensure gcc is installed as that provides libgfortran
brew install gcc

# find path to libgfortran.5.dylib
find /opt/homebrew -name "libgfortran*.dylib"

# use environment variable to specify the path to libgfortran
# in my case, my path looked like the following
export DYLD_LIBRARY_PATH=/opt/homebrew/Cellar/gcc/14.2.0_1/lib/gcc/current:$DYLD_LIBRARY_PATH

# now you should be able to use the magicc7 executable
./magicc-darwin-arm64

For using pymagicc, you'll also need the MAGICC_EXECUTABLE_7 env var as mentioned at the bottom of the README. I have the following cell in my python notebook before using pymagicc:
%env DYLD_LIBRARY_PATH=/opt/homebrew/Cellar/gcc/14.2.0_1/lib/gcc/current:$DYLD_LIBRARY_PATH
%env MAGICC_EXECUTABLE_7=/path/to/magicc/magicc-v7.5.3/bin/magicc-darwin-arm64

source: gharchive/issue

authors: Aq0amancer grahamas rgieseke seangtkelley

repo: openscm/pymagicc


text:
RFE: automatic deployment using Saltstack

Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature
What you expected to happen:
According to #60,  "automatic deployment and installation become more and more important in a distributed system cluster. As a cloud-native storage system, opensds controller project MUST support (at least one of) automatic deployment
I'm interested in storage management, backup/restore, & SDS standards (i.e DMTF SMI-S, OpenLMI).
I want to work on automatic deployment of OpenSDS via saltstack.
Do we have list of use case to satisfy?
Anything else we need to know?:
I'm thinking of publishing an opensds-formula upstream, and integration instructions can go here.
Environment:

Hotpot(release/branch) version:
OS (e.g. from /etc/os-release): Linux/Darwin Containerized.
Kernel (e.g. uname -a): any supported
Install tools: saltstack
Others:

Hi, firstly thanks for your precious suggestion, and we are glad to see someone join and contribute in this community. Here I have some questions and I would appreciate if you can give some answers:

Can you give a simple introduction about yourself? (Who you are, which company do you work for)
What do you expect from this project? (ie. you want to utilize the standard, or make it a product)
Where do you prefer to do code development, under this project or in other communties?

Thanks!

Hey @leonwanghui,  I working for Overstock.com based in Salt Lake City, but I'm here as hobbyist. As mentioned, I am interested in storage management, service availability, and systems automation. My primary skills are systems integration and good knowledge of Next Generation Operations Systems Support (OSS), Backup and Restore requirements and solutions.   I found OpenSDS today when reviewing  Stor4NFV and believe software-defined storage management and orchestration is somewhat neglected - so I'm interested in the concept.
I would like to start code development on opensds-formula for saltstack, to start building my competence in OpenSDS, and thereafter I would hope to be contributor to this community.
Thanks

@noelmcloughlin That's great!  Thanks for your interest in contributing to OpenSDS!  Let us know how it goes with the opensds-formula and if you have any questions.

Welcome @noelmcloughlin!
Just a thought for everyone - if we are going to eventually get multiple ways to deploy OpenSDS like this, would it make sense to create repos for each. Something like opensds-salt, opensds-ansible, etc. I'd be interested to know folks thoughts on how to best structure things long term.

There is an official repo for SaltStack formulas.  Usually formulas get stored over there.  I think we can fork that repo instead of creating another repo here.

https://github.com/saltstack-formulas

Thanks @leonwanghui @xing-yang @stmcginnis pls let me know how to join the community as contributor, etc.
Yes, opensds-formula can be stored in the official repo.  Does anyone know if there is official repo for ansible - I searched once but it was unclear to me.

Oh, thanks @xing-yang. That makes sense to keep it where other salt formulas are kept. I wasn't aware of that (very limited salt experience).
I don't think there is an equivalent for ansible. At least none that I have come across so far.

@nolemcloughlin Hi, currently there are two deployment ways:

ansible script deployment for users

shell script deployment for developers

BTW, they will both be moved to standalone project after getting permitted by TSC.

Let me know when project is ready to be forked upstream to https://github.com/saltstack-formulas and I'll fork it

Thanks Nitin!

Thanks, @nmadhok  Just finishing other saltstack work, then starting this.

@noelmcloughlin @nmadhok Great, please be free to to ask if you need any help.

Hi @noelmcloughlin @nmadhok, recently I'm working on proposing a new idea about how to simplify opensds deployment and configuration (see the link). And I think it's closely related to what you are working, so if you are interested I think we could submit a propsal that creates a new code repo to do code development onto TSC meeting. And if it gets permitted, I wish one of you could serve as PTL of this project.
Any thoughts, questions or suggestions?

I believe saltstack-formulas is where users of Salt expect to go to be able to find formulas. I don't have much Salt experience, so correct me if I'm wrong.
If that is the case, then even though this would be OpenSDS specific, I'm not sure if being under the opensds namespace would be the best option. I suppose we could look into mirroring the same repo between the two, but I don't believe there would be an automatic way to do that.
Another option would be to just have the opensds.io website link to this from some place where potential users could discover it.
So not sure what would be best here with not really knowing how the SaltStack ecosystem works. Any recommendations @noelmcloughlin?

I just chatted with @nmadhok and he said he is interested in Leon's proposal.  Salt formulas do have an official repo, but Nitin said we can fork it to make sure it is in sync.  Leon's proposal also includes other deployment tools such as Ansible and Helm.  For Helm, there's also an official repo so we probably need to fork it as well.  We actually already talked about creating a separate repo to host all OpenSDS deployment tools and TSC has already approved that.  The only thing we have not discussed is whether to have a PTL for that, but I think we can bring it up again since it seems that there's interest from the community.  Waiting to hear back from @noelmcloughlin as well.  Thanks.

@xing-yang IMO there is no need to fork Ansible and Helm official repo into opensds (but we can do it for communities collaboration), because we can directly download binary-built tools from them and run them without copying the source code. So (if I get it right) we could do the same thing to develop the opensds-formulas code in opensds-installer repo and install it by calling the saltstack tool, which is the best option for avoiding the inconsistency between opensds releases and opensds-formulas upstream. But if there are some problems to enforce it in this way, then we can consider the current option that keeps opensds source code and opensds-formulas separate.

@leonwanghui My understanding is that it's better to store salt-formulas under the official Salt repo, but I'll leave that to you, @nmadhok, and @noelmcloughlin to decide what's the best way.

@xing-yang, I'm a bit confused by the statement "talked about creating a separate repo to host all OpenSDS deployment tools". Is there a plan to have one repo with all types of deployment tooling in it?
I would find it confusing, or at least pretty odd, to have one repo that contained ansible, helm, puppet, etc. modules, all in one place.
In this case, I think we need to have it where saltstack users are used to going to find formulas. But the other ones I think would make more sense as separate repos for opensds-ansible, opensds-helm, opensds-pickatool.

@stmcginnis Currently we have multiple deployment tools in both Sushi and Hotpot.  Members from EUAC and TSC told us it is hard to navigate and find out which deployment tool to use.  Leon had proposed to have a common repo and move all the deployment tools in it so that user can just go to one place to install OpenSDS.  This was discussed in previous Technical Meetings and TSC meeting and everyone was onboard with that.  So this is basically to move deployment tools we already have into a common place.
I know OpenStack has different projects for different install tools, but OpenSDS is at a very early phase compared to OpenStack.  Right now OpenSDS is still a very small community so we are trying to consolidate resources and not to have too many repos and projects.

Thanks for the excellent feedback, the comments are interesting.
@leonwanghui You are proposing 'SDSDS-d' ('software defined [open] SDS deployer")? :smiley:  Abstraction from underlying deployment technologies is important aspiration, so I support your proposal.  And @xing-yang confirms TSC agreement that the code be hosted here.
@stmcginnis, I agree ansible, puppet, helm, saltstack, etc, integrations with OpenSDS may live upstream.  I original propose to fork an OpenSDS formula into official saltstack-formulas but it can be forked anywhere on github.com, including unofficial salt-formuls repo, if someone wishes.
My understanding therefore, is that @leonwanghui's  proposedOpenSDS-d (deployer) component will be part of this project, and deliver integration with ansible/salt/helm components forked into some  stable repo from a unstable/development repo, possibly a Dev branch here or upstream?

This site appears to be an official Ansible hub for "formulas".

@noelmcloughlin This is great, I think we can develop the opensds-ansible code under opensds repo, and publish the release into this Ansible Hub.

I think a better name isopensds-deployer component/project/repo - the deployment use cases will be- install, upgrade, and uninstall, in the first iteration.  Later the Controller may require API support for other related use cases- downgrade, and backup, for example.
I think the opensds-salt code can be lightweight.  The following upstream building blocks could be integrated with opensds-salt code base, is my initial thought-

packages formula
This has native Linux package support, and support for newer snap packaging technology. Helm could be added here.
openssh
Keys and permitRootLogin
etcd formula
Distributed keystore
ceph formula
Software defined storage dependency
docker formula
Depoy containers / compose

I'm am rewriting the etcd-formula at the moment and will raise a PR with integration with Linux, MacOS, and Windows.  @nmadhok you are more experienced with Saltstack - what are your thoughts?
thanks.

Hi @nmadhok @noelmcloughlin , just want to know how is going on about your code development of opensds-salt : )

Last week I worked on PR for etcd-salthere, since openSDS needs etcd. Ceph may will need work too.   Next few weeks are very busy time for me until after April 20th. will examine ceph next.

@noelmcloughlin @nmadhok Hi, this is just a warm remind that the opensds-installer code repo has been created and ansible and helm deploy scripts have been moved into this repo. Please be free to push it if your code is ready. Thanks!

The etcd-salt pull request was merged.   I'm going to look at opensds-ansible next and Ceph next.  My intention is to reuse ceph-formula but another extremely interesting option is DeepSea.  I need to do some further research.

Hi @leonwanghui  Sorry, no further updates. do you want to close this issue until a design/Pr proposal is ready?

@noelmcloughlin It's ok to leave it here, I'm just wondering if this feature can be finished by late June, since we would publish the Aruba release then.

hi @leonwang, it cannot be planned into Aruba, but definitely the following
release.
On Tue, May 22, 2018 at 1:40 AM, Leon Wang notifications@github.com wrote:

@noelmcloughlin https://github.com/noelmcloughlin It's ok to leave it
here, I'm just wondering if this feature can be finished by late June,
since we would publish the Aruba release then.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/opensds/opensds/issues/320#issuecomment-390827103,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMtKQlPyR32-AOQOD8wWArcVZ0quG1izks5t016FgaJpZM4St_Ct
.

Cool!

@nmadhok  could you please fork DeepSea formula to saltstack-formulas community please?  This might be useful for salt-opensds installer. Thanks.

Hi all,  the following is a brief progress update on prototyping opensds deployment via Salt.
OpenSDS Cluster Installation through SaltStack progress report.
The following prototyping has been completed on GNU Linux (Centos and Suse) cluster.  Refer to the attached logs for pseudo-design and prototyping logs.

Bootstrap the SDS controller (bash script)  OK
Review yaml models   OK
Deploy SDS Controller Node (bash command to deploy salt-masters)   OK
Deploy SDS payloads  (bash command to deploy salt-minions)   OK

The following prototypes are outstanding:

Configure opensds cluster variables (Ceph, Cinder, LVM).  For CEPH backend, I think DeepSea is best solution right now. I have engaged with the upstream project (issues & pull requests).
How to test opensds cluster
How to purge and clean opensds cluster

I hope to open a pull request on opensds-installer dev branch in one-two weeks.  A demo maybe possible  in next few weeks.
Prototyping logs:
opensds-salt-poc1-notes.txt
Thanks for your patience.
N.

Awesome, thanks @noelmcloughlin!

Hi @noelmcloughlin, that is great progress!  Looking forward to seeing your PR.  Thanks!

@noelmcloughlin Well done, and BTW you can just open a PR on opensds-installer master branch directly.

I have an outline framework for opensds-installer/salt support  completed. Please see this ...

Thanks @noelmcloughlin !  If you have tested your PR, please go ahead and submit it so we can review it.  I noticed that you referenced your own fork in the PR.  That should be changed when you submit it?

PR: https://github.com/opensds/opensds-installer/pull/13   I referenced my own fork for DeepSea (aka openATTIC) because that upstream project needs some work for GNU Linux / Distro agnostic solution.

Thanks @noelmcloughlin!
@nmadho can you help review this PR? https://github.com/opensds/opensds-installer/pull/13.  Thanks!

Would close it : )

authors: leonwanghui nmadhok noelmcloughlin stmcginnis xing-yang

repo: opensds/opensds


text:
Patch for #1303

I saw that #1303 was "up for grabs" and simple enough to just get it done. So here it is.

Sounds good to me! I saw that the options were documented but wasn't sure if there was reason or if it was documented elsewhere already. Will do.

@joedf Hmm... Now that I've merged it, I'm realizing there's an additional missing feature. Most of the time, people create navigators by using the navigatorId option when creating their viewer. This gets passed on to the navigator here:
https://github.com/openseadragon/openseadragon/blob/master/src/viewer.js#L434
This patch makes it possible to pass an element into the navigator, but what we really need is a navigatorElement option for the viewer. Would you be up for making a follow-up PR to add that? It would need to be added to the area linked above as well as here:
https://github.com/openseadragon/openseadragon/blob/d956b2ae9dcc3845e31a5183256d3fe99515a8cf/src/openseadragon.js#L413

Sounds good, I'll have a look!

authors: iangilman joedf

repo: openseadragon/openseadragon


text:
Document reason for datastream timestamp field name

Is your feature request related to a problem? Please describe.
I have tried looking at the code and the documentation for datastream. I can not find documentation that describes why a timestamp field is needed on each document.
Rollover is based on index timestamp and each new index is created with a generationId. I can not spot anywhere the timestamp on a document has a technical impact.
Describe the solution you'd like
Update https://opensearch.org/docs/2.10/im-plugin/data-streams/ with a technical reason for the timestamp requiremement on documents.
Describe alternatives you've considered
Make the field optional if it has no functional impact

@opensearch-project/admin Please transfer this to https://github.com/opensearch-project/documentation-website. Thanks!

authors: cthtrifork noCharger

repo: opensearch-project/OpenSearch


text:
[BUG] FailEngine does not trigger ShardFailure

Describe the bug
In production environment, we find our cluster can not recover from a disk failure.
Although the cluster is green, bulk_requests on some shard fail because of AlreadyClosedException.

It seems like the InternalEngine is closed, but ShardFailure is not sent to master.
The root cause is that when failEngine is called by a FlushFailure, markStoreCorrupted() in failEngine() throws one java.nio.file.DirectoryIteratorException that can't be caught, as DirectoryIteratorException is not a IOException.

DirectoryIteratorException is directly thrown out, and eventListener.onFailedEngine(reason, failure); can't be executed
The result is that the index can't process bulkRequest, and master won't reroute the this shard as shardFailure is not reported.
Related component
Cluster Manager
To Reproduce

Manually mock a  DirectoryIteratorException when markStoreCorrupted() is called

Expected behavior
ShardFailure can be sent to master when filesystem throws DirectoryIteratorException during failEngine.
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):

OS: [e.g. iOS]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Thanks @jrj0823 for raising the issue and pull request. will get back on it.


text:
[Segment Replication] Remote store integration high level design.

Before starting a POC for this integration we should start with a high level design.
Some questions that come to mind.

How do we compute and send the list of segments to fetch from the store?
How do we fetch the SegmentInfos byte[] required?
Are there any changes to failover steps?  Currently we commit on replicas during failover and replay from xlog, can we ensure the newly selected replica always has the latest segments from store before promotion?
How would we guarantee segments are present before fetching from remote store?
Should we use existing push segrep design or have replicas poll remote store for updates?  If so, how would this work?

lower lvl questions:

Do we still need commit points on replicas?
Can we now disable fsyncs on primaries?

Also, How do we ensure that the segments which are being copied by replicas doesn't get deleted by primary?

Based on discussion on writes on NRT replica with translog, it does not look like primary and replicas are completely decoupled for translog writes. What would we then gain by decoupling primary and replicas completely during segment copy? I believe that we should start from this as that will inform and potentially simplify a lot of the design trade-offs you captured above.

Should we use existing push segrep design or have replicas poll remote store for updates? If so, how would this work?

I definitely think we should valuate a poll based model for checkpoints and segment transfer on replica, this should simplify cases not only with remote segment store but also with CCR use cases where we currently have remote cluster poll for operations on the leader cluster.

How do we compute and send the list of segments to fetch from the store?

Now if we go with polling mechanism the replicas can request for segments from the primary or the remote store alike based on the diff between the latest checkpoint and the last processed checkpoint

Are there any changes to failover steps? Currently we commit on replicas during failover and replay from xlog, can we ensure the newly selected replica always has the latest segments from store before promotion?

I guess we should do that @sachinpkale would be sharing the related issue around this, translog replay might be expensive
@itiyamas I guess remote translogs would only be relying on a replication proxy for the control flow(single writer invariant), the data flow however is decoupled. This is only to support existing blob stores which don't support OCC but eventually the goal is to provide extension points for stores that support OCC for single writer use cases which might totally decouple the translog flow in such cases.
Decoupling primary and replica for segment copy will help optimise lots of cases around recovery, primarily deployments/failovers, CCR replication etc. With remote store we could also think about publishing checkpoints to a remote queue like Kafka, ActiveMQ which should effectively decouple the segment copy and can be consumed not only internally within the cluster but across clusters for CCR with the appropriate access controls.

I guess we should do that @sachinpkale would be sharing the related issue around this, translog replay might be expensive

Created an issue to track this: https://github.com/opensearch-project/OpenSearch/issues/4578. Downloading segments for lagging replica from remote segment store would definitely be faster than replaying all the operations from translog.
We may need to think about common abstraction which will be used to get segments downloaded from remote segment store. Once SegRep is integrated with remote segment store, we can use the same APIs to download segments during failover.

@itiyamas I guess remote translogs would only be relying on a replication proxy for the control flow(single writer invariant), the data flow however is decoupled. This is only to support existing blob stores which don't support OCC but eventually the goal is to provide extension points for stores that support OCC for single writer use cases which might totally decouple the translog flow in such cases.

Do you mean that primary term check is control flow and actual document or segments is data flow?

Do you mean that primary term check is control flow and actual document or segments is data flow?

Yes thats how I would look at it :)

Thank you folks for the inputs. The comments really helped in identifying different problem areas and the probable solutions.
From POC perspective, pull based model where replica directly polls the remote store for new replication checkpoints seems better bit. The data deletion from remote store is not considered as part of this POC though remote store currently purges older than N commit points. Also, there are few changes which may needed on remote store side, created issue for tracking https://github.com/opensearch-project/OpenSearch/issues/4628.

Design consideration
Replication model
Pull based mechanism provides certain advantages, works well with CCR issue (https://github.com/opensearch-project/OpenSearch/issues/4577), and provide simplication comment (https://github.com/opensearch-project/OpenSearch/issues/4555#issuecomment-1255940855). It is better to provide both options and evaluate based on benchmarking results. As part of this POC, poll based mechanism will be attempted.
1. Push based mechanism
Primary in charge of starting replication by sending shard refresh checkpoints. Replica compares these checkpoints with its store and initiates replication when it falls behind. Primary maintains state of on-going replications to prevent data deletion. This is existing approach.
Pros

Existing design, may need minor changes for remote store integration

Cons

Does not gel well with CCR.

2. Pull based mechanism
Replica is in-charge of updating their local copy of data. Replica pings remote store to check if there is any new data. Need to trade off with network consumption Vs how soon data is available.
Pros

Works well with CCR
Aligns better with de-coupling primary (writer) and replica (reader) shards.

Cons

Not implemented right now. May need more effort.

Data purging
Data once stored on remote store needs to be cleaned up as well. This deletion shouldn't impact the on-going file copy operations on replica node. From POC perspective, data will not be deleted from remote store. Below are some options which can be considered as future enhancements.
1. Background process on primary
On replica node, a background process ping replicas to fetch on-going replication checkpoints and cleans up commit points older than N, where N is least replication checkpoint active in replication amont all replicas. This approach couples primary and replica nodes and is not a scalable solution
2. Distributed queue
Similar to previous approach, but instead of transport calls from primary to replica, primary uses a distributed queue to synchronize active replication checkpoints. When event is processed by all replicas, event can be purged (exact N semantics) and cleaned up from remote store.
Pros:

Primary & replica nodes are decoupled

Cons:

With queue storage on primary, leaves memory & storage footprint on primary
Need to identify behavior on small grade instances (t2/t3)

3. Deletion policies ( cleanup after M days, or keep only last N commits)
No synchronization among nodes to identify active replication checkpoints. Data is instead deleted based on data deletion policy. This completely decouples primary and replica nodes and simplify the design. Remote store by default keeps last 10 commits on store. Replica nodes handle file not found exception and retry for latest commit point. The solution can result in replica starvation when primary is aggressively committing. This solution is attempted for POC and doesn't need any extra work on integration side.

Overall proposal looks good to me. Few open questions that we probably need to elaborate.

How often do we want to commit? Wouldn't it make the consistency worse?
Can we also expand on the the control flow for replica to access primary's remote store?
Just to confim, for pull based method, we'd be syncing Replication checkpoint to the remote store as well right?

With remote store, where fsyncs are not mandatory on primary, it is better to perform a commit rather than refresh

Does this replace refresh operation with commit? Commit also performs other operations like translog purge. Let's take a look at all the operations involved in commit flow.
Also, this would require recovery flow to be integrated with remote store. Otherwise, recovery can cause data loss.

Thanks @ankitkala for the comments. Please find my response below.

How often do we want to commit?

The commit is performed on shard refreshes.

Wouldn't it make the data consistency worse between primary and replicas?

Can you please elaborate more on this ?

Can we also expand on the the control flow for replica to access primary's remote store?

I am thinking of having a poller process on replica nodes which scans remote store for new commit point. Poller initiates replication process whenever it sees a new commit point. Replica then start to copy files related to the latest commit point.

Just to confim, for pull based method, we'd be syncing Replication checkpoint to the remote store as well right?

I think the poller process on replica should be able to identify when a new commit point is added and trigger replication events on replica. With this, I think replication checkpoints will not be needed.

The existing implementations of both segrep & remote store operate on refresh, not commit.  So in order for replicas to fetch the latest checkpoint (which is only in memory on the primary) we'd need to create a new file representing the in memory SegmentInfos at a refresh point and write/upload it.  Right now with the node->node implementation that is sent directly from memory as a byte[].  The idea to create an fsyncless commit instead of the normal refresh and avoid creating the new file.  The fsync removal makes this about as expensive as a regular refresh, which we are able to do withe the durability guarantee of the remote store.
So the flow would be.
Commit -> push to remote store -> on upload ack purge the xlog & push checkpoint to replicas.
The alternative is we replicate after normal commit which will cause unacceptable staleness.

We need to make sure we are not breaking the refresh semantics with remote store. So in essence its not one vs the other, we need to ensure we add features without changing staleness

Err yeah, with what we are proposing, the refresh operation would need to be extended when these features are enabled to effectively execute what a flush does today without the xlog purge.  This should not impact staleness however.

I think we also need to think from a failover perspective, we cannot have commits only in memory as it will increase the failover time and impact availability.

Are you referring about the absence of fsync here or the need to write the incremental refresh's SegmentInfos to disk if we are pushing a new file to the store?

Refresh Vs Commit

We need to make sure we are not breaking the refresh semantics with remote store. So in essence its not one vs the other, we need to ensure we add features without changing staleness

Thank you @Bukhtawar for the feedback and calling this out. Yes, there will be changes in refresh mechanism, but the advantages we seek from commit only backup seems to outweigh this short-coming as also mentioned in last comment by @mch2. We plan to call this out clearly in the documentation and perform benchmarking/tests to verify the claim.

Avoid update disk store

I think we also need to think from a failover perspective, we cannot have commits only in memory as it will increase the failover time and impact availability.

In order to identify the failover time and availability, we need to add the tests/benchmark. We can make fsync-less commits bounded in order to keep failover/recovery time bounded. One way is to ensure is to ensure at least last N commit is fsynced.

Deletion policy

A reasonable retention policy should be fine as long as there is no additional cost or trade-offs

The deletion policycleanup after M days, or keep only last N commits de-couples primary and replica. The remaining approaches need some sort of collaboration among nodes. For POC, I am planning to rely on existing remote store cleanup policy which keeps only last 10 commits. With PITR (14 days data retention) implementation on remote store side, may be sufficient

Closing this issue and POC work is tracked in https://github.com/opensearch-project/OpenSearch/issues/4536

authors: Bukhtawar ankitkala dreamer-89 itiyama mch2 sachinpkale


text:
[Enhancement] Verify the chunk size of the snapshot when restoring a searchable snapshot

Description
Verify that the chunk size of a snapshot used in the construction of RemoteSnapshotDirectory is a multiple of the block size. If it is not, an IllegalArgumentException will be thrown.
Related Issues
Resolves #11739

Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[x] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Thanks @bugmakerrrrrr! This is definitely an improvement on the existing (broken) behavior. We do have #9676 to track this issue. I believe a more comprehensive fix is to make the code work for any chunk size. Have you looked into that issue?
In the meantime, if we add validation like this, is it possible to do the validation up front at restore time so that it doesn't result in creating a red index and just fails the restore request?

Thanks @bugmakerrrrrr! This is definitely an improvement on the existing (broken) behavior. We do have #9676 to track this issue. I believe a more comprehensive fix is to make the code work for any chunk size. Have you looked into that issue?

No, I haven't. Sorry for missing that. Actually, I have considered adapting to all chunk sizes through multi parts downloading, but at this stage, I want to simplify this problem through verification. Btw, is there any progress on #9676

In the meantime, if we add validation like this, is it possible to do the validation up front at restore time so that it doesn't result in creating a red index and just fails the restore request?

To my understanding, RestoreService currently does not load the IndexShardSnapshot during restore, which means we are unable to obtain the precise chunk size of a snapshot file. Furthermore, given the possibility of incremental snapshots and the flexibility for users to modify chunk sizes, it may be necessary to examine all file information in order to retrieve all chunk sizes.

Btw, is there any progress on https://github.com/opensearch-project/OpenSearch/issues/9676

@bugmakerrrrrr There's no progress on that issue so feel free to take it up if you're interested

authors: andrross bugmakerrrrrr


text:
Segment Replication - Fix NoSuchFileException errors caused when computing metadata snapshot on primary shards.

Manual backport of https://github.com/opensearch-project/OpenSearch/pull/4366 to 2.x

Last run failed with below flaky test failure. Refiring!
REPRODUCE WITH: ./gradlew ':plugins:repository-s3:yamlRestTest' --tests "org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" -Dtests.seed=2EAC0D04131BEEB7 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=en-CA -Dtests.timezone=Africa/Libreville -Druntime.java=17 -Dtests.rest.denylist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*,repository_s3/60_repository_eks_credentials/*

org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT > test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials} FAILED
    java.lang.AssertionError: Failure at [repository_s3/20_repository_permanent_credentials:201]: expected [2xx] status code but api [snapshot.create] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"repository_exception","reason":"[repository_permanent] concurrent modification of the index-N file, expected current generation [17544] but it was not found in the repository","stack_trace":"RepositoryException[[repository_permanent] concurrent modification of the index-N file, expected current generation [17544] but it was not found in the repository]\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository.ensureSafeGenerationExists(BlobStoreRepository.java:2199)\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$57(BlobStoreRepository.java:2080)\n\tat org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80)\n\tat org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:126)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343)\n\tat org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120)\n\tat org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:112)\n\tat java.****/java.util.ArrayList.forEach(ArrayList.java:1511)\n\tat org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:112)\n\tat org.opensearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:160)\n\tat org.opensearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:141)\n\tat org.opensearch.action.StepListener.innerOnResponse(StepListener.java:77)\n\tat org.opensearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:55)\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$55(BlobStoreRepository.java:2064)\n\tat org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.****/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.****/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.****/java.lang.Thread.run(Thread.java:833)\n"}],"type":"snapshot_exception","reason":"[repository_permanent:snapshot-two/KStlb4grTqG_DBCYMHglNg] failed to update snapshot in repository","caused_by":{"type":"repository_exception","reason":"[repository_permanent] concurrent modification of the index-N file, expected current generation [17544] but it was not found in the repository","stack_trace":"RepositoryException[[repository_permanent] concurrent modification of the index-N file, expected current generation [17544] but it was not found in the repository]\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository.ensureSafeGenerationExists(BlobStoreRepository.java:2199)\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$57(BlobStoreRepository.java:2080)\n\tat org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80)\n\tat org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:126)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:343)\n\tat org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:120)\n\tat org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:112)\n\tat java.****/java.util.ArrayList.forEach(ArrayList.java:1511)\n\tat org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:112)\n\tat org.opensearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:160)\n\tat org.opensearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:141)\n\tat org.opensearch.action.StepListener.innerOnResponse(StepListener.java:77)\n\tat org.opensearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:55)\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$55(BlobStoreRepository.java:2064)\n\tat org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)\n\tat org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)\n\tat java.****/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.****/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.****/java.lang.Thread.run(Thread.java:833)\n"},"stack_trace":"SnapshotException[[repository_permanent:snapshot-two/KStlb4grTqG_DBCYMHglNg] failed to update snapshot in repository]; nested: RepositoryException[[repository_permanent] concurrent modification of the index-N file, expected current generation [17544] but it was not found in the repository];\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$39(BlobStoreRepository.java:1389)\n\tat org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:88)\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository$8.onResponse(BlobStoreRepository.java:2207)\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository$8.onResponse(BlobStoreRepository.java:2204)\n\tat org.opensearch.repositories.blobstore.BlobStoreRepository$4.clusterStateProcessed(BlobStoreRepository.java:1870)\n\tat org.opensearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:655)\n\tat org.opensearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:524)\n\tat java.****/java.util.ArrayList.forEach(ArrayList.java:1511)\n\tat org.opensearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:524)\n\tat org.opensearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:352)\n\tat org.opensearch.cluster.service.MasterService.publish(MasterService.java:344)\n\tat org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:321)\n\tat org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:196)\n\tat org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:176)\n\tat org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:214)\n\tat org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)\n\tat

Codecov Report

Merging #4422 (ebed94f) into 2.x (b202fd1) will increase coverage by 0.00%.
The diff coverage is 75.52%.

@@            Coverage Diff             @@
##                2.x    #4422    +/-   ##
==========================================
  Coverage     70.54%   70.55%
- Complexity    56942    57101   +159
==========================================
  Files          4572     4584    +12
  Lines        273816   274453   +637
  Branches      40152    40220    +68
==========================================
+ Hits         193170   193629   +459
- Misses        64455    64595   +140
- Partials      16191    16229    +38

Impacted Files
Coverage Δ

.../org/opensearch/client/support/AbstractClient.java
34.70% <0.00%> (-0.44%)
:arrow_down:

...pensearch/common/settings/IndexScopedSettings.java
100.00% <ø> (ø)

.../java/org/opensearch/common/util/FeatureFlags.java
50.00% <ø> (ø)

...rc/main/java/org/opensearch/index/IndexModule.java
82.03% <ø> (ø)

.../java/org/opensearch/plugins/IndexStorePlugin.java
100.00% <ø> (ø)

.../java/org/opensearch/snapshots/RestoreService.java
55.55% <0.00%> (-6.55%)
:arrow_down:

...opensearch/cluster/routing/ShardRoutingHelper.java
95.65% <ø> (ø)

...ore/restore/TransportRestoreRemoteStoreAction.java
25.00% <25.00%> (ø)

...c/main/java/org/opensearch/index/IndexService.java
74.25% <28.57%> (+0.12%)
:arrow_up:

...on/admin/cluster/RestRestoreRemoteStoreAction.java
33.33% <33.33%> (ø)

... and 542 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


text:
Fix DCO CLI example in CONTRIBUTING.md

Signed-off-by: Tyler Ohlsen ohltyler@amazon.com
Description
Fixes the example flag passed in the DCO CLI example to not have added whitespace.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

start gradle check

start gradle check

authors: adnapibar ohltyler saratvemulapalli


text:
Add BACKPORT.md

Signed-off-by: Annie Lee leeyun@amazon.com
Description
Add backport workflow documentation.
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #184 (d7d4fd2) into main (c7b97e8) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #184   +/-   ##
=======================================
  Coverage   57.52%   57.52%
=======================================
  Files         195      195
  Lines        4490     4490
  Branches      599      599
=======================================
  Hits         2583     2583
  Misses       1905     1905
  Partials        2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c7b97e8...d7d4fd2. Read the comment docs.

Give a link to the documentation in our README, we don't need to copy this over.

I removed the md file and added the backport documentation link in developer guide since that seems to be more related.

authors: codecov-commenter leeyun-amzn


text:
Add analyzer endpoints

Signed-off-by: Steve Murphy stevemurphy@BCSR-11413.local
Description
Added:

Analyzer APIs in one file - perform-text-analysis.md
terminology.md section for conceptual information not currently in the doc set
index.md of course

Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
Checklist

[x] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Resolved with PR - https://github.com/opensearch-project/documentation-website/pull/1594

authors: Steve-O-Murphy ariamarble


text:
[BUG] Updating a template unnecessarily checks for the existence of a template we'll overwrite

What is the bug?
When updating a template, we perform two get requests, one to check for the existence of the template (that we are going to overwrite), and a second one to get the provisioning status:
https://github.com/opensearch-project/flow-framework/blob/bae3aa2b8d7168d4da43327cc3b37d2bb759f223/src/main/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandler.java#L409-L423
What is the expected behavior?
We should query the state index once for the provisioning status, and assume if the state doesn't exist, the template doesn't either.  Since we are overwriting the template it doesn't matter whether it exists if the state is other than NOT_STARTED.

Hi, @dbwiddis. I haven't come across an any protocol in the documentation on how to assign an Issue to myself. So to prevent anyone else from working on it I am assigning it to myself now. Please correct me if I am mistaken.

Hey @JamieBort welcome! Not sure if you have permission to assign to yourself, but I'm assigning it to you! Let me know if you have any questions.

Does it matter what branch name I use?
I was expecting a button somewhere on this issue that generated a branch for me; which in turn created a name based on the title of the Issue. From there I would fork the repo, clone my fork, and then checkout the new branch.
However that's the workflow for GitHub's dependabot.
Maybe this isn't available in this scenario.

Does it matter what branch name I use?

Nope, we use a triangle workflow (you can google that) but basically:

you will fork this repository to your own GitHub repository (look for a "Fork" button in the upper right corner of the main page)
you will clone your personal repo to your local machine (clicking the dropdown on the "code" button on the main page gives you the git clone commands or other ways to do it)
you will start with an updated main branch (git pull upstream main)
create your own feature branch with whatever name you want (git checkout -b pick-a-name-here)
write your code!
write tests for your changes!
commit as often as you want as you complete portions of the work
Eventually when you think you're ready to submit, format the code with ./gradlew spotlessApply
Also make sure you've done your javadoces ./gradlew javadoc
And make sure tests pass ./gradlew test

If all that works, push (git push) to upload the code to your fork.   The response from GitHub will give you a URL you can go to, OR you can go to the main repo and there's usually a pop up showing you recently updated a branch, OR you can go to your fork and look for the "contribute" button.   Follow that dialogue to open a PR.
To respond to PR review comments, just edit your code, commit, and push (to your fork) and the PR will automatically get updated.

I was expecting a button somewhere on this issue that generated a branch for me; which in turn created a name based on the title of the Issue.

Not here.

From there I would fork the repo, clone my fork, and then checkout the new branch. However that's the workflow for GitHub's dependabot. Maybe this isn't available in this scenario.

You can fork at any time.

EDIT - if the answers to these questions are in the contribution documentation, please let me know. I know you're busy and while I have looked there already, I rather get familiar with the documentation sooner than later.

A lot of this is at https://github.com/opensearch-project/flow-framework/blob/main/DEVELOPER_GUIDE.md but you can let us know if there's any way we can clarify that!

All of this is very helpful. Thank you!
I am familiar with the Developer Guide that you shared. (Thank you for that.)
But neglected to read about ./gradlew spotlessApply. (4. under the Build section.)
And I am sorry to say that this will be my introduction to writing tests as well.
I feel I have a clear understanding for how the code needs to be refactored. (To me that's the easiest part of this task.)
However I am concerned about accomplishing this in a timely manner. Do you have a deadline for this?

No deadline. It's a nice-to-have.  Next release code freeze is about 6 weeks away but if you don't make that... not an issue!

As far as testing, the method you're changing is already covered by tests, so either it will fail, or it will pass but include an unneeded mock of an index response.  Either way, find the associated test and make sure it accurately reflects what the method does.

I've made my changes (Step 5 from https://github.com/opensearch-project/flow-framework/issues/678#issuecomment-2087489856).
However when I run ./gradlew check (Step 1 from ) the Building from the command line section of the Developer Guide, I am receiving a testFailedUpdateTemplateInGlobalContextNotExisting FAILED error.
I am new to Gradle, however so I'm trying to do my due diligence by understand it better. And by attempting to answer my own questions with a little research before I start posting them here.
That said, I'm sharing the complete output in a .txt file for completeness:
flow-framework_test_20240507.txt
To complicate matters the report states 100% successful:

Something I'll be looking into is my JDK version.
The documentation states 11 is sufficient but 14 is required for the full suite of tests:

[To run the full suite of tests, download and install JDK 14 and set JAVA11_HOME, and JAVA14_HOME.](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#jdk-14)

If that is the culprit I'll need up switch to 14 from 13:
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.7
  OS Info               : Mac OS X 10.16 (x86_64)
  JDK Version           : 13 (Oracle JDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/jdk-13.0.2.jdk/Contents/Home
  Random Testing Seed   : A3CFA76AA32C86BC
  In FIPS 140 mode      : false

Hey Jamie, thanks for the update!

testFailedUpdateTemplateInGlobalContextNotExisting FAILED error.

This is probably a test depending on the old behavior that you just removed as I implied in this comment.
The stack trace after the failed test shows the failed assertion:

org.opensearch.flowframework.indices.FlowFrameworkIndicesHandlerTests.testFailedUpdateTemplateInGlobalContextNotExisting(FlowFrameworkIndicesHandlerTests.java:184)

I'm not sure exactly which line 184 corresponds to yours, but it seems to be the last assertion in this test:
https://github.com/opensearch-project/flow-framework/blob/e3a878405848613ed0ed6166bec281bcbdb97346/src/test/java/org/opensearch/flowframework/indices/FlowFrameworkIndicesHandlerTests.java#L170-L192
Note that whole test was designed to check the line of code that you're removing (checking whether the template exists before updating it) so it seems you might be able to remove that test entirely.
However, I assume you must have added error handling for the state document (or its index) not existing, so you might want to look at testCanNotDeleteWorkflowStateDocInProgress() and copy that into a similar test method which tests the new behavior you added (except testing for non-existence rather than a value other than not started).

The documentation states 11 is sufficient but 14 is required for the full suite of tests:

Don't worry about any JDK other than 11, 17, or 21.   We use JDK17+ on the main branch and 11 on the 2.x branch which this will be backported to, but since you're building against main then 17 or 21 (or any JDK post 14) should work.

copy that into a similar test method which tests the new behavior you added

One other note, we do measure test coverage; codecov will make a comment on a PR (if tests are successful) indicating any uncovered lines, so even if you don't add a new test immediately then after that runs, it can show you uncovered new lines.  It's possible to run this code locally with ./gradlew diffCoverage, see https://github.com/opensearch-project/flow-framework/blob/main/CONTRIBUTING.md#code-coverage

testFailedUpdateTemplateInGlobalContextNotExisting FAILED error.

One other helpful hint.  You can see the logged error message in your output:
  1> [2024-05-07T19:44:02,236][INFO ][o.o.f.i.FlowFrameworkIndicesHandlerTests] [testFailedUpdateTemplateInGlobalContext] before test
  1> [2024-05-07T19:44:02,605][ERROR][o.o.f.i.FlowFrameworkIndicesHandler] [testFailedUpdateTemplateInGlobalContext] Failed to update template for workflow_id : 1, global_context index does not exist.
  1> [2024-05-07T19:44:02,697][INFO ][o.o.f.i.FlowFrameworkIndicesHandlerTests] [testFailedUpdateTemplateInGlobalContext] after test

So the test failure was looking for an exception message containing "Failed to get template" but I expect the message was something about global context index not existing.

Hey @JamieBort hope you're doing well and making good progress.
@joshpalis is working on implementing #717 which is using the same "PUT" update API that was assumed when writing this issue.  In his case, he needs the previous template in order to compare steps.    I do think he may be fetching that template earlier in the REST/Transport sequence but since he doesn't have a draft PR out yet I haven't seen his implementation.  So I wanted to call his attention to this issue and make sure he's not making any changes to the updateTemplateInGlobalContext() method.   @joshpalis can you take a quick look and make sure this issue isn't overlapping any planned code changes you have?  If so, let's discuss a path forward.

Hi, @dbwiddis. Thank you for letting me know. And yes, I'm still here!
However hit a road block with my task. I sent you an email (on May 17) because I thought that would be more appropriate for my discussion. When I had not heard back from you I assumed you were busy.
That said, I have not worked on this since then. I believe my issue is related to me using VS Code rather than Intellij. And I did not know who else to consult about it.

Hi, @dbwiddis. I just sent you a message via LinkedIn. In short, while I'd very much like to contribute to this repo, I think it is best that I unassign myself from this Issue. At least for the time being.
Thank you very much for all of your guidance and help.

authors: JamieBort dbwiddis

repo: opensearch-project/flow-framework


text:
add sample templates for text to visualziation

Description
Add sample templates for text to visualization
Related Issues
Resolves #[Issue number to be closed when this PR is merged]

Check List

[ ] New functionality includes testing.
[ ] New functionality has been documented.
[ ] API changes companion pull request created.
[ ] Commits are signed per the DCO using --signoff.
[ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dbwiddis @owaiskazi19 Could  you take a look the PR please?

@ruanyl Will do. I saw it earlier but it was in draft until just no.
Note the spotless check is failing (looks like trailing whitespace).  Please run ./gradlew spotlessApply and commit the difference.   Also add a section to the 2.x documentation on the change log.

@ruanyl can you also some description about the use case for this template in the PR?

@ruanyl Is there going to be another PR on t2vega_instruction_based_agent?

@ruanyl Is there going to be a PR adding t2vega_agent (without instruction)?

Probably a changelog conflict with #968. Will wait until that's merged and toggle the backport label

authors: dbwiddis owaiskazi19 ruanyl yuye-aws


text:
[OSD][1.3.1] Add ISM plugin

Description
Add ISM plugin to the OSD manifest
Signed-off-by: Kawika Avilla kavilla414@gmail.com
Related Issue:
https://github.com/opensearch-project/opensearch-build/issues/1885
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@opensearch-project/index-management

authors: kavilla tianleh

repo: opensearch-project/opensearch-build


text:
Use JSON11 to parse and serialize long numerals

Description
Changing from the RegEx based approach to an AST one dramatically reduces the encounters with false-positives, reducing the amount of time it takes to parse very large values.
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] Linter check was successfull - yarn run lint doesn't show any errors
[X] Commits are signed per the DCO using --signoff
[ ] Changelog was updated.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Are _stringifyWithBigInt and _parseWithBigInt used anywhere after this or are they dead code? Remove.

Removed dead code.

Are there tests for this that do big ints that cover these paths?
Yes; I cleaned them up to remove the unnecessary tests.

What are the performance numbers, even if anecdotal?
An experiment with 10,000 false positives took over 90s earlier. With JSON11, it takes 200-ish ms.

Working on the tests.

Is JSON11 a superset of sjson?

JSON11 is a superset of JSON5 which is superset of JSON. secure-json-parse is just a wrapper around the built-in JSON.parse to throw an exception if it finds a key named __proto__. secure-json-parse has no special abilities beyond that.

Why not just use that in all cases and get rid of all the special handling of big ints?
JSON.parse being a built-in method is 17 times faster than JSON5.parse and by proxy JSON11's. We are giving users the choice between (A) fast and imprecise for large integers, and (B) slow and precise. I would hazard a guess that not many users deal with precise large integers because BigInt is a newer concept in JS.

Would it make sense to let users configure the serializer/deserializer instead of having special handling of big ints and importing mutliple libraries?

It is a user provided option named enableLongNumeralSupport which allows them to choose between A and B.

On a side note: I believe the way secure-json-parse works is not useful. It either removes the property or throws an exception. When there is a correct way of handling it by retaining __proto__ but preventing it from poisoning the prototype, we shouldn't destroying data.

On a side note: I believe the way secure-json-parse works is not useful. It either removes the property or throws an exception. When there is a correct way of handling it by retaining __proto__ but preventing it from poisoning the prototype, we shouldn't destroying data.

I don't think I understand this, but open issues!

This is better than the previous custom implementation, so good to go.

I am glad I was able to make time for this.

Generally, in the combination of shouldHandleLongNumerals=true, numeralsAreNumbers getting set by the first parse, and JSON11.stringify succeeding, we parse twice.

I could just make it look at shouldHandleLongNumerals and then use JSON11. I feel with numeralsAreNumbers, we get to save on running the slower method unnecessarily. If we assume t is the time it takes to run the built-in methods and 17t would be the time for the JSONx methods, if a user asks for long numerals and only half their calls reference long numerals:

With giving them the option to choose the JSONx library:

All calls will take 17t and making 100 calls will take 1700t.

With the current detection:

calls that don't need treatment will use the built-in one and take t.
calls that need treatment will use t + 17t because they have to run both and will take 18t.

Making 100 calls will take 950t.

This just begs the question of why all this complication, what would not be possible if the caller specified that they want to use JSON vs. JSON11 vs. JSONXYZ?

That is certainly an option but for mixed-use customers, it would lead to maintaining too many options.

authors: AMoo-Miki dblock

repo: opensearch-project/opensearch-js


text:
Create Tooling to Simulate Cluster Traffic

Task
In order to test our cluster duplication tooling, we need to have tooling to simulate traffic being sent to the Cluster.  While in the long term we'll probably want to have a fairly sophisticated tool that can be run on a collection of hosts and pointed at a Cluster across a real network, for the time being we can likely get away with something simple that's run off a developer laptop.
Acceptance Criteria

Tooling should be runnable on a developer laptop via command line and pointed at a hostname/port
Tooling should be able to simulate data reads (get indexes/docs), data writes (put indexes/docs), configuration reads (get cluster status), and configuration writes (change index settings, etc).
The ratio and rate of the requests sent by the tooling should be configurable.  For this task, we'll focus on constant-rate requests instead of variable rate requests.
The behavior of the tooling should be repeatable.  That is, if an issue is identified with one run of the tooling, we should be able to do that exact same run again to provoke the same response from the system under test.
Demo tool to team

Task might be obviated by this existing tool: https://github.com/opensearch-project/opensearch-benchmark-workloads

Estimate: 5 - Assumption is that we will not have to write code but if we do it will be basic.

authors: chelma sumobrian


text:
enable automatic backport to 2.x

since we now have a 2.x branch here (since #69) it'd be good if the automated backport bot would be set up the same way as for the other opensearch repos (i'm not familiar with the details - i just know that it exists 😄) to ensure that 2.x is kept in sync where relevant.
until this is the case cherry-picks will have to be done manually.

@rursprung You can just copy the backport workflow from maybe https://github.com/opensearch-project/opensearch-java/blob/main/.github/workflows/backport.yml

@rursprung You can just copy the backport workflow from maybe https://github.com/opensearch-project/opensearch-java/blob/main/.github/workflows/backport.yml
#83

i guess this has to be done by a maintainer with admin access as you need to set up the secrets? thus i will refrain from contributing this as it presumably won't work

authors: dblock rursprung


text:
Change SDKClusterSettings constructor to use same params as ClusterSettings

Description
Updates the SDKClusterSettings class to use the same constructor arguments as the ClusterSettings class.
NOTE: This will break current AnomalyDetection implementations using a no-arg constructor.  I will be submitting a companion PR on AD soon, but this will need to be merged first.
Issues Resolved
Fixes #503
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Conflicts with main :/

Codecov Report

Merging #559 (31b5c4f) into main (7e8a722) will increase coverage by 1.67%.
The diff coverage is 100.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main     #559      +/-   ##
============================================
+ Coverage     68.07%   69.75%   +1.67%
- Complexity      208      213       +5
============================================
  Files            37       37
  Lines           874      886      +12
  Branches         26       26
============================================
+ Hits            595      618      +23
+ Misses          269      257      -12
- Partials         10       11       +1

Impacted Files
Coverage Δ

...main/java/org/opensearch/sdk/ExtensionsRunner.java
77.34% <100.00%> (+5.28%)
:arrow_up:

...ain/java/org/opensearch/sdk/SDKClusterService.java
100.00% <100.00%> (ø)

...rch/sdk/handlers/ExtensionsInitRequestHandler.java
100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Conflicts with main :/

Rebased, need re-review. :)

repo: opensearch-project/opensearch-sdk-java


text:
Add icons to icon pack

This issue supports OSD release 2.7
The following icons need to be added to OUI & the OUI documentation website - retain the alphabetical order of the Glyphs list.
Use these guidelines for design and adding icons to OUI : https://github.com/opensearch-project/oui/blob/main/wiki/creating-icons.md#style

[ ] polygon (https://github.com/opensearch-project/oui/issues/309)
[ ] radial (out of band request, supports maps tools)
[ ] visBuilder (https://github.com/opensearch-project/ux/issues/45)
[ ] visBuilderSavedObject (https://github.com/opensearch-project/oui/issues/637) to represent the visualizations created in Observability visualizations in the Saved Objects table.
[ ] visBuilderDQL (https://github.com/opensearch-project/ux/issues/45) (this is a strategic have-in-back-of-pocket)
[ ] visBuilderPPL (https://github.com/opensearch-project/oui/issues/637)
[ ] visBuilderPromQL (https://github.com/opensearch-project/oui/issues/637)
[ ] visBuilderSQL (https://github.com/opensearch-project/oui/issues/637)

Glyphs for GeoMap draw tools: radial.svg / polygon.svg

@BSFishy all icons for the 1.1 release are here in the comments. Please refer to https://github.com/opensearch-project/oui/blob/main/wiki/creating-icons.md#style on next steps

Reworked the visbuilder icons as there was a discrepancy in the lettering of "PPL" - all reworked versions in the .ip file below. @KrooshalUX please review.
visBuilder.zip

I did some cleanup to the final SVG files:

Add SPDX header
Indent paths
Remove fills on paths
Re-order svg tag
Remove fill on svg tag

(for reference)

Thanks @BSFishy - wondering if we should use that to update anything on this doc:  https://github.com/opensearch-project/oui/blob/main/wiki/creating-icons.md#style - I think earlier you mentioned you had some ideas.
Also for SOP-ifying this: I would like to keep UX out of preparing the files beyond the original export, in case a mistake is made while editing. Otherwise, engineering would need to wait for UX to re-export, this way engineering has the original export and can modify as necessary with less likelihood of errors.

Thanks @BSFishy - wondering if we should use that to update anything on this doc: https://github.com/opensearch-project/oui/blob/main/wiki/creating-icons.md#style - I think earlier you mentioned you had some ideas.

I think it is mostly covered in the Clean the SVG section. I can submit a PR to update it with more detailed info from what I did for these icons.

Also for SOP-ifying this: I would like to keep UX out of preparing the files beyond the original export, in case a mistake is made while editing. Otherwise, engineering would need to wait for UX to re-export, this way engineering has the original export and can modify as necessary with less likelihood of errors.

I agree. I wanted to update this issue for knowledge sharing reasons for the future.

Updated names as per feedback from pull request #658

authors: BSFishy KrooshalUX canascar

repo: opensearch-project/oui


text:
Deprecating Master nomenclature

Is your feature request related to a problem? Please provide an existing Issue # , or describe.
Master Nomenclature Issue
Describe the solution you are proposing
This change is to stop the usage of master keyword in the performance-analyzer-rca codebase.
Additional context
Add any other context or screenshots about the feature request here.
Check List

[ ] New functionality includes testing.

[x] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
This PR is broken down into multiple PRs

pr1
pr2
pr3
pr4
pr5
pr6

Review PR #206
LGTM +1
Ref: https://github.com/opensearch-project/performance-analyzer-rca/pull/206#pullrequestreview-1077919887

Approved PR#212 (pr6)

Approved PR#208 (pr2) with some nits

Codecov Report

Merging #205 (23ec4d9) into main (212f82a) will decrease coverage by 0.03%.
The diff coverage is 71.18%.

@@             Coverage Diff              @@
##               main     #205      +/-   ##
============================================
- Coverage     71.57%   71.53%   -0.04%
  Complexity     2987     2987
============================================
  Files           380      380
  Lines         18952    18984      +32
  Branches       1463     1463
============================================
+ Hits          13565    13581      +16
- Misses         4792     4807      +15
- Partials        595      596       +1

Impacted Files
Coverage Δ

...manceanalyzer/decisionmaker/deciders/Decision.java
70.58% <ø> (ø)

...ceanalyzer/metrics/PerformanceAnalyzerMetrics.java
37.97% <ø> (ø)

...ts/temperature/CompactNodeTemperatureFlowUnit.java
15.78% <ø> (ø)

...rca/framework/api/summaries/HotClusterSummary.java
77.14% <ø> (ø)

...r/rca/framework/api/summaries/HotShardSummary.java
86.70% <ø> (ø)

...mmaries/temperature/ClusterDimensionalSummary.java
0.00% <ø> (ø)

...es/temperature/CompactClusterLevelNodeSummary.java
0.00% <ø> (ø)

.../api/summaries/temperature/CompactNodeSummary.java
23.60% <ø> (ø)

...maries/temperature/FullNodeTemperatureSummary.java
34.61% <ø> (ø)

...manceanalyzer/rca/persistence/SQLitePersistor.java
75.30% <ø> (ø)

... and 44 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@dzane17  reviewed & approved PR https://github.com/opensearch-project/performance-analyzer-rca/pull/209 (pr3)
@sruti1312   reviewed & approved PR https://github.com/opensearch-project/performance-analyzer-rca/pull/209 (pr4)
@sgup432  reviewed & approved PR https://github.com/opensearch-project/performance-analyzer-rca/pull/209 (pr5)


text:
[OSCI] Adds new author abhinaenae

Description
Adds a new author Abhinaenae
Issues Resolved
[List any issues this PR will resolve]
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Hello @Abhinaenae check out our Day 2 video and start watching from 49:20 and you'll see how to solve DCO issue.
https://www.youtube.com/watch?v=v56sNHkvRrU

Closing as we are past deadline and there are outstanding changes.

authors: Abhinaenae davidosorno dtaivpp

repo: opensearch-project/project-website


text:
Index permission "indices:admin/resolve/index" missing from Kibana Security UI [Enhancement]

Describe the bug
I figured out that the "indices:admin/resolve/index" index permission can’t be add from the Security UI of Kibana when editing/creating a role.
However, it can be done through API request (PUT _opendistro/_security/api/roles/).
To Reproduce
Steps to reproduce the behavior:

Go to 'Security' section
Select a role and click on 'Edit role'
Scroll down to 'Index permissions'
Try to add "indices:admin/resolve/index" which doesn't appear

Expected behavior
This permission should appear.
Additional context
The only way I found to add this permission is from the API:
PUT _opendistro/_security/api/roles/users {   "cluster_permissions": [     "some_actions"   ],   "index_permissions": [     {       "index_patterns": [         "*"       ],     "allowed_actions": [       "indices:admin/resolve/index",       "some_actions"     ],   "tenant_permissions": [     {       "tenant_patterns": [         "global_tenant"       ],       "allowed_actions": [         "kibana_all_read",         "kibana_all_write"       ]     }   ] }

Thanks for the report! Checked in 2.1.0 and the permission is there now.

authors: davidlago zibrata


text:
Bump plugin version to 1.3.1.0 and OpenSearch dependency version to 1.3.1

Signed-off-by: cliu123 lc12251109@gmail.com
Description
[Describe what this change achieves]

Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Maintenance

Testing

UTs
Manual Testing:

Install security plugin 1.4.0.0 on OpenSearch 1.4.0-SNAPSHOT
Start OpenSearch cluster
Cat indices:

HTTP/1.1 200 OK
Warning: 299 OpenSearch-1.3.1-SNAPSHOT-01c99f761603dc8a63795c17d132a9cdbaedb5a8 "this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default"
content-type: text/plain; charset=UTF-8
content-length: 77

green open .opendistro_security QaZBpKzWRI-T73ul2rS8FA 1 0 9 0 60.1kb 60.1kb

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1697 (69f750e) into 1.3 (9d44ff4) will decrease coverage by 0.07%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                1.3    #1697      +/-   ##
============================================
- Coverage     64.58%   64.51%   -0.08%
+ Complexity     3214     3213       -1
============================================
  Files           247      247
  Lines         17351    17350       -1
  Branches       3082     3073       -9
============================================
- Hits          11207    11194      -13
- Misses         4594     4599       +5
- Partials       1550     1557       +7

Impacted Files
Coverage Δ

...security/auditlog/sink/InternalOpenSearchSink.java
69.23% <0.00%> (-11.54%)
:arrow_down:

...ava/com/amazon/dlic/auth/ldap2/MakeJava9Happy.java
39.13% <0.00%> (-4.35%)
:arrow_down:

.../org/opensearch/security/support/ConfigHelper.java
80.39% <0.00%> (-1.97%)
:arrow_down:

...com/amazon/dlic/auth/ldap2/PrivilegedProvider.java
49.18% <0.00%> (-1.64%)
:arrow_down:

.../org/opensearch/security/auth/BackendRegistry.java
58.85% <0.00%> (-0.53%)
:arrow_down:

...ic/auth/ldap/backend/LDAPAuthorizationBackend.java
57.57% <0.00%> (-0.38%)
:arrow_down:

...zon/dlic/auth/ldap2/LDAPAuthorizationBackend2.java
28.35% <0.00%> (-0.38%)
:arrow_down:

...a/org/opensearch/security/tools/SecurityAdmin.java
47.39% <0.00%> (-0.20%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9d44ff4...69f750e. Read the comment docs.

repo: opensearch-project/security


text:
rename to opensearch and dashboard

Signed-off-by: Prudhvi Godithi pgodithi@amazon.com
Description
rename to opensearch and dashboard
Issues Resolved
https://github.com/opensearch-project/terraform-provider-opensearch/issues/18
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Hey @phillbaker @bbarani @peterzhuamazon @serge-r can you please review this rename PR, I have done some changes in few go files and refactored the code.
Thank you

I started reviewing and it mostly looks good, I'll be able to finish reviewing tomorrow.

authors: phillbaker prudhvigodithi


text:
Baseline MAINTAINERS, CODEOWNERS and external collaborator permissions

I have baselined the permissions of this repository.  Manage access, MAINTAINERS.md, and CODEOWNERS are in sync.
Please close this issue if this is correct.

Following the guidance from https://github.com/opensearch-project/.github/issues/125

Closed via #22

authors: dblock peternied

repo: opensearch-project/traffic-comparator


text:
Update README.md and README.zh-Hans.md

Refine README and add logo && landscape.
Signed-off-by: Eric Zhao sczyh16@gmail.com

流量调度这里用了Load Balancing是专门指负载均衡？

这里 load balancing 指的是广义的通过调整权重方式来保障稳定性的手段，包括 conventional load balancing.
之前担心的点是 traffic scheduling 业界没有类似表述；不过看了一下还是有相关表述的，也可以考虑用这个: https://www.tutorialspoint.com/qos-traffic-scheduling
后续每个名词的具体含义都需要归纳到 术语表 中。

repo: opensergo/opensergo-specification