text:
fix: Hide Robotoff banner on the onboarding

Hi everyone,
On the onboarding, we quickly have the Robotoff banner appearing.
This PR totally hides it.
Old behavior:
https://github.com/openfoodfacts/smooth-app/assets/246838/22772a47-7f0a-4d47-8f7b-3f72cf267349

Codecov Report
Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 7.44%. Comparing base (4d9c7fc) to head (d0ced68).
Report is 185 commits behind head on develop.

Files
Patch %
Lines

...ges/smooth_app/lib/pages/product/summary_card.dart
0.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           develop   #5369      +/-   ##
==========================================
- Coverage     9.54%   7.44%   -2.11%
==========================================
  Files          325     380      +55
  Lines        16411   19456    +3045
==========================================
- Hits          1567    1448     -119
- Misses       14844   18008    +3164

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


text:
Ecosystem modelling docs

This PR adds some guidance for new users on how all the parts of OpenFF fit together, and how to find the software they want. The page added in this PR focuses on users who want to apply our force fields to their systems; users who want to produce their own force fields will come later.
I've started with a flowchart that has a lot of information and links, and I'm opening this in case anyone has any thoughts before I put time into writing stuff up. Any ideas on what a new user might find useful would be helpful!
Rendered: https://openff-docs--5.org.readthedocs.build/en/5/modelling.html

@j-wags @xyzmjf I would love reviews on this!

"OpenFF Interchange is our package for producing simulation data in various formats." --> Instead of "producing", maybe "converting" or "exporting"?

Just a quick hitter on this point (I won't dig through everything unless you need a second pair of eyes) I think "transform" might work as the operative verb. I think the information content is roughly the same but it sounds cooler. Up to you.

Comments attached as Open Document Text (.odt) and PDF formats. I did this so that I can highlight images in the draft documentation and explicitly comment on them.
2023April27MF_Comments_OpenFF_Docs_v1.odt
2023April27MF_Comments_OpenFF_Docs_v1.pdf

A couple of subtler points:

The block of OpenMM code produces objects in-memory; OpenMM doesn't use the filesystem as an intermediary between system preparation and running a simulation. The objects can be serialized out but any practitioner knows how to get those objects into a running simulation
SMIRNOFF does not do "atom typing" in the sense that atom types (or bond types, etc.) are not produced, but it does do "typing" in the sense that parameters are applied to chemistry. "atom typing" is a no-no but "typing" is fine

Again I am taking the viewpoint of a very new and inexperienced user here. I would that that to run the code snippet they need to do 2 things. [1] Save it to a file - say example1.py [2] Run this in their command line shell - python3 example1.py
No doubt this is setting the bar very low, but that is how we aim to get new users on board.

SMIRNOFF does not do "atom typing" in the sense that atom types (or bond types, etc.) are not produced,
but it does do "typing" in the sense that parameters are applied to chemistry.
"atom typing" is a no-no but "typing" is fine

That is helpful thanks. My language is inexact as I am new and this is a novel paradigm for me !

The block of OpenMM code produces objects in-memory; OpenMM doesn't use the filesystem as an intermediary between
system preparation and running a simulation. The objects can be serialized out but any practitioner knows how to get those
objects into a running simulation

I realise that objects are produced in memory, but a new user sees no output (unless there is an error).
Can we just add a simple energy minimization and print out something for the user?

It's up to Josh but adding energy minimization and reporting would double the size of that cell block and go against the message of how a simulation can be prepared in just a few LOC. Perhaps the user could be dedicated to examples that are directed at this purpose

Thanks for taking a look Mark and Matt! Matt I'm gonna stick with export because I want to emphasize the natural path through the flowchart more than I want to describe how much Interchange can do. Transform is a great word though and I will definitely use it elsewhere! Export always feels clumsy.
Thanks for your detailed comments Mark! I'll take them quickly one at a time before I head to bed...

[1] Can we have an introduction paragraph – something like this.

I want readers to see the flowchart without scrolling, so I don't want to add more text to the top of this page. I'm also not expecting users to come to this page without knowing the basics of what OpenFF is - most users discovering us will go through https://docs.openforcefield.org first. However, that page could definitely use an introduction like the one you've written! I'll add a condensed version of it to that page before I merge this PR.

[3] I’m not so happy about this RHS of the pipeline image:

I'm curious to hear more about this. From my perspective, this entire flowchart is intended to be new information for the user at this point - we haven't introduced Molecules or Topologies or certainly Interchanges at this point either. We've received a lot of feedback that BespokeFit is one of our "killer apps", so I want to make sure prospective users encounter it early on, and the orange box itself explains its function. So I'm inclined to leave this as it is, but maybe I could visually de-emphasize it somehow so that readers naturally focus on the other parts of the flowchart and don't get distracted?

[4] It is really great to include some code for the user to run, and this is (almost) a good example.

I agree with Matt on this one - users familiar with Python will probably be more comfortable with in-memory output like what the current example has than writing out a file, for instance, which are a minor pain to work with in Python. For those users, the last three lines basically act as output (which they don't even have to run the example to see). If we just wanted file-type output, we could substitute the OpenMM lines for GROMACS or Amber or CHARMM, which all would then create files; I don't really think that adds anything (except complication for the test environment).
Users who are not familiar with Python will struggle either way - almost all of our tools really require some level of comfort with Python. The point of this code example is to emphasize how easy our tools are, not for users to necessarily copy it - even with an energy calculation a vacuum MM point energy of a drug molecule is not going to be super useful.
Having said that, if there were a one-liner to compute a point energy or something like that in OpenMM I would totally add it here - but the best I could come up with is this:
from openff.toolkit import Molecule, Topology, ForceField
from openff.interchange import Interchange
import openmm

paracetamol = Molecule.from_smiles("CC(=O)NC1=CC=C(C=C1)O")
paracetamol.generate_conformers(n_conformers=1)

topology = Topology.from_molecules([paracetamol])

force_field = ForceField("openff-2.0.0.offxml")

interchange = Interchange.from_smirnoff(force_field, topology)

openmm_system = interchange.to_openmm()
openmm_topology = interchange.to_openmm_topology()
openmm_positions = interchange.positions.to_openmm()

simulation = openmm.app.simulation.Simulation(
    openmm_topology,
    openmm_system,
    openmm.VerletIntegrator(2.0 * openmm.unit.femtosecond),
)
simulation.context.setPositions(openmm_positions)
simulation.minimizeEnergy()
simulation.context.getState(getEnergy=True).getPotentialEnergy()

Which adds 9 lines to a previously 10 line example, and is mostly gobbledygook for people not familiar with OpenMM.
The point about examples generally lacking throughout the docs is well taken, and I've already started work on a centralized example page that will make this much easier (and which we'll link to from here when it becomes available). But I don't think complicating the ad for simplicity is the way to solve that problem.

[5] Now I have a quite general point about SMIRNOFF. As I now understand it, we do not actually do ‘atom typing’ using SMIRKS patterns as SMIRKS relates to chemical reactions. We do ‘atom typing’ or chemical environment perception and
parameter assignment using SMARTS patterns – as SMARTS relates to chemical substructures.

This is exactly right and it annoys me too. SMIRKS is a superset of SMARTS, so if we use SMARTS it is technically correct to say that we also use SMIRKS. The part of SMIRKS that is used is the atom indexing - you can distinguish eg. the hydrogens in a water SMIRKS with "[H:1]O[H:2]". SMIRKS intended this to show which atoms are the same in the reactant and product, but SMIRNOFF uses this to specify which atoms actually get the parameter - this way we can use neighboring atoms to specify parameters that don't actually affect them. So, I dunno, a carbonyl carbon's LJ parameter SMIRKS might be "C=[O:1]" for instance, and could be distinguished from an ester "OC=[O:1]".
Daylight later added this indexing to the SMILES (which SMARTS is a superset of) spec, but SMARTNOFF is not a popular vodka brand, so we're stuck with SMIRNOFF.

Maybe one day we'll come up with a coarse grained force field and call it SMARTINI

This is really great discussion - we don't often go deep into the pareto front of our choices but it's really handy to have a new set of eyes to point out things that I overlook when seeing for the thousandth time.
Given that this is intended to be the first point of contact for all new arrivals to our ecosystem it is worth a lot of polishing. Ideally this will be a straightforward document that is easy for newcomers to parse, but which experts will be able to read and come away with deeper levels of meaning. That's a tall order.
That said, it's worth keeping in mind:

@Yoshanuikabundi is product owner for openff-docs so he's free to unilaterally make any choice he wants.
This page will continue to evolve over time, so merging this PR doesn't end these discussions, but it does get something useful in the hands of users immediately.
Taking into consideration time zone overlaps, we can only expect a max of 4 rounds of feedback per week on this discussion.

@xyzmjf For consideration, I tried to minimally run an openmm simulation that writes familiar files (pdb and dcd) to disk and came up with the code below. I could probably reduce the number of lines a little, but it would come at the cost of making the remaining lines look more like the "gobbledygook" mentioned above.
One compromise could be defaulting to collapsing the "simulate" and "visualize" sections, so users have to click to expand them. Is this possible @Yoshanuikabundi? If not I support sticking with the current version and linking to an example that shows how to do a simulation.
from openff.toolkit import Molecule, Topology, ForceField
from openff.interchange import Interchange

paracetamol = Molecule.from_smiles("CC(=O)NC1=CC=C(C=C1)O")
paracetamol.generate_conformers(n_conformers=1)

topology = Topology.from_molecules([paracetamol])

force_field = ForceField("openff-2.0.0.offxml")

interchange = Interchange.from_smirnoff(force_field, topology)

openmm_system = interchange.to_openmm()
openmm_topology = interchange.to_openmm_topology()
openmm_positions = interchange.positions.to_openmm()

# Run a simulation
import openmm
import openmm.unit as openmm_unit
integrator = openmm.VerletIntegrator(1 * openmm_unit.femtoseconds)
platform = openmm.Platform.getPlatformByName("Reference")
simulation = openmm.app.Simulation(openmm_topology, openmm_system, integrator, platform)
simulation.context.setPositions(openmm_positions)
# Add a reporter to record the structure every 10 steps, and write a pdb with the basic topology for visualization
paracetamol.to_file("paracetamol.pdb", file_format="pdb")
dcd_reporter = openmm.app.DCDReporter("paracetamol.dcd", 10)
simulation.reporters.append(dcd_reporter)
simulation.context.setVelocitiesToTemperature(300 * openmm_unit.kelvin)
simulation.step(1000)

# Visualize the trajectory in a notebook (skip this if you prefer VMD)
import nglview
import mdtraj

mdt_traj = mdtraj.load("paracetamol.dcd", top="paracetamol.pdb")
w = nglview.show_mdtraj(mdt_traj)
w

In a notebook, the last bit pops up a simulation visualization:

Had an idea about an API point that would make a full energy minimisation possible in the constraints of this page, so I've opened https://github.com/openforcefield/openff-interchange/issues/713
I've made the changes I discussed above and I'm merging this now as I really want this out there before the meeting - if there are other changes people think we should do, please raise an issue!

source: gharchive/pull-request

authors: Yoshanuikabundi j-wags mattwthompson xyzmjf

repo: openforcefield/openff-docs

license: MIT

license_type: permissive

license_source: github-api


text:
openff-units 0.2.1 breaks mypy for openff-models

Not sure if this belongs here or on openff-models, but here's a MCVE:
# file named example.py
from openff.models.types import FloatQuantity
import openff.units  # type: ignore

print(f"{openff.units.__version__=}")
models_quantity: FloatQuantity["kelvin"]

Then, with openff-units 0.2.0 (and openff-models at current release, 0.0.5):
$ python example.py && mypy example.py
openff.units.__version__='0.2.0'
Success: no issues found in 1 source file

With openff-units 0.2.1:
$ python example.py && mypy example.py
openff.units.__version__='0.2.1'
example.py:5: error: Variable "openff.models.types.FloatQuantity" is not valid as a type  [valid-type]
example.py:5: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
example.py:5: error: Name "kelvin" is not defined  [name-defined]
Found 2 errors in 1 file (checked 1 source file)

see also https://github.com/OpenFreeEnergy/openfe/issues/461

0.2.1 breaks lots of stuff for reasons that have caused me physiological aging (blood pressure, hair loss, etc.). For now the recommendation is to stick to 0.2.0, since I think that works, and we'll have fixes out as soon as we can (one attempt at https://github.com/openforcefield/openff-units/pull/68).

I'm also not sure if this affects both Pint 0.21 and 0.22, but I'm pretty sure one of them works with 0.2.0

With #84 I have it passing on Pint 0.21, 0.22, 0.23
With https://github.com/openforcefield/openff-models/pull/42 it's also passing with Pydantic models. The SomeQuantity['preferred_unit'] paradigm is likely going away, I unfortunately don't think that pattern is future-compatible with how tooling interprets string literals brackets inside of things that look like types.
#77 would be my preferred solution but that's around three years out of being compatible with OpenFF's Python support

This still sticks around on the v1 API, it should be gone with v2
$ cat example.py && python example.py && mypy example.py                                                                           14:00:02  ☁  0.2.2 ☂
# file named example.py
import openff.units
import openff.models

from openff.models.types import FloatQuantity
from pydantic.v1 import BaseModel
from openff.units import Quantity

print(f"{openff.units.__version__=}")
print(f"{openff.models.__version__=}")

class Model(BaseModel):
    temperature: FloatQuantity["kelvin"]

print(f"{Model(temperature=Quantity("419.0 kelvin")).temperature=}")
openff.units.__version__='0.2.2'
openff.models.__version__='0.1.2'
Model(temperature=Quantity("419.0 kelvin")).temperature=<Quantity(419.0, 'kelvin')>
example.py:15: error: Variable "openff.models.types.FloatQuantity" is not valid as a type  [valid-type]
example.py:15: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
example.py:15: error: Name "kelvin" is not defined  [name-defined]
Found 2 errors in 1 file (checked 1 source file)

source: gharchive/issue

authors: dwhswenson mattwthompson

repo: openforcefield/openff-units


text:
Institution availability issues

When archiving an institution, either archinve all open projects, or block user from archiving.  Either way warn the user.

Clean up existing projects where the associated institution is archived.

Add route security to check for "archived" on institution routes.

repo: openforis/collect-earth-online

license: mit

license_source: bigquery


text:
Api/order target courses

Purpose
Like we did for product serializer, we now bind order's target courses into the
OrderSerializer representation. This information are useful for frontend so to
avoid numerous requests to the api, we can assume to bind directly this
information within order.
Proposal

[x] Bind full target courses object to OrderSerializer reprensetation
[x] Update tests

@sampaccoud I save 2-3 db queries per read detail requests by updating the queryset to retrieve order. In fact, we were missusing prefetch_related.

repo: openfun/joanie


text:
Attributes class, function, usage for the classes Window and Door

Should the classes Door and Window also define the attributes class, function, and usage, to be able to specify more precisely the type of door (e.g. front door, cellar door, interior door) or window?

I would say yes. We should be consequent in such a way, that each "real world thing" (represented by a Space) should have these three attributes.

authors: TatjanaKutzner thomashkolbe


text:
Provide stable, clean release tarballs.

The tarballs that github.com produces seem to be arbitrarily structured. May you post tarballs for php-openid releases at least slightly manually, such as having a subdirectory called packagename-version instead of username-reponame-commitid?
For example, ftp://ohnopub.net/mirror/php-openid-2.2.2.tar.bz2
Thanks.

This repo is being archived. Closing issue.

authors: binki timcappalli

repo: openid/php-openid


text:
feat: optimize create element and create text node.

将 Node 和 Element 上的属性定义移动到原型链上，避免每次创建 Element 和 Node 都要初始化一遍属性。

缺少 TextNode 的重构

authors: andycall yuanyan

repo: openkraken/kraken


text:
Mark ol.Observable.prototype.unByKey deprecated.

The ol.Observable.prototype.unByKey was supposed to handle removing of
listeners registered using ol.Observable.prototype.on and
ol.Observable.prototype.once.
However, the underlying google library code called by unByKey only requires
the key, not the ol.Observable object on which the registration was
triggered. Moreover, this method was unconvenient and in practice misused.
Example or real world code:
var map = new ol.Map();
var key = map.on(listener);
view.unByKey(key);

A function was added in #2794, in particular to fix this kind of code.
This PR pushes forward the removal process of the old method in favor of the
unByKey function by marking it @deprecated and pointing to the right
usage.
Please merge when deemed appropriate.

I don't think we should mark this as deprecated, much less say we are going to remove it (and that wouldn't happen anyway before 4.0).
We're facing a huge challenge in making it easy for people to discover what's in the library.  The ol.Observable.unByKey function is arguably harder to find than the unByKey method, and I don't think there is a good argument for removing, deprecating, or otherwise making the unByKey method even harder to discover.  In addition, when generating a custom build, I imagine people will make use of wildcard exports until we have an extremely user-friendly and flexible hosted build tool.  Exporting ol.Map#* gets you everything you need to work with a map.  Having to discover ol.Observable.unByKey and export that as well is an additional (unnecessary challenge).
So, since we've already got the unByKey method and we're not going to break the stable API, I'm opposed to making it any less convenient to use.

The ol.Observable.unByKey function is mentioned in the ol.Observable.prototype.on method  and similarly easy to discover as the prototype version and it prevents misleading uses as explained in my previous comments.
I am confident people able to generate a custom build are able to export the unByKey function.
That being said, since there is no removal policy before 4.0, I am OK seeing my PR rejected.

authors: gberaudo tschaub


text:
Update to layer-extent example

This follows up on the remaining comments in #5482 by removing the extra interaction from the layer extent example.

thanks @tschaub

authors: bartvde tschaub


text:
How can get the coordinate of the visible polygon on Bounding Box map

Hi this is my feature MultiPolygon visible on map:

If I zoom on map how I can get the coordinate of the visible polygon on map?

I ask this because I have to recalculate the center point of the visible polygon.
Thanks

This should get you going.
const size = map.getSize();
const ring = [
  map.getCoordinateFromPixel([0, 0]),
  map.getCoordinateFromPixel([0, size[1]),
  map.getCoordinateFromPixel(size),
  map.getCoordinateFromPixel([size[0], 0]),
  map.getCoordinateFromPixel([0, 0]),
].map((c) => toLonLat(c));
const viewpoly = turf.polygon([ring]);
const intersection = turf.intersect(viewpoly, otherpolygon);

https://turfjs.org/docs/#intersect
https://turfjs.org/docs/#center or https://turfjs.org/docs/#centerOfMass

Thanks very useful!

repo: openlayers/openlayers


text:
webgltilelayer style filter

Describe the bug
A clear and concise description of what the bug is.
In openlayers v9.1.0, filter is used when applying the style of WebGLVectorTileLayerRenderer, and the three filters below work.
geometry-type does not work.
filter: ['==', ['get', 'layer'], 'water'],
filter: ['<=', ['resolution'], 10],
filter: ['>=', ['zoom'], 9],
LineString also doesn't work.
filter:['==',['geometry-type'],'Polygon']
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

//ol/render/canvas/style.js
export function rulesToStyleFunction(rules) {
  const parsingContext = newParsingContext();
  const evaluator = buildRuleSet(rules, parsingContext);
  const evaluationContext = newEvaluationContext();
  return function (feature, resolution) {
    evaluationContext.properties = feature.getPropertiesInternal();
    evaluationContext.resolution = resolution;
   // if (parsingContext.featureId) {
      const id = feature.getId();
      if (id !== undefined) {
        evaluationContext.featureId = id;
      } else {
        evaluationContext.featureId = null;
      }
    }
 //   if (parsingContext.geometryType) {  //Since this part is not true, the geometry value cannot be obtained.
      evaluationContext.geometryType = computeGeometryType(
        feature.getGeometry(),
      );
    }
    return evaluator(evaluationContext);
  };
}

Thank you for your report, indeed the geometry type had somehow become unusable in expressions. I have come up with a fix, see https://github.com/openlayers/openlayers/pull/16087

Hey @nssang00 , the issue should be fixed on main, could you please try the latest version and confirm that it works? Thanks


text:
document descending order of resolutions array

Fixes #12354

Perhaps there should be an explanation of why, such as  "If specified the resolutions define the view zoom levels and must be in descending order."

Yeah, maybe rephrase the whole thing, because the main purpose of resolutions is not the constraint, but setting the zoom levels. Good point!

Thanks, @simonseyock.

authors: mike-000 simonseyock tschaub


text:
Cut 0.9.0 release?

As soon as the renaming PRs make it in, can we cut an 0.9.0 release, counting up with 0.9.x releases until the paper is submitted (1.0.0)? I'd really like to start building test conda builds to make sure there are no issues at this point but don't want to do this for git hashes.

I see renamings as the only thing that will break backward compatibility at this point (@jhprinz, do you have anything else coming in that would break compatibility?) As far as I'm concerned, we can start cutting releases at any time. I just don't know how frequently we want to, in the build-up to 1.0. Is twice a week enough? It feels like after each merge or even daily would be a bit intense.
My plans for the week to make the SRTIS bias easy to create (it's already possible to do SRTIS, but the input bias is currently a pain to make), progress on renamings/reorgs, and starting on the custom-path mover example for paper 2.
Oh, one other thing that might change the API a little: I think we should move the factory functions outside of a factory class and into separate functions. @jhprinz and I discussed this a while ago (and I think we agreed to do that). I think that's only relevant for the VolumeFactory and EnsembleFactory classes.
Most of what's left after that is either simplifying things that are currently complicated, or polishing things up to make it all look like it was written by actual professionals (docstrings, documentation, metadata, etc.)

I think it's fine to increment from 0.9.0 whenever you squash some important bugs or make API changes at this point.

My plans for the week to make the SRTIS bias easy to create (it's already possible to do SRTIS, but the input bias is currently a pain to make)

Would it be easier just to add a SAMS-based weight adaptation scheme? The implementation takes just a few lines and I'm happy to tackle it.

do you have anything else coming in that would break compatibility?

Just merging the stuff for the alanine example which will add  some convenience functions and add some parameters to the initial_conditions_from_trajectories. This is almost ready. I am still cleaning the notebook. But the API should be fine now. I also move some of the code to be part of other classes for better reusability.
I think we have done all of the renaming for now. Means next is rearranging folders.

Would it be easier just to add a SAMS-based weight adaptation scheme? The implementation takes just a few lines and I'm happy to tackle it.

No. That would require a different subclass of PathSimulator. It isn't the same as standard path sampling. Feel free to do that, because I think it would be great to have, but I think we still need the simplifications I'm talking about.
Everything I need to do is already in the SRTIS notebook. However, previously it needed user intervention to specify what value of the order parameter was associated with each interface. With additions from recent PRs, I can automate that.

Means next is rearranging folders.

I have local branches starting on that already. Waiting for the AD MSTIS example to merge before going too far (in the meantime, creating READMEs for the various subdirectories of the examples)

It should be almost ready. I keep bumping in this I need to fix that are caused by my cleaning up.

Looks like travis is still failing. Any idea how you can fix that?
Would be great to get the test issues fixed before cutting 0.9.0.

Looks like travis is still failing. Any idea how you can fix that?

Re-run it? (which I just did) We may need to increase the number of steps. The last error I saw was due to never running a minus mover. You can't calculate the rate (flux) without that.
The long-term solution is, as I said in #556, to have a separate data file for analysis. In other words, fully separate the integration test for running a simulation from the test of the analysis. We'd do this by running a long simulation (10k+ steps), hosting the resulting .nc file somewhere, and curling it down for tests.
The trade-off is in how frequently we need to update that file. As long as we're making changes that affect the file format, we would have to update that file with each such change. If that it frequently, then it is more of a pain than what we're doing now.
Once we hit 1.0, that file should not change, and in fact it serves as a check for backward compatibility.
@jhprinz : are we close to being able to do that? I'll need to look through my stuff to see what else might affect file format, but I think most of it is already in.

@jhprinz : are we close to being able to do that? I'll need to look through my stuff to see what else might affect file format, but I think most of it is already in.

Yes, I just wrote that in the WHAM discussion. The UUID storage allows you to save a long file without the actual coordinates and still run the analysis completely fine (no showing real trajectories of course)
From my side there will not be any changes to the storage for 1.0 (unless bug appear) so we can go ahead and get these files. I will make some little changes to the way to generate initial samples, but that is independent of creating files.

Note, that for alanine examples we need to use uuid storage (default) for the toy example storing the full file should be fine.

A new proposal:
Let's just fix storage for 0.9.0 until 0.9.5 (if necessary) and final 1.0.0. So that we can start fixing all the rest using working test files.
I would merge #521 if David has the time before 0.9.0 otherwise we release it and keep on iterating. If we need to make changes on storage then only two times. It should not be too much.
Let's gather, what we want to fix already.
Release 0.9.0

fix storage
fix ...

I probably won't have a chance to complete review until Sunday. Today is going to be a short day, and tomorrow I'm actually taking a day off.

0.9.0 package is now online:
https://anaconda.org/omnia/openpathsampling/files
I'm going to delete the 0.0.0 dev package since it's out of date.

0.9.0 package is now online:

Woohoo! Thanks @jchodera!

I'm going to delete the 0.0.0 dev package since it's out of date.

Is there a way for us to properly set up a dev-labeled OPS? Won't be much different from the 0.9.x series while those are being released (so doesn't matter in the near future), but once we release 1.0, I'll rapidly start working on 1.1-dev...

Do you need a dev-labeled OPS? It's actually super easy to install via pip, and if deploying on a new platform, you can always use
conda install --yes openpathsampling
conda remove --yes openpathsampling
pip install git+git://github.com/openpathsampling/openpathsamplin.git@master

I want to deprecate the old packagename-dev scheme, which was causing too much trouble, but we can set up nightly builds pushed to the dev label if you really want. Alternatively, we can figure out how to make your travis builds push just the linux builds to dev label, but this may be of limited utility.

Do you need a dev-labeled OPS?

Probably not -- depends on if that's how people prefer to stay on the edge of the code. Personally, my preference (and suggestion) is to install the git repo as editable (pip install -e), because if you're really on the bleeding edge, you might want to check out an un-merged branch.

authors: dwhswenson jchodera jhprinz

repo: openpathsampling/openpathsampling


text:
GEX/FB intersection module

Hey all,
As discussed here is the ticket to create a module allowing intersection between GEX and FB.
Here is the code we were using:

Thanks!
Isabelle

Thanks for the suggestion! Let's create a component for subsetting the mudata to the set of intersecting cells.

Also, would it be possible to copy paste the code instead of screenshots? This makes it a lot easier for us to work with the code.

Hey Robrecht,
Thanks! Yes indeed it would be nice to keep the info on what amount of cells are removed from each layer and which cells. And potentially add a warning message in case. It could even be all on same object as we commonly look at both the intersection and non-intersection ones.
As a side note, the filtering on FB data is currently super lenient, so we will likely remove lots of cells for which some antibody info is still there but no good quality RNA is detected.
Here are the chunks I used:
import scanpy as sc
import muon as mu

mdata = mu.read_h5mu(path)
mdata = mdata[(mdata.obs.index.isin(mdata["rna"].obs.index))&(mdata.obs.index.isin(mdata["prot"].obs.index))]
mdata.update()

# remove calculated layers and re-run with new subset
del mdata['rna'].var['highly_variable']
del mdata['rna'].uns['pca_variance']
del mdata['rna'].obsm['X_pca']
del mdata['rna'].obsm['X_umap']
del mdata['rna'].varm['pca_loadings']
del mdata['rna'].obsp['distances']
del mdata['rna'].obsp['connectivities']
del mdata['rna'].uns['neighbors']

# remove calculated layers and re-run with new subset
del mdata['prot'].uns['pca_variance']
del mdata['prot'].obsm['X_pca']
del mdata['prot'].obsm['X_umap']
del mdata['prot'].varm['pca_loadings']
del mdata['prot'].obsp['distances']
del mdata['prot'].obsp['connectivities']
del mdata['prot'].uns['neighbors']

mdata['rna'].uns['pca_variance'] = {}
del mdata['rna'].uns['pca_variance']

mdata['prot'].uns['pca_variance'] = {}
del mdata['prot'].uns['pca_variance']
mdata.update()

mdata['rna'].uns['log1p']['base'] = None

tmp_rna = mdata['rna'].X.copy()
mdata['rna'].X = mdata['rna'].layers['log_normalized']

hvg_object = sc.pp.highly_variable_genes(mdata['rna'],  flavor = "seurat", batch_key = "sample_id", subset=False, inplace=False) # careful, 'layer' option not working!
mdata['rna'].var['highly_variable'] = hvg_object['highly_variable']
sc.tl.pca(mdata['rna'], n_comps = 25, copy = False, use_highly_variable = True)
mdata['rna'].X = tmp_rna

tmp_prot = mdata['prot'].X.copy()
mdata['prot'].X = mdata['prot'].layers['clr']

sc.tl.pca(mdata['prot'], n_comps= 25, copy=False, use_highly_variable = False)

mdata['prot'].X = tmp_prot

mdata.write(path_v2)

Main library versions:
scanpy==1.9.2 anndata==0.9.2 umap==0.5.3 numpy==1.22.4 scipy==1.6.3 pandas==1.5.3 scikit-learn==0.24.2 statsmodels==0.12.2 python-igraph==0.10.6 louvain==0.8.1 pynndescent==0.5.10
I have everything in my project folder. I can point it to you too.

authors: Isabelle-b rcannood

repo: openpipelines-bio/openpipeline


text:
Add getting amount keywords

This change is

Review status: 0 of 1 files reviewed at latest revision, 2 unresolved discussions.

op_robot_tests/tests_files/brokers/openprocurement_client.robot, line 290 at r1 (raw file):

  ${reply}=  Call Method  ${USERS.users['${username}'].client}  patch_question  ${tender}  ${answer_data}
  [return]  ${reply}

Має бути ще один порожній рядок :)

op_robot_tests/tests_files/brokers/openprocurement_client.robot, line 293 at r1 (raw file):

Отримати кількість запитань до тендера
  [Arguments]  ${username}  ${tender_uaid}
  ${tender}=  openprocurement_client.Пошук тендера по ідентифікатору  ${username}  ${tender_uaid}

Оскільки більш-менш однаковий код написаний тричі, раджу замінити його пітонівською функцією.
Щось типу отакого:
get_length_of_item(data, key):
    return len(data.get(key, []))

В такому випадку ключове словов можна буде переписати отак:
Отримати кількість запитань до тендера
  [Arguments]  ${username}  ${tender_uaid}
  ${tender}=  openprocurement_client.Пошук тендера по ідентифікатору  ${username}  ${tender_uaid}
  ${number_of_questions}=  get_length_of_item  ${tender.data}  questions
  Log  ${number_of_questions}
  [Return]  ${number_of_questions}

Як на мене, виглядає красивіше.

Comments from Reviewable

Review status: 0 of 2 files reviewed at latest revision, 2 unresolved discussions.

op_robot_tests/tests_files/brokers/openprocurement_client.robot, line 290 at r1 (raw file):
Previously, mykhaly (Yurii Mykhalchuk) wrote…
Має бути ще один порожній рядок :)

Done.

op_robot_tests/tests_files/brokers/openprocurement_client.robot, line 293 at r1 (raw file):
Previously, mykhaly (Yurii Mykhalchuk) wrote…
Оскільки більш-менш однаковий код написаний тричі, раджу замінити його пітонівською функцією.
Щось типу отакого:
get_length_of_item(data, key):
    return len(data.get(key, []))

В такому випадку ключове словов можна буде переписати отак:
Отримати кількість запитань до тендера
  [Arguments]  ${username}  ${tender_uaid}
  ${tender}=  openprocurement_client.Пошук тендера по ідентифікатору  ${username}  ${tender_uaid}
  ${number_of_questions}=  get_length_of_item  ${tender.data}  questions
  Log  ${number_of_questions}
  [Return]  ${number_of_questions}

Як на мене, виглядає красивіше.

Справді красивіше, дякую)

Comments from Reviewable

authors: OrysiaDrabych mykhaly


text:
Fix/PC console refactor

This improves PC console performance by attempting to load all data in parallel (profiles still must be loaded after other data loads). In addition the configuration table is now loaded independently so that it shows almost immediately. Other tabs are not rendered until actually clicked on.
This PR also fixes a bug where selecting all papers in a filtered list actually selected all submitted papers.
Related to https://github.com/openreview/openreview/issues/1888

Whew tests are finally passing! This is ready to review.

repo: openreview/openreview-py


text:
PC/SAC/AC console - add new sort options

Alina requested to add 2 new sort options
rating range/discrepancy =max rating - min rating
confidence range/discrepancy =max confidence - min confidence
this is to "flag where there is a significant discrepancy in scores"

@melisabok @celestemartinez i'm going to add this to neurips pc/sac/ac console


text:
[BUG] case_insensitive wildcard query didn't return expected result

Describe the bug
When perform wildcard query on wildcard field with case_insensitive set to true, the expected doc didn't occur.
Related component
Search
To Reproduce

Create a simple index contains wildcard field
PUT case_test
{
    "mappings": {
        "properties": {
            "wildcard": {
                "type": "wildcard"
            }
        }
    }
}

Bulk insert documents contains capital letters
PUT _bulk?refresh=true
{"index": {"_index": "case_test"}}
{"wildcard": "TtAa"}
{"index": {"_index": "case_test"}}
{"wildcard": "ttaa"}
{"index": {"_index": "case_test"}}
{"wildcard": "TTAA"}

Search on wildcard field with case_insensitive set to true and false respectly
case sensitive:
POST case_test/_search
{
    "query": {
        "term": {
            "wildcard": {
                "value": "TtAa",
                "case_insensitive": false
            }
        }
    }
}

case insensitive:
POST case_test/_search
{
    "query": {
        "term": {
            "wildcard": {
                "value": "TtAa",
                "case_insensitive": true
            }
        }
    }
}

Check result

Expected behavior
case insensitive query should return all three documents, instead of TtAa alone
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):

OS: [e.g. iOS]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

bugfix is here #15882

repo: opensearch-project/OpenSearch


text:
Support for disabling BKDReader packedIndex off heap

Is your feature request related to a problem? Please describe.
In Lucene 8.4 the BKDReader packedIndex was moved off heap (LUCENE-8932).  In some cases this can cause the BKDReader to consume more CPU cycles and increase request latency
Describe the solution you'd like
Add a setting to allow the BKDReader packedIndex to be moved on heap.
Describe alternatives you've considered
Move the BKDReader packedIndex on heap by default, but this would likely increase memory consumption for users.
Additional context
Similar issue - https://github.com/opensearch-project/OpenSearch/issues/825
Flame graphs are included for Elasticsearch versions 7.9 and 7.1
ES 7.9 Lucene 8.6.2

ES 7.1 Lucene 8.0.0

I think I'm hitting this too... 8-10x search latency increase on 7.9 vs 7.1. Do certain types of queries / operations cause this more than others?
Also, from the Lucene bug I see this:
"I have added a second constructor that takes both the IndexInput and a boolean for on/off-heap. The first constructor will select off-heap iff the IndexInput extends ByteBufferIndexInput."
I'm wondering if there's any way to make IndexInput not an instance of ByteBufferIndexInput. I don't know the code, so I don't know what else it possibly could be, or how to get there, or what (if any) ramifications that might have.

It was made default in LUCENE-9398 from 8.6. We need support in lucene that can be exposed as setting in Opensearch for moving it on/off heap as per the use case

+1 regarding the feature to enable/disable BKDReader on-heap/off-heap, for select use cases. Teams that are upgrading from ES 7.1 to ES 7.9 are observing increased SearchLatencies as called out on this issue.
My limited understanding is that BKDReader data structure is used for searching and indexing high-dimensional data. BKDReader off heap was a performance optimization to reduce memory usage and improve overall scalability of Elasticsearch.
May I request additional insights and guidance to below mentioned topics - as part of this github issue Or should this be discussed and addressed separately?

List the "query patterns" that exercise BKDReader. In my limited experience, BKDReader gets exercised when range queries are executed on numeric fields. Are there additional query patterns that exercise BKDReader? or Is there additional guidance for users of OpenSearch to understand the "query patterns" that would exercise BKDReader under the covers.
What are the  "access patterns", "use cases" or "scenarios" wherein BKDReader on-heap performs better than BKDReader off-heap (and vice-versa)? It appears that BKDReader on-heap may be better suited when query performance compromise is not acceptable, and dimensionality of underlying data is not very high. Thoughts?

More context is needed here. Direct memory (mmaped files) is subject to swapping. Was this tested w/ swapping disabled which can seriously impact a nodes performance? e.g.,:

Was swapping enabled? Try rerunning w/ and w/o swap enabled and report findings.

Try memory locking the process bootstrap.memory_lock true

We're ulimits set to unlimited?

opensearch soft memlock unlimited
opensearch hard memlock unlimited

Was swapping enabled?

No, swapping is disabled due to performance degradation concerns

We're ulimits set to unlimited?

Yes they are:
%cat '/etc/security/limits.d/10-memlock.conf'
*         hard    memlock           unlimited
*         soft    memlock           unlimited

In some cases this can cause the BKDReader to consume more CPU cycles and increase request latency

Can we get more information on the "some cases"? The flamegraph is pretty but we need more context. Can we get a query profile before/after? Field mappings? Segment geometry?
More specifically, this change was made in Lucene which impacts not just OpenSearch but Solr and Elasticsearch. I have yet to see any concern from the latter two. It's possible they haven't hit it yet, but any changes to upstream Lucene will need more context with supporting benchmarks to clearly show this regression results from the BKD on/off heap changes.
With supporting query information it will also be easier to repro this using the lucene benchmark tool to support a change in the upstream. It's unfortunate no benchmarks were provided with the original change.

As long as the index is fully hot (is it here?), moving the BKD index off-heap should not cause anything near the performance regression that the o.p. flame charts seem to indicate.  The BKD index is still in RAM in both cases, just accessed via slightly different APIs.  Also, traversing the BKD index is not the hot spot over the overall BKDReader.intersect -- visiting the leaf blocks (which are off-heap in both cases) is where the heavy work happens.
Could you attach a delta flame-chart as well?  That would show us precisely what's slower and what's faster.  And, any way to attach or link to a fully interactive flame chart?  We need the equivalent of Pastebin but slightly interactive so I can drill down on the flame chart.
Also note that in Lucene's nightly benchmarks, the IntNRQ query tasks tests BKD performance, and it does not show any regression in the June 20, 2020 time-frame:
https://home.apache.org/~mikemccand/lucenebench/IntNRQ.html
Aside: it's nice to see the GC reduction from old to new Lucene versions in the o.p. flame charts.

As long as the index is fully hot (is it here?)

Good question. I assumed it was during the run but the description doesn't give any specifics on the configuration when the the flame graph was obtained. Was this obtained from a single node in a cluster? Was this a single node setup? What else was going on during the query (e.g., concurrent indexing) or is this isolated to the query only?  Is there a hot threads dump?
The more details to help here the better.

For the offending queries you might also try warming the page cache on those indexes using the "index.store.preload": ["*"] index setting to make sure the index is hot. Although if this regression is seen in a production cluster I think that could be evicted based on the 30second Shard Idle default. That might be another area to look at, increase index.search.idle.after to try and prevent shard idle (e.g., file cache evictions). I think if you set this to -1 it will disable? (double check that)

Which Directory implementation is in use here @travisbenedict?  If it's a buffered implementation (SimpleFSDirectory or NIOFSDirectory) can you try switching to MMapDirectory instead?  The buffered reads are sometimes costly, e.g. see https://github.com/apache/lucene/issues/12356.

Also, if possible, please render flame charts to SVG so they remain interactive after attaching to GitHub issues.

authors: jainankitk manojfaria mikemccand nknize superawesome travisbenedict


text:
Point in time based searches should provide an option to ignore missing indices

Is your feature request related to a problem? Please describe.
If a point in time is created against an index that is then deleted, searches using that point in time will fail. This is problematic if a point in time is created against all indices, and one of those indices is then dropped while the PIT is still in use. An example of this occurring is that lifecycle management may drop an old index in the middle of a search.
A minimal way to reproduce this is:

Stand up an OpenSearch cluster.
Add 2 indices -- index-1 and index-2
Create a PIT against _all
(Optionally) Test that a search with the PIT works
Delete index-2
Test that a search with the PIT fails with a 404 indicating that an index is missing.

Describe the solution you'd like
Allow PITs to specify something like allow_partial_search_results which then returns search results while ignoring missing indices similar to non-PIT searches.
Describe alternatives you've considered
This can be managed on the client by creating a new PIT and searching again, however this is cumbersome.
Additional context
Elasticsearch have tracked a similar issue in https://github.com/elastic/elasticsearch/issues/81256 and have implemented a fix to the problem described in https://github.com/elastic/elasticsearch/pull/81349

As an admin task, we should probably create a dedicated label for point-in-time search, to ensure that the right folks get their eyes on this. Those of us who follow the Search label probably don't have enough context to dive in on PIT stuff.
In the meantime, @bharath-techie  -- do you know of a project where we can assign this so that it gets some traction? Thanks!

The one in which we were tracking PIT backend tasks is closed - https://github.com/opensearch-project/OpenSearch/issues/1147
We can create a new one if needed.

authors: SingingTree bharath-techie msfroh


text:
Gradle custom java zippublish plugin

Description
This change will add custom zippublish plugin, that can publish plugin zips to maven repo.
Issues Resolved
https://github.com/opensearch-project/opensearch-build/issues/1916
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Will gladly review once you think this is ready!

Thanks @dblock ,
the reason I still left as draft is, I still dont see  the plugin gets added to local maven repo (using build tools) during runtime so that downstream components can use it while the plugin is applied as apply plugin: 'opensearch.zippublish'
However when I add the plugin manually to local maven repo then it gets picked when apply plugin is added.
So just looking for some suggestions if anything I have missed. Just to summarize the plugin functionality works fine as expected, just the pending part is how to integrate with the existing custom plugin setup, so that even this is  added to local maven repo during runtime and each component can then use it.
@saratvemulapalli

@dblock checking to add some more unit tests, as this not an actual task (tasks unit tests could be just calling the task), but a apply() method, I might need to add some dummy task that can use this apply() method.

Hey @reta and @dblock I have done some code cleanup as per feedback and added the following.

Removed the entire extension file, as this only used to fetch the zip's groupID, since its fixed as org.opensearch.plugin, I have hardcoded it, which makes the extensions file not required anymore.
Edited the existing PublishPlugin, to skip if the publication has name zip in it.
Added proper unit tests, that covers edge cased related to maven zip publish.
Tested with job-scheduler, alerting and anomaly-detection plugin, able to get the zips published excluding the sourceJar and javadocJar only for this zip publication.

@bbarani

@prudhvigodithi WDYT? Possible?

yes @dblock possible, but how about the META-INF, is the existing one (opensearch.zippublish) fine ?, users should be able to use this with apply apply plugin: 'opensearch.zippublish'

I didn't pay attention and this got merged onto 2.0, it shouldn't have been as 2.0 is stabilizing for RC1 😢

Created Manual Backport PR's
Backport to main: PR
Backport to 2.x: PR

authors: dblock prudhvigodithi


text:
Add additional logging for future debugging of HA proxy mode flaky test

Signed-off-by: Ryan Bogan rbogan@amazon.com
Description
The current logging is not sufficient to fix flaky test failures for testHAProxyModeConnectionWorks . The failure occurs due to the numer of connected sockets being 0.  However, the RemoteConnectionInfo containing this data is obtained through a transport request, so the cause of the sockets not connecting is unknown.  Since the error is non-reproducible, this PR adds additional logging when there are no connected sockets, printing out the cluster health at the time of failure.  If the flaky test failure occurs again, there will be more logged information that can hopefully lead to a solution.
Check List

[X] New functionality includes testing.

[X] All tests pass

[X] New functionality has been documented.

[X] New functionality has javadoc added

[X] Commits are signed per the DCO using --signoff
[X] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

I'm not sure if this change warrants being in the CHANGELOG, since nothing was technically fixed.  Can someone advise me on where to put it in the CHANGELOG if it should be there?

Codecov Report

Merging #5667 (edd06fc) into main (f2b5044) will increase coverage by 0.11%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #5667      +/-   ##
============================================
+ Coverage     70.96%   71.08%   +0.11%
- Complexity    58550    58628      +78
============================================
  Files          4760     4760
  Lines        279515   279515
  Branches      40348    40348
============================================
+ Hits         198367   198697     +330
+ Misses        65037    64596     -441
- Partials      16111    16222     +111

Impacted Files
Coverage Δ

.../admin/cluster/reroute/ClusterRerouteResponse.java
60.00% <0.00%> (-40.00%)
:arrow_down:

...luster/routing/allocation/RoutingExplanations.java
62.06% <0.00%> (-37.94%)
:arrow_down:

.../opensearch/client/indices/CloseIndexResponse.java
56.25% <0.00%> (-35.00%)
:arrow_down:

...cluster/routing/allocation/RerouteExplanation.java
70.00% <0.00%> (-30.00%)
:arrow_down:

...rc/main/java/org/opensearch/ingest/IngestInfo.java
51.72% <0.00%> (-27.59%)
:arrow_down:

...a/org/opensearch/index/mapper/MapperException.java
75.00% <0.00%> (-25.00%)
:arrow_down:

...nsearch/index/shard/IndexingOperationListener.java
63.93% <0.00%> (-22.96%)
:arrow_down:

...pensearch/index/mapper/MapperParsingException.java
77.77% <0.00%> (-22.23%)
:arrow_down:

...a/org/opensearch/transport/netty4/Netty4Utils.java
42.42% <0.00%> (-21.22%)
:arrow_down:

...main/java/org/opensearch/ingest/ProcessorInfo.java
47.36% <0.00%> (-21.06%)
:arrow_down:

... and 479 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter ryanbogan


text:
Implement equals/hashcode for named DocValueFormat inner classes

Description
Implements  equals and hashCode methods for the named inner class DocValueFormat.DateTime.
Per the contract as described in the Writeable interface, any class implementing equals/hashcode must ensure that a copy made be serializing and deserializing must be equal. As InternalDateHistogram /  InternalHistogram implement equals/hashcode and are Writable classes, this requires that DateTime also implement equals and hashcode , as the DateMathParser will otherwise cause it to fail equivalence.
Check List

[x] New functionality includes testing.

[x] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #6357 (fe01144) into main (c772596) will increase coverage by 0.12%.
The diff coverage is 42.85%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #6357      +/-   ##
============================================
+ Coverage     70.58%   70.71%   +0.12%
- Complexity    58841    59041     +200
============================================
  Files          4799     4799
  Lines        282421   282428       +7
  Branches      40717    40720       +3
============================================
+ Hits         199341   199711     +370
+ Misses        66584    66315     -269
+ Partials      16496    16402      -94

Impacted Files
Coverage Δ

...org/opensearch/common/time/JavaDateMathParser.java
96.46% <42.85%> (-3.54%)
:arrow_down:

...java/org/opensearch/client/indices/DataStream.java
0.00% <0.00%> (-76.09%)
:arrow_down:

...adonly/AddIndexBlockClusterStateUpdateRequest.java
0.00% <0.00%> (-75.00%)
:arrow_down:

...readonly/TransportVerifyShardIndexBlockAction.java
9.75% <0.00%> (-73.18%)
:arrow_down:

...rch/test/junit/listeners/ReproduceInfoPrinter.java
9.52% <0.00%> (-68.26%)
:arrow_down:

...a/org/opensearch/client/cluster/ProxyModeInfo.java
0.00% <0.00%> (-60.00%)
:arrow_down:

...n/admin/indices/readonly/AddIndexBlockRequest.java
17.85% <0.00%> (-53.58%)
:arrow_down:

...ch/transport/ReceiveTimeoutTransportException.java
50.00% <0.00%> (-50.00%)
:arrow_down:

...pensearch/indices/breaker/CircuitBreakerStats.java
27.77% <0.00%> (-41.67%)
:arrow_down:

...indices/readonly/TransportAddIndexBlockAction.java
20.68% <0.00%> (-41.38%)
:arrow_down:

... and 537 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@bryanlb What would you like to do with this?

@dblock I'd still like to see this addressed, with the most recent suggestion from @reta. We can close this PR out and if there's interest in fixing this, I can followup with a PR that introduces those equals()/hashcode() fixes on the DocValueFormat class.

I ran across this when attempting to compare the equivalence of an InternalDateHistogram instance that had been serialized and deserialized.

@bryanlb Can you share more details about what you're trying to do at a higher level that led you to need to do this?

@andrross sure! I have a slightly unique case where I'm importing the server package as a dependency, and then using the AggregationBuilder classes to create Lucene collectors. I've had no problems with this approach - consuming the collectors, or utilizing the returned InternalAggregation classes.
I only ran into this issue when attempting to verify that serializing an InternalAggregation and deserializing it resulted in an equivalent object. While the Writeable interface notes that implementers aren't required to implement equals/hashcode, it does state that "If the implementer also implements equals and hashCode then a copy made by serializing and deserializing must be equal and have the same hashCode". In the case of InternalDateHistogram this class does implement both an equals and hashcode, but does not maintain this stated contract.

@bryanlb So are you using the OpenSearch server as a library in a separate (non-OpenSearch) application? It's always interesting to hear unique use cases :) I will caution that there are no stability guarantees with these internal classes so you may frequently encounter breaking changes.
That's a fair point about not correctly implementing the contract defined by the Writeable interface. I'm not sure about implementing equals/hashcode in DocValuesFormat (though I am no expert in this area). A possible alternative: the only information InternalDateHistogram sends over the wire for its DocValuesFormat field is it's name (DocValuesFormat is a NamedWriteable), would it be fair to use format.getWriteableName() in the equals/hashcode implementation of InternalDateHistogram?

That's a fair point about not correctly implementing the contract defined by the Writeable interface. I'm not sure about implementing equals/hashcode in DocValuesFormat (though I am no expert in this area).

@andrross Just to give some context on the issue at large, the InternalDateHistogram and others like  InternalHistogram do implement the equals / hashCode using internal state. That basically assumes that internal components do implement the equals / hashCode as well, which in case of InternalDateHistogram means equality for DocValueFormat:
    ...
    private final DocValueFormat format;
    ...

It think this is indeed a problem (may be uncovered in unusual way by @bryanlb) but we need to fix.

It think this is indeed a problem (may be uncovered in unusual way by @bryanlb) but we need to fix.

@reta Got it. Seems like we should override equals/hashcode in each DocValueFormat concrete class. Should be pretty straightforward.

@andrross @reta I can get an update done for those changes next week. Do we want to keep it tied to this PR or would you prefer a new one that references this?

@bryanlb LGTM, could you please add the entry to CHANGELOG? thank you

Backport: https://github.com/opensearch-project/OpenSearch/pull/6584

authors: andrross bryanlb codecov-commenter dblock kotwanikunal reta


text:
[Extensions] Removing support for onIndexModule extension point

Description
onIndexModule extension point was the first one to POC OpenSearch <-> extension communication.
The change was half baked with most of the feature of IndexModule not implemented.
Removing this code for now as its un-used and opened up https://github.com/opensearch-project/opensearch-sdk-java/issues/776 to implement the feature cleanly.
Related Issues
Part of #7402
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15997/
CommitID: 3f696ba
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus
org.opensearch.backwards.IndexingIT.testSyncedFlushTransition
org.opensearch.backwards.IndexingIT.testSeqNoCheckpoints
org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus
org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus
org.opensearch.backwards.IndexingIT.testUpdateSnapshotStatus
org.opensearch.upgrades.RecoveryIT.testAutoExpandIndicesDuringRollingUpgrade
org.opensearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testAutoExpandIndicesDuringRollingUpgrade
org.opensearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testAutoExpandIndicesDuringRollingUpgrade
org.opensearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testAutoExpandIndicesDuringRollingUpgrade
org.opensearch.upgrades.RecoveryIT.testRecoveryWithConcurrentIndexing
org.opensearch.upgrades.RecoveryIT.testRelocationWithConcurrentIndexing
org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness


text:
remove jcenter dependency

Signed-off-by: Amit Galitzky amgalitz@amazon.com
Description
Removed jcenter dependency as it is no longer operating. We have mavenCentral() as a fallback.
Issues Resolved
https://github.com/opensearch-project/anomaly-detection/issues/357
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

fyi: if you add closes or resolves keyword in front of the issue number, it will automatically link it and close it once the PR is merged

#357

Thanks for the tip, just added it

fyi: if you add closes or resolves keyword in front of the issue number, it will automatically link it and close it once the PR is merged

thanks for the tip, just added it

Codecov Report

Merging #361 (c1a8f80) into main (af2763c) will decrease coverage by 0.36%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #361      +/-   ##
============================================
- Coverage     79.12%   78.75%   -0.37%
+ Complexity     4090     4063      -27
============================================
  Files           295      295
  Lines         17207    17207
  Branches       1826     1826
============================================
- Hits          13615    13552      -63
- Misses         2700     2755      +55
- Partials        892      900       +8

Flag
Coverage Δ

plugin
78.75% <ø> (-0.37%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...java/org/opensearch/ad/task/ADBatchTaskRunner.java
77.81% <0.00%> (-3.96%)
:arrow_down:

...g/opensearch/ad/model/DetectorValidationIssue.java
64.91% <0.00%> (-3.51%)
:arrow_down:

src/main/java/org/opensearch/ad/model/ADTask.java
94.96% <0.00%> (-2.78%)
:arrow_down:

...c/main/java/org/opensearch/ad/util/ParseUtils.java
70.03% <0.00%> (-1.56%)
:arrow_down:

...ain/java/org/opensearch/ad/task/ADTaskManager.java
74.03% <0.00%> (-1.29%)
:arrow_down:

...ava/org/opensearch/ad/task/ADTaskCacheManager.java
88.36% <0.00%> (-0.96%)
:arrow_down:

...rch/ad/transport/ForwardADTaskTransportAction.java
93.22% <0.00%> (-0.85%)
:arrow_down:

...in/java/org/opensearch/ad/model/AnomalyResult.java
82.27% <0.00%> (-0.67%)
:arrow_down:

...opensearch/ad/indices/AnomalyDetectionIndices.java
72.28% <0.00%> (ø)

...ava/org/opensearch/ad/task/ADHCBatchTaskCache.java
91.35% <0.00%> (+2.46%)
:arrow_up:

... and 1 more

authors: amitgalitz codecov-commenter ohltyler

repo: opensearch-project/anomaly-detection


text:
implements explorer data grid column reordering

Description
Implements data grid column reordering within the column toolbar drag and drop. Also implements it within the sidebar fields drag and drop, where the order of the selected fields can be manipulated but the available fields will still be sorted alphabetically.
Issues Resolved
This PR also resolves an issue within app analytics where the sidebar isn't populated with any fields.
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@paulstn please resolve the conflicts

authors: mengweieric paulstn


text:
Update Developer guide

Signed-off-by: Rupal Mahajan maharup@amazon.com
Description

Add changes wrt new repo in Developer guide

Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Changes to my branch are not reflecting here since the repo became public. Created this instead https://github.com/opensearch-project/dashboards-reporting/pull/18 from new fork.

repo: opensearch-project/dashboards-reporting


text:
Navigate to Import Sample Data Page without Refresh when New Home is Enabled

Description
This PR addresses the issue of navigating to the import sample data page without causing a page refresh when the new home page and workspace are enabled.
Changes:

Modified the CreateIndex component to check if the new home page and workspace are enabled.
Added an onClick handler to the "Add Sample Data" link when the new home page and workspace are enabled.
In the onClick handler, called application.navigateToApp with import_sample_data to navigate to the import sample data page without causing a page refresh.

Issues Resolved
#456
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Hi @sejli , Yes. I think this changes should be backport to 2.x branch. Feel free to help me add the label, thank you.


text:
Wire in source_coordination config and SourceCoordinator interface to Source plugins

Description
This adds boilerplate and wiring for taking a source_coordination configuration from the data-prepper-config.yaml and instantiating the correct type of SourceCoordinator
The SourceCoordinator interface is provided to sources that implement RequiresSourceCoordination. This interface is not implemented or finalized, but is a starting point to allow for calls to this interface from the source.
The S3Source now implements RequiresSourceCoordination as a proof of concept for passing the SourceCoordinator interface (I can remove this from the S3Source before merging since it is unused)
Issues Resolved
Check List

[x] New functionality includes testing.
[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #2395 (cced817) into main (b3142e0) will increase coverage by 0.13%.
The diff coverage is 86.88%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #2395      +/-   ##
============================================
+ Coverage     93.67%   93.80%   +0.13%
- Complexity     1998     2021      +23
============================================
  Files           234      241       +7
  Lines          5546     5605      +59
  Branches        443      447       +4
============================================
+ Hits           5195     5258      +63
+ Misses          243      237       -6
- Partials        108      110       +2

Impacted Files
Coverage Δ

...per/parser/config/PipelineParserConfiguration.java
100.00% <ø> (ø)

...sourcecoordination/SourceCoordinatorAppConfig.java
0.00% <0.00%> (ø)

.../sourcecoordination/DynamoDbSourceCoordinator.java
33.33% <33.33%> (ø)

...arch/dataprepper/model/source/SourcePartition.java
100.00% <100.00%> (ø)

.../opensearch/dataprepper/parser/PipelineParser.java
87.64% <100.00%> (+0.14%)
:arrow_up:

...prepper/parser/model/DataPrepperConfiguration.java
88.73% <100.00%> (+1.77%)
:arrow_up:

...ination/DynamoDBSourceCoordinationStoreConfig.java
100.00% <100.00%> (ø)

.../model/sourcecoordination/DynamoStoreSettings.java
100.00% <100.00%> (ø)

...l/sourcecoordination/SourceCoordinationConfig.java
100.00% <100.00%> (ø)

.../org/opensearch/dataprepper/pipeline/Pipeline.java
81.39% <100.00%> (+0.67%)
:arrow_up:

... and 1 more

... and 2 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

repo: opensearch-project/data-prepper


text:
Research on operational readiness review for AWS managed service

Some work need to be done before the admin UI feature is taken into the AWS OpenSearch Service:

Provide ORR for the admin UI feature.
Provide some technical support documentation.
Provide some metrics dashboard if need.

All of these things need to do research firstly.

Some work are done:

Write some technical support document for shrink index operation.
Learn how to collect metrics in managed service.
Learn how to do ORR reivew.


text:
Allow a dynamic target index in ISM Rollup

 Issue by mark-meyer
Wednesday Mar 31, 2021 at 00:46 GMT
Originally opened as https://github.com/opendistro-for-elasticsearch/index-management/issues/428

I would like to apply a policy to daily log indices that performs a rollup in one of the states before deleting the index. However is seem the target_index requires a string and there is no option to interpolate something like {{ctx.index}} or similar. This makes it impossible to apply this policy to more than one index unless you are okay with rollup overwriting the previous one, which you probably aren't.
Describe the solution you'd like
I would like to specify the target_index field of an ism rollup dynamically based on the name (or some other metadata) of the index being rolled up such as "target_index": "rollup_{{ctx.index}}"

 Comment by dbbaughe
Wednesday Mar 31, 2021 at 00:51 GMT

I believe @thalurur just did something similar for snapshot name too.
That being said, correct me if I'm wrong Ravi, but if @mark-meyer applies this policy to daily log indices even if they roll up into the same target index they will not overwrite each other. We do support multiple rollups in the same index. Each index that is managed by the policy would have a rollup job created for itself when the rollup action is created which will use its own rollup ID as a namespace in the target index to handle conflicts. And the rollup ID seems to consist of a hash of the job configuration and the source index which will be unique per daily log index.
Edit: Do we delete these temporary rollup jobs after they finish @thalurur?

 Comment by thalurur
Monday Apr 05, 2021 at 22:30 GMT

At the moment ISM action will not delete the jobs after they are finished. Rest of the behavior described by Drew is accurate - at the moment the same policy will be creating multiple jobs (based on the source index the policy is being executed on) and all of these jobs writing data to the same target index. And data from each job cannot be touched by other jobs as their namespace includes the rollup job id.
Rollup implementation have some constraints imposed at the moment:

If multiple rollup jobs have written data to same target index then data from only one job is picked when searching the rollup index, there is no ability to make the search check through all the jobs at the moment.
If search requests consists of multiple rollup indices then the search on rollup indices is failed.

The above makes the only way to rollup data for daily log indices is to create one job that writes to one target index. So the rolled up data can be searched together.
What happens with current policy:
In this case for each new daily log index a new rollup job is created, but all the jobs write to the same target index.
Though the documents from different rollup jobs inside target index are shielded from overwriting, fetching documents from this index is constrained by constraint "1". During search on the target index a single job is picked from multiple jobs and this will prevent ever fetching all the data or user desired job data from target index.
What happens with proposed policy:
If we update the rollup ISM policy to take in a scripted target index, then for each new daily log index a new rollup job is created, writing to a new target index.
In this case since only one job is writing to target index during search there is no issue of which job is created and user can pick which index they want to search from. But, if user want to search across target indices or all target indices its not possible.
We have few options if we need to support either:

Adjust the constraint "2" - multiple rollup indices cannot be searched together. Adjusting this constraint and making the ISM rollup policy target index scriptable like requested will make it possible to search across all ISM policy created rollup jobs as long as all jobs wrote to different index.
Adjust the constraint "1" - make it possible to search multiple rollup job data on the same target index, this makes the current policy work
A combination of both - make it possible to search multiple rollup jobs on same target index and allow multiple rollup indices to be searched in same search request

@mark-meyer Couple of questions:

Is there an use case where you would need to search all the data from different source indices that is rolled up through ISM policy together? (I am assuming you do cause since rollup definition is same in policy for all jobs, the data is just spread out temporally)
If 1 is yes, do you need the data to be rolled up to different indices or having it single index works as well?

 Comment by garlicsauce
Wednesday Apr 28, 2021 at 11:00 GMT

+1
I was trying to setup rollup policy with target index containing date math expression "target_index": "<test-{now/d}>",. Index was created - test-2021.04.28 however rollup job fails with an error saying
Status
Failed: Failed to update mappings of target index [<test-{now/d}>] with rollup job

Without such possibility I guess I need to create some script running every day that would create rollup jobs itself which is much uglier solution. To sum up - it would be nice to be able to setup a rollup job/rollup policy with dynamic target index.

Additional comments:

Be able to use an alias as the rollover target
Be able to have date math in the rollover target index name so it dynamically changes each day
Allow for dynamic index naming based off index rolled up (e.g., if done via ISM action on a rolled index, against a source index of logs-90-000422, a config like %{indexname}-rollup would create logs-90-000422-rollup)

Main issue being that the target of the rollup is currently static, which means once that target index is too large then you need to manually stop the rollup job and start a new one to point to a new index. We need to be able to have a single continuous rollup job that can write to a target that can change (by using an alias where backing indices are rolled over, maybe data streams, dynamically configured index names, etc).

To sum up, the ask is

Allow target-index name in a rollup job to be dynamic so it is possible to rollup data into multiple time-based indices instead of one index (as it is today).
As a follow up to 1, when the data is available in multiple target indices, a user should be able to run search across all rolled up indices. Currently search is only supported for one rollup index at a time since the rollup data goes into just one target index. Tracking issue - https://github.com/opensearch-project/index-management/issues/321

@mark-meyer @garlicsauce does this address your use case?

@setiah Here is example when using scripted target_index field:

Index template

PUT _index_template/ism_rollover
{
  "index_patterns": ["log*"],
  "template": {
   "settings": {
    "plugins.index_state_management.rollover_alias": "log"
   }
 }
}

ISM policy

PUT _plugins/_ism/policies/rollover_with_rollup_policy
{
  "policy": {
    "description": "Example rollover policy.",
    "default_state": "rollover",
    "states": [
      {
        "name": "rollover",
        "actions": [
          {
            "rollover": {
              "min_doc_count": 1
            }
          }
        ],
        "transitions": [
          {
            "state_name": "rp"
          }
        ]
      },
      {
        "name": "rp",
        "actions": [
          {
            "rollup": {
              "ism_rollup": {
                "target_index": "rollup_{{ctx.source_index}}",
                "description": "Example rollup job",
                "page_size": 200,
                "dimensions": [
                  {
                    "date_histogram": {
                      "source_field": "ts",
                      "fixed_interval": "60m",
                      "timezone": "America/Los_Angeles"
                    }
                  },
                  {
                    "terms": {
                      "source_field": "message.keyword"
                    }
                  }
                ],
                "metrics": [
                  {
                    "source_field": "msg_size",
                    "metrics": [
                      {
                        "sum": {}
                      }
                    ]
                  }
                ]
              }
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": [
        "log*"
      ],
      "priority": 100
    }
  }
}

Create index

PUT log-000001
{
  "aliases": {
    "log": {
      "is_write_index": true
    }
  }
}

Insert some docs

POST log/_doc?refresh=true
{
  "ts" : "2022-08-26T09:28:48+00:00",
  "message": "aaa1234",
  "msg_size": 10
}

Notice rollover_with_rollup_policy has been added to log-000001. Rollover action will be executed and then rollup action. After executing rollup action, rollup job would create rollup_log-000001 index.
log-000002 is created after rollover and rollover_with_rollup_policy will be attached to it

Scripted target index is complete for mustache templates but still needs support for aliases.
There are two clear options I see for alias:

Add a flag when creating a rollup that the target index should be an alias, then the alias can be created at Rollup initialization time and we can create a template to make sure all backing indices always have the correct index mappings. Then a user can add a rollover policy for this alias and the work is done.
Add support for an alias by having the user create an alias, create an empty backing index with no mappings and set it as the write index, and then assign the alias as the target index in the rollup policy. When we write to the alias on the rollups side, we will check that the index has empty mappings and then update the mappings and make the backing index a rollup index. When the alias points to a new index we would do the same, so it would be important that the user sets up the alias without mappings.

@setiah thoughts?

@setiah any thoughts on this?
#1 solution makes the most sense to me. It is the most user friendly. We would add another boolean field in Rollup struct, let's say "is_target_index_alias". This would require dashboards work too to include this field in rollup wizard/create page.

@petardz, can rollup target_index be mapped to a index alias and can we have a ism policy for that target_index? if we use this method then it would be more powerful and dynamic. i tried to do this.

create a index template for log*
create a ism policy for log* that has roll_over, warm_migration and delete stages.
add docs to log index that would create log-000001 index and ism policy will be applied and it performs roll_over based on conditions.
create a index template for rollup-log*
create a different ism policy for rollup-log* that can any stage independent of log* ism
create a continuous rollup job with source index as log* and target_index as "rollup-log". rollup-log is a alias that should point to the current write index for rollup-log* index pattern.

when i tried to do this in aws opensearch v7.9 the rollup job fails with this error
"Failed: Failed to create target index [rollup-debug]"
maybe this idea could be a independent issue and not related to this. but i stumpled on this github issue while trying to troubleshoot the above error and felt this would be cleaner approach.

@JathinSanghvi We don't have currently alias capability in target_index, but your example can be setup through "scripted target_index". Checkout this comment with example: link
Regarding alias, if we support it in target_index it would be very similar flow to "scripted target_index" variant:

Setup source index template and rollover policy
Create continuous rollup job with alias in target_index and target_index_is_alias:true param:

PUT _plugins/_rollup/jobs/example
{
  "rollup": {
    "source_index": "log*",
    "target_index": "log_rollup",
    "target_index_is_alias":true
    "continuous": true,
    ...
    "dimensions": [
       ...
    ],
    "metrics": [
       ...
    ]
  }
}

During creation of the job, if alias doesn't exists, we would create backing index and alias.

Setup rollover policy for log_rollup alias

Difference between alias variant from "scripted" variant is that with alias you would be able to control separately when rollup index rolls over. In scripted variant rollup index is tied to current source_index and when source_index rolls over then new rollup index is created.
Regarding searching, both variants would work the same.

i feel there wont be additional work required for new variant, just the support for alias on target index should be enough. So then who ever is picking up the alias work should leave it to be flexible so that both variants can exist side by side. what do you think?

authors: JathinSanghvi aditjind dbbaughe downsrob petardz setiah

repo: opensearch-project/index-management


text:
[FEATURE] Add image embedding model type

Is your feature request related to a problem?
Right now it is not possible to register image embedding ML models in an OS cluster
What solution would you like?
Users should be able to register image embedding ML models. We can achieve this by adding an ImageEmbedding function name with it respective DJL translator and model
We will also need to add an image embedding ingest processor so that it feeds the docs to the model when doc ingest happens
What alternatives have you considered?
It is possible to register image embedding models through remote models but it would be great to have models working inside the OS cluster

[Catch All Triage - 1, 2, 3, 4]

authors: IanMenendez dblock

repo: opensearch-project/ml-commons


text:
Refactored HybridQueryPhaseSearcherTests to remove knn specific classes

Description
Refactored HybridQueryPhaseSearcherTests to remove knn specific classes
Related Issues
Resolves #https://github.com/opensearch-project/neural-search/issues/859
Check List

[ ] New functionality includes testing.
[ ] New functionality has been documented.
[ ] API changes companion pull request created.
[x] Commits are signed per the DCO using --signoff.
[ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Code looks good to me, thanks for contribution @owaiskazi19
I do see other tests where we do have dependency on the same knn classes: https://github.com/opensearch-project/neural-search/blob/main/src/test/java/org/opensearch/neuralsearch/query/HybridQueryTests.java and https://github.com/opensearch-project/neural-search/blob/main/src/test/java/org/opensearch/neuralsearch/query/HybridQueryBuilderTests.java.
Did you look into those tests as well, can we do similar refactoring?

Did you look into those tests as well, can we do similar refactoring?

Handled HybridQueryTests.  HybridQueryBuilderTests.java is tightly coupled with knn and might need another issue of it's own.

Did you look into those tests as well, can we do similar refactoring?

Handled HybridQueryTests. HybridQueryBuilderTests.java is tightly coupled with knn and might need another issue of it's own.

@owaiskazi19  could you please create a tracking issue for this if it doesn't exist? Thanks

@owaiskazi19 could you please create a tracking issue for this if it doesn't exist? Thanks

https://github.com/opensearch-project/neural-search/issues/979

authors: martin-gaievski minalsha owaiskazi19

repo: opensearch-project/neural-search


text:
Added K-NN APIs get_model, search_model, delete_model, train_model

Description

Added K-NN APIs get_model, search_model, delete_model, train_model

Tested the api with different path and query params here.

k-NN doc https://opensearch.org/docs/latest/search-plugins/knn/api

plugin code ref: https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/plugin/rest

Issues Resolved
Related to https://github.com/opensearch-project/opensearch-api-specification/issues/168,
https://github.com/opensearch-project/opensearch-py/issues/300
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

There's a Smithy Build error: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/8011904023/job/21886085420?pr=192https://github.com/opensearch-project/opensearch-api-specification/actions/runs/8011904023/job/21886085420?pr=192

There's a Smithy Build error: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/8011904023/job/21886085420?pr=192https://github.com/opensearch-project/opensearch-api-specification/actions/runs/8011904023/job/21886085420?pr=192

There's a Smithy Build error: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/8011904023/job/21886085420?pr=192https://github.com/opensearch-project/opensearch-api-specification/actions/runs/8011904023/job/21886085420?pr=192

Sorry, Its passing now.

@Xtansia, @nhtruong Please take a look and let me know if any changes needed.

@saimedhi Try adding the same APIs to the OpenAPI feature branch so we can compare? https://github.com/opensearch-project/opensearch-api-specification/tree/feature/native_openapi

authors: dblock nhtruong saimedhi


text:
Update the ref for opensearch repo

Signed-off-by: Zelin Hao zelinhao@amazon.com
Description
CI fails right now because OpenSearch has bumped to 2.6 for 2.x branch.
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #3059 (24c2639) into main (af1be01) will decrease coverage by 0.30%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #3059      +/-   ##
==========================================
- Coverage   93.17%   92.87%   -0.31%
==========================================
  Files         167      167
  Lines        4602     4602
==========================================
- Hits         4288     4274      -14
- Misses        314      328      +14

Impacted Files
Coverage Δ

src/system/temporary_directory.py
64.44% <0.00%> (-28.89%)
:arrow_down:

src/system/os.py
94.73% <0.00%> (-5.27%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Seems like this PR and #3048 are in a chicken eggs problems and need to be fixed same time for the CI to pass. I will close this one and migrate the change from this PR to that one.

authors: codecov-commenter zelinh

repo: opensearch-project/opensearch-build


text:
[BUG] Cannot create coordinating nodes

What is the bug?
I get the following error when assigning no role to a node.
Error: UPGRADE FAILED: cannot patch "opensearch-cluster" with kind OpenSearchCluster: OpenSearchCluster.opensearch.opster.io "opensearch-cluster" is invalid: spec.nodePools[1].roles: Required value

But this is a valid value for an OpenSearch cluster for a node to act as a client/coordinating node.
How can one reproduce the bug?
Helm configuration:
opensearchCluster:
  enabled: true
  general:
    httpPort: "9200"
    version: 2.11.0
    serviceName: "opensearch-cluster"
    drainDataNodes: true
    setVMMaxMapCount: true
  nodePools:
    - component: masters
      diskSize: "20Gi"
      replicas: 3
      roles:
        - "master"
        - "data"
      resources:
        requests:
          memory: "4Gi"
          cpu: "2000m"
        limits:
          memory: "4Gi"
          cpu: "2000m"
    - component: clients
      diskSize: "20Gi"
      replicas: 3
      roles: []
      resources:
        requests:
          memory: "4Gi"
          cpu: "2000m"
        limits:
          memory: "4Gi"
          cpu: "2000m"
  security:
    tls:
      transport:
        generate: true
      http:
        generate: true

What is the expected behavior?
Set an empty array of roles for the node when setting roles: [].
What is your host/environment?
Ubuntu 22.04 with Kubernetes 1.28.2.

Thank you for this snippet @welkson
It might be useful to me for another try later. I've moved to another project to solve my issue. I close the issue because it looks like a good workaround.

I did this:
    - component: coordinating
      replicas: 2
      diskSize: "54Gi"
      jvm: -Xmx1024M -Xms1024M
      pdb:
        enable: true
        minAvailable: 1
      nodeSelector:
      resources:
          requests:
            memory: "2Gi"
            cpu: "2"
          limits:
            memory: "6Gi"
            cpu: "4"
      roles:
        - ""
      additionalConfig:
        node.master: "false"
        node.data: "false"
        node.ingest: "false"
        node.remote_cluster_client: "false"
      persistence:
        pvc:
          accessModes:
          - ReadWriteOnce
          storageClass: longhorn

cc @flavienbwk

Your genius. thanx a lot


text:
Auto-increment request IDs.

Description
Auto-increment request IDs.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

LGTM, is this robust to multithreading (Atomic)?

It looks like it is: https://stackoverflow.com/questions/35088139/how-to-make-a-thread-safe-global-counter-in-python

repo: opensearch-project/opensearch-sdk-py


text:
Fluent Bit 1.9 Announcement

Describe the blog post your would like to write
An update that Fluent Bit 1.9 was released and a short Getting Started Guide
What is the title of the blog post?
Getting started with Fluent Bit and OpenSearch
Who are the authors?
Anurag Gupta
What is the proposed posting date?
3/24/2022

The blog post was launched.  Closing.

repo: opensearch-project/project-website


text:
Proof reading Upgrade from Open Distro to OpenSearch.

Signed-off-by: dblock dblock@amazon.com
Description
Blog post is great! Some proof-reading.
By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

@stockholmux it looks like your requested changes were addressed. Pls give approval if this looks good to you now.

authors: dblock elfisher


text:
Create 2021-12-21-oracle-announcement.markdown

Description
Create 2021-12-21-oracle-announcement.markdown
Signed-off-by: Andrew Hopp andhopp@amazon.com
Issues Resolved
n/a
Check List

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Previous PR: https://github.com/opensearch-project/project-website/pull/597


text:
Trigger cd workflow for all tags

opensearch-security pull request intake form
Please provide as much details as possible to get feedback/acceptance on your PR quickly

Category: (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation)
Maintenance

Github Issue # or road-map entry, if available:

Description of changes:

Trigger CD workflow for all tags pushed.
Check out OpenSearch after cache.

Why these changes are required?

OpenSearch community follows new format of tags(not starting with v) in order to distinguish from ODFE tags.
Be consistent with CI workflow and fix CD workflow

What is the old behavior before changes and new behavior after changes? (Please add any example/logs/screen-shot if available)

Testing done: (Please provide details of testing done: Unit testing, integration testing and manual testing)
CD run

TO-DOs, if any: (Please describe pending items and provide Github issues# for each of them)

Is it backport from main branch? (If yes, please add backport PR # and commits #)

By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

-1: I don't see a need to trigger CD on all tags.

repo: opensearch-project/security


text:
Remove unnecessary scripts after repo split

Signed-off-by: Joshua Li joshuali925@gmail.com
Description
Remove unnecessary scripts after repo split
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

But repo is not split yet...

But repo is not split yet...

we no longer maintain workbench in this repo for 2.5+, it's in https://github.com/opensearch-project/dashboards-query-workbench/tree/2.x

repo: opensearch-project/sql


text:
[Spotless] Applying Google Code Format for entire project #22

Description
Applies spotless on entire project with **/*.java
Removes checkstyles from project
Issues Resolved
https://github.com/opensearch-project/sql/issues/1101
Check List

[x] New functionality includes testing.

[x] All tests pass, including unit test, integration test and doctest

[x] New functionality has been documented.

[x] New functionality has javadoc added
[x] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

removing backport-failed since there's a PR merged for 2.x