text:
Fixes extent calculation in WMTS optionsFromCapabilities when BoundingBox exists

As mentioned  in #11350 when creating the extent, the WMTS optionsFromCapabilities dose not consider the Wgs84BoundingBox. This pull request attempts to fix this by using the Wgs84BoundingBox extent in cases where it exists.

@amir-ba A minimal fix would look something like this:
diff --git a/src/ol/format/WMTSCapabilities.js b/src/ol/format/WMTSCapabilities.js
index 9e72eb213e..9dceae171d 100644
--- a/src/ol/format/WMTSCapabilities.js
+++ b/src/ol/format/WMTSCapabilities.js
@@ -103,7 +103,7 @@ const LAYER_PARSERS = makeStructureNS(
   makeStructureNS(OWS_NAMESPACE_URIS, {
     'Title': makeObjectPropertySetter(readString),
     'Abstract': makeObjectPropertySetter(readString),
-    'WGS84BoundingBox': makeObjectPropertySetter(readWgs84BoundingBox),
+    'WGS84BoundingBox': makeObjectPropertySetter(readBoundingBox),
     'Identifier': makeObjectPropertySetter(readString),
   })
 );
@@ -196,6 +196,7 @@ const TMS_PARSERS = makeStructureNS(
   makeStructureNS(OWS_NAMESPACE_URIS, {
     'SupportedCRS': makeObjectPropertySetter(readString),
     'Identifier': makeObjectPropertySetter(readString),
+    'BoundingBox': makeObjectPropertySetter(readBoundingBox),
   })
 );

@@ -304,9 +305,9 @@ function readResourceUrl(node, objectStack) {
 /**
  * @param {Element} node Node.
  * @param {Array<*>} objectStack Object stack.
- * @return {Object|undefined} WGS84 BBox object.
+ * @return {Object|undefined} BBox object.
  */
-function readWgs84BoundingBox(node, objectStack) {
+function readBoundingBox(node, objectStack) {
   const coordinates = pushParseAndPop(
     [],
     WGS84_BBOX_READERS,
diff --git a/src/ol/source/WMTS.js b/src/ol/source/WMTS.js
index b61af5eee2..b618a1edbc 100644
--- a/src/ol/source/WMTS.js
+++ b/src/ol/source/WMTS.js
@@ -6,6 +6,7 @@ import TileImage from './TileImage.js';
 import WMTSRequestEncoding from './WMTSRequestEncoding.js';
 import {appendParams} from '../uri.js';
 import {assign} from '../obj.js';
+import {containsExtent} from '../extent.js';
 import {createFromCapabilitiesMatrixSet} from '../tilegrid/WMTS.js';
 import {createFromTileUrlFunctions, expandUrl} from '../tileurlfunction.js';
 import {equivalent, get as getProjection} from '../proj.js';
@@ -479,7 +480,9 @@ export function optionsFromCapabilities(wmtsCap, config) {
   const tileSpanX = matrix.TileWidth * resolution;
   const tileSpanY = matrix.TileHeight * resolution;

-  const extent = [
+  const matrixSetExtent = matrixSetObj['BoundingBox'];
+
+  let extent = [
     origin[0] + tileSpanX * selectedMatrixLimit.MinTileCol,
     // add one to get proper bottom/right coordinate
     origin[1] - tileSpanY * (1 + selectedMatrixLimit.MaxTileRow),
@@ -487,6 +490,10 @@ export function optionsFromCapabilities(wmtsCap, config) {
     origin[1] - tileSpanY * selectedMatrixLimit.MinTileRow,
   ];

+  if (!containsExtent(matrixSetExtent, extent)) {
+    extent = matrixSetExtent;
+  }
+
   if (projection.getExtent() === null) {
     projection.setExtent(extent);
   }

I think the wrapX handling you added makes sense, and could be added here too.
Would you be able to modify your pull request like this? If not let me know, then I'll start a new one.

thank you @ahocevar for your suggestion I will fix my pull request shortly.

Thank you, @amir-ba

@ahocevar can you tell why the tests are failing? I cant quite figure out what to fix to make them work.

@amir-ba The failing test are due to a random problem that's unrelated. I restarted all tests. They should all be green in a few minutes.

I can add the tests, but since Its my first time contributing to this repo, I am not clear about the test data structures. should I extent the exiting capabilities_wgs84.xml with a layer that has such configuration and add the tests or do you have a better Idea?

Thanks for your continued effort on this, @amir-ba! You could save a copy of https://www.basemap.at/wmts/1.0.0/WMTSCapabilities.xml in the test/spec/ol/format/wmts/ folder and use that for the tests.

@amir-ba Thanks for your work on this so far! Do you think you'd be able to add the test as discussed above, or do you need additional guidance?

This issue is still unresolved and should not have been closed.

@ahocevar , sorry for the Absence, can I still add my test and push the current develop into this feature branch?

Feel free to continue working on this PR here.

@MoonE can you please restart the failing spec test here.

Thanks, @amir-ba

Thanks for your review🎉🎉

authors: FrankyBoy MoonE ahocevar amir-ba


text:
Rename the GitHub repository to ol2

This pull request should be merged at the same time as openlayers/openlayers.github.io#56. For more information, also see there.

This sounds good to me.  GitHub will set up redirects.  So URLs for issues and git clone etc will still work (as long as we don't use the name openlayers for anything else).

source: gharchive/pull-request

authors: ahocevar tschaub

repo: openlayers/openlayers

license_type: permissive

license_source: github-api


text:
ReShare reports - update report titles output on various report pages

Hi @scwiek,
As promised in our call yesterday, attached here is a list of the title, label changes proposed for the ReShare reports to make them read a bit more user friendly. Please let me know if any of this work is something that Ian Hardy could do independent of you.
ReShare Reports Title_Label Updates.odt
Please let me know if you have any questions.
Debra

Hey @debradenault, I think all of this is controlled within the reports web UI, so I think this might be an Ian thing.

source: gharchive/issue

authors: debradenault scwiek


text:
Web templates config 5.4.0

See https://trello.com/c/A8AjAxmj/122-fix-omerowebindextemplate-config-docs

Read doc changes and see if they make sense.
Try setting bin/omero config set omero.web.index_template ...  and see if it works (do the docs explain enough what this does)?
Try setting login_redirect (fixed in https://github.com/openmicroscopy/openmicroscopy/pull/5485) and again see if docs explain what this does clearly enough.
Try setting base_include_template from PR: https://github.com/openmicroscopy/openmicroscopy/pull/5463

Looks good. Tried all the steps mentioned in the docs, works fine. 👍

authors: dominikl will-moore

repo: openmicroscopy/ome-documentation


text:
Install the latest dev release feature is broken on new OSes.

Test on any recent system.

This is fixed for version 4.0.0 with new SSL.

Ibrahim confirmed fixed.

authors: iabdalkader kwagyeman

repo: openmv/openmv-ide

license: mit

license_source: bigquery


text:
The Monitor Class for new poller service is N/A and the service is always marked as running

I added a poller service named MySQL3307 to the poller-configuration.xml file and using the TCPMonitor for it then run the event reloadDaemonConfig for Pollerd service. But after that, in the Web UI, the Monitor Class is N/A, and this service is always marked as running.
I defined the MySQL3307 in the configuration file as below:
<service name="MySQL3307" interval="300000" user-defined="true" status="on">
         <parameter key="retry" value="1"/>
         <parameter key="timeout" value="3000"/>
         <parameter key="port" value="3307"/>
         <parameter key="banner" value="*"/>
      </service>
<monitor service="MySQL3307" class-name="org.opennms.netmgt.poller.monitors.TcpMonitor"/>

Here is a screenshot when I open the Web UI.

I also tried to modify some parameters and the class-name of the service MySQL (from the TCPMonitor to JDBCMonitor) and send the event reloadDaemonConfig. When I check on the Web UI, the parameter is updated but the Monitor Class of MySQL is still TCP.
How can I fix it? Or if there is any document or link related to this issue, please show me.
I installed OpenNMS using docker with the latest version currently. Just clone the repository and run the command docker-compose up -d.
Thank you

Hello @namnhatdoan this topic is not related to the docker image itself and seems to be a problem with the application itself. Would you be so kind, moving it to our Community Support where the peoples live answering this type of questions?

I'll close this issue, further discussions on this topic can be followed up in this topic here

I'll close this issue, further discussions on this topic can be followed up in this topic here.

repo: opennms-forge/docker-horizon-core-web

license: MIT


text:
Fix hvg subsetting bug

Changelog
This line in scvi integration component is supposed to subset highly-variable genes:
adata = adata[:,adata.var['var_input']].copy()
However, this will not happen for the following reasons:

Instead of "var_input" par["var_input"] should be used
Even with proper key, adata.var[key] will return a column from adata.var and not the list of selected genes as expected

This PR fixes the issue
Checklist before requesting a review

[x] I have performed a self-review of my code

[ ] Conforms to the Contributor's guide

Check the correct box. Does this PR contain:

[ ] Breaking changes
[ ] New functionality
[ ] Major changes
[ ] Minor changes
[x] Bug fixes

[ ] Proposed changes are described in the CHANGELOG.md

[ ] CI tests succeed!

@VladimirShitov Good catch! Would you be able to implement tests?

Added test and fixed small comments. The function for HVG subsetting is now moved to a separate utils script, as it is used in other components as well. The problem is that the test pipeline fails because there is "No space left on device", so I'm not 100% sure that importing utils works correctly

@VladimirShitov The disk issue seemed like a temporary one (?). I am now getting this:

Yep, this is a problem from my side now 😅 Fixing

Fixed it :) @DriesSchaumont , can you check?

@VladimirShitov Could you adress the comment from Robrecht?

Done, please check :)

LGTM! @rcannood If you would like to have a look

authors: DriesSchaumont VladimirShitov

repo: openpipelines-bio/openpipeline


text:
RuntimeError of Shape op during Calibration dataset progress and finetune progress

配置信息：
TARGET_PLATFORM   = TargetPlatform.NXP_INT8          # choose your target platform
MODEL_TYPE        = NetworkFramework.ONNX                 # or NetworkFramework.CAFFE
INPUT_LAYOUT          = 'chw'                             # input data layout, chw or hwc
NETWORK_INPUTSHAPE    = [16, 1, 40, 61]                  # input shape of your network
CALIBRATION_BATCHSIZE = 16                                # batchsize of calibration dataset
EXECUTING_DEVICE      = 'cuda'                            # 'cuda' or 'cpu'.
REQUIRE_ANALYSE       = True
DUMP_RESULT           = False
SETTING = UnbelievableUserFriendlyQuantizationSetting(
platform = TARGET_PLATFORM, finetune_steps = 2500,
finetune_lr = 1e-3, calibration = 'percentile',
equalization = True, non_quantable_op = None)
dataloader = DataLoader(
dataset=calibration_dataset,
batch_size=32, shuffle=True)
quantized = quantize(
working_directory=WORKING_DIRECTORY, setting=SETTING,
model_type=MODEL_TYPE, executing_device=EXECUTING_DEVICE,
input_shape=NETWORK_INPUTSHAPE, target_platform=TARGET_PLATFORM,
dataloader=dataloader, calib_steps=250)
问题描述：
在213次迭代时shape算子报上述错误，计算后发现这一次迭代batch size=19, 在dataload迭代器内部打印了下log，发现这一批次finetune确实只送出来了19个样本。后来发现数据集样本数刚好在213次迭代时遍历完一遍。
后面我将finetune step和calib_step都改为100， Calibration数据集样本数调整为32*100个之后就能正常运行。
下面是模型文件：
model.zip

你现在导出的reshape2算子上的shape是写死的，必须要batchsize=16，在这种情况下你不能送入其他batchsize。
你是如何导出的该模型？torch导出好像不会有这样的问题

我是通过torch.onnx转的：
input1 = torch.randn(16, 1, 40, 61).cuda()
input_names = [ "input"]
output_names = [ "output" ]
torch.onnx.export(net, input1, model_path, verbose=True, input_names=input_names, output_names=output_names)

这里确实指定了input tensor，请问应该如何导出不指定batchsize的模型呢？

你可以参考这个 https://blog.csdn.net/ChuiGeDaQiQiu/article/details/119065818
不过如果你是要做推理加速，我建议你导出固定尺寸的onnx模型。
你需要注意，对于不同shape的tensor，推理时选用的算法不会一样，只有比较方的shape在计算gemm和conv时会达到最高执行效率。
比如 shape: [1,3,224,224]就不太方，算的就比较慢
比如 shape: [96, 96, 48, 48]，就是个方形的 tensor，计算效率就高得多。

非常感谢，按照您的提示我的代码能正确运行了，那我是否可以这么理解：
PPQ接收的量化模型需要再训练的话，最好导出batchsize为dynamic 的onnx模型做finetune,
否则需要保证calibration dataset的数据数目是所固定的batchsize的整数倍？

是的，我们无法得知你的onnx模型里面那一维是batchsize，或者说那一维是可变的，对于你导出的reshape而言，它要求的输入尺寸是固定的，我们只能给你报错，因为你的模型有语义上的错误。
不过更希望你对推理部署有更深入的了解，在pytorch中我们可以随意改变batchsize，这样你的模型一直都可以跑。但是对于推理而言，batchsize=1的模型与batchsize=128的模型是不一样的，会触发不同的部署优化过程。

明白了，非常感谢！

authors: ZhangZhiPku lycfly

repo: openppl-public/ppq


text:
Set the TTL for NS delegation of subdomains on openregister.org

I think the right thing was trying to be achieved here (ie trying to
keep the NS delegation for {test, discovery, ...}.openregister.org
short so it can be changed easily). However, the TTL here is for the
NS record in the wrong zone.
NS records at the point-of-delegation are non-authoritative. It is the
TTL of the NS record in the authoritative zone that is respected:

Defined in RFC 1035. NS RRs appear in two places. Within the zone file,
in which case they are authoritative records for the zone's name
servers. At the point of delegation for either a subdomain of the zone
or in the zone's parent in which case they are non-authoritative. Thus,
the zone example.com's parent zone (.com) will contain
non-authoritative NS RRs for the zone example.com at its point of
delegation (point of delegation is the term frequently used to describe
the NS RRs in the parent that delegate a zone of subdomain) and
subdomain.example.com will have non-authoritative NS RRS in the zone
example.com at its point of delegation. NS RRs at a point of delegation
are never authoritative only NS RRs within the zone are regarded as
authoritative. While this may look a fairly trivial point, is has
important implications for DNSSEC.

Source: http://www.zytrax.com/books/dns/ch8/ns.html
Deploying
The current plan for this in the test environment is:
-/+ module.core.aws_route53_record.zone_delegation
    fqdn:               "test.openregister.org" => "<computed>"
    name:               "test.openregister.org" => "test.openregister.org"
    records.#:          "4" => "4"
    records.1217762515: "ns-607.awsdns-11.net" => "ns-607.awsdns-11.net"
    records.1440870510: "ns-2041.awsdns-63.co.uk" => "ns-2041.awsdns-63.co.uk"
    records.3777510865: "ns-1310.awsdns-35.org" => "ns-1310.awsdns-35.org"
    records.4230963353: "ns-201.awsdns-25.com" => "ns-201.awsdns-25.com"
    ttl:                "300" => "300"
    type:               "NS" => "NS"
    zone_id:            "<parent-zone>" => "<delegated-zone>" (forces new resource)

I think we'll need to be careful about applying this change because it appears as though it will try to delete NS records in the parent zone in R53 which should(?) cause an error. It might be sensible to first remove the resource from Terraform's state file (with terraform state rm) before applying.
The TTL on the NS records at the point-of-delegation should probably be reset to their defaults too.

Please don't merge until #252 has been pushed all the way to beta. This should be on Friday 3 Feb.

This is ready to merge.

👍

repo: openregister/deployment


text:
Add direct links to recipes

Previously, we linked to the top-level Git repository from recipes. Now, we should link directly to the file that contains the recipe.
This handles both imperative and declarative recipes as well as recipes from nested folders in rewrite.

Now that I've posted this, I realize that I could optimize this a bit by making it so things aren't quite as hard-coded. I could use just the list coded as the core stuff and build up links like that. I'll make that change later today.

nice!


text:
Update 51-ganglia-ce-dashboards.conf to configure reset metrics

Enable reset metrics functionality in the condor_gangliad, which first appears in HTCSS version 23.6.x.  This functionality is critical to keeping the CE Dashboard data correct. Note the ospool-container container should ideally not be rebooted until it such time as the image will contain HTCSS 23.6.x.

LGTM. @brianhlin please review/merge.

Go for it. Thanks!
On Mon, Apr 8, 2024 at 8:49 AM Brian Lin @.***> wrote:

@.**** approved this pull request.
Sure. Let me know when it's safe to merge
—
Reply to this email directly, view it on GitHub
https://urldefense.us/v2/url?u=https-3A__github.com_opensciencegrid_images_pull_177-23pullrequestreview-2D1986832410&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=3Rv5mAmzpJXJ0sHJErwJog&m=Fx7wM4Ff0BAbrUkMkGVXVa9IgbrGZcXZ3s65bv3KC1K8VSxqsdjen7tF71EVsdQM&s=B6GWU6aeU7EdyxruM81vz9Xbr0irtkxCgvQQgPSj-M8&e=,
or unsubscribe
https://urldefense.us/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAIEDDBI4SNRZMYJGQI7IXDY4K36RAVCNFSM6AAAAABF42ISQKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTSOBWHAZTENBRGA&d=DwMFaQ&c=qzHnJIRvjI6L-clJH8JwLQvf_Iq43fzikf6aoxZgMb8&r=3Rv5mAmzpJXJ0sHJErwJog&m=Fx7wM4Ff0BAbrUkMkGVXVa9IgbrGZcXZ3s65bv3KC1K8VSxqsdjen7tF71EVsdQM&s=qQSAX2ebUuBtnbvni04MahWgIocIlJchselmgFWHuIY&e=
.
You are receiving this because you were assigned.Message ID:
@.***>

authors: rynge tannenba

repo: opensciencegrid/images


text:
Restriction of pandas < 2 presents bottleneck

Since scmdata currently restricts pandas to <2 it is no longer compatible with the latest version of pyam-iamc.
This is a problem since for Scenario Explorer processing I would require both.
Is updating to pandas 2 currently on the radar, if so is there a time estimate?
Thanks :)
UPDATE: I just saw that there is already an automatically created PR #235 seems like some things are failing for now.

Yes it is on the radar, but I don't have an ETA and I won't get to it in the next few weeks if it requires any real time. Maybe @znicholls can find the time as we do need to get it done. Happy
IIRC most of the failures were relatively minor changes related to somewhat "cosmetic" changes, but I'm a little worried that it will cause a big mess with lots of if pandas_less_than_2: blocks sprinkled around the tests.
Was the pyam migration relatively painless?

Hmm just rebased #235 and only one test is failing now. Maybe they back-pedalled some things as we didn't change anything on our end

The failing test is also in our (thin) pyam compatibility layer https://github.com/openscm/scmdata/blob/master/src/scmdata/pyam_compat.py. @phackstock Does an IamDataFrame support dates later than 2300? If so we can likely remove this code and the associated tests which would make supporting pandas==2 easy

I can have a look before you @lewisjared. Thanks for rebasing, that's a great start.

Thanks a lot @lewisjared and @znicholls for taking a look so quickly.

@phackstock Does an IamDataFrame support dates later than 2300?

I haven't encountered yet since all of the modeling goes until the end of the century but in principle I don't see why pyam wouldn't support that.

I think if you had datetimes that went beyond 2267 you'd hit the same issue that we hit (and might need our long data frame fix). However, probably a while before you hit that use case.

#235 is now just waiting on a review then we can merge and release

@phackstock 0.15.3 should have the restriction removed

Sweet, thanks a lot

authors: lewisjared phackstock znicholls

repo: openscm/scmdata


text:
Database

Pull request
Please confirm that this pull request has done the following:

[x] Tests added
[x] Documentation added (where applicable)
[x] Example added (either to an existing notebook or as a new notebook, where applicable)
[x] Description in CHANGELOG.rst added

Closes #103

If you're happy, merge

authors: lewisjared znicholls


text:
Support Put, Get acl for bucket and object

Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature
Anything else we need to know?:
The PR's requirement is from the link https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html. We only need to support Canned ACL.
The most common cases for ACL  are used as the following cases:

Object whose ACL is "private" is not allowed to access by other user.
Object whose ACL is "public-read" is allowed to read access by anyone.

The Canced ACL include "private", "public-read", "public-read-write", "authenticated-read", "bucket-owner-read", "bucket-owner-full-control", "aws-exec-read".
Environment:

Gelato(release/branch) version:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

the feature have been supported

repo: opensds/multi-cloud


text:
Modified the ceph driver to adapt the southbound changes

Due to the OpenSDS southbound  has changed, so ceph driver need to modify .

@xxwjj It looks good, but IMO I prefer using GetId() rather than Id when fetching a filed in structure:
name := opt.Name
size := opt.Size

Because usually we use opt.Name to change Name field value of opt, what do you think?

@leonwanghui ok, I will change it.  but I am afraid this usage design is too complex.

I think it's a good way to prepare for designing some read-only field, because in this way we just need to change the first letter of field from upper to lower, rather than modifying too much code.

authors: leonwanghui xxwjj

repo: opensds/opensds


text:
navigate a predefined path

Is it possible to configure the viewer so that it moves along a predefined path that takes a user to specific points of interest? My use case is one where I would like to override the mouse-scroll to move along this predefined path?

One way is to use the OpenSeadragonViewerInputHook plugin. Then you can override a viewer's default handling of touch, mouse (and possibly keyboard) input, something like:
    var viewer = OpenSeadragon({...});

    viewer.addViewerInputHook({hooks: [
        {tracker: 'viewer', handler: 'dragHandler', hookHandler: onViewerDrag},
        {tracker: 'viewer', handler: 'dragEndHandler', hookHandler: onViewerDragEnd}
    ]});

    function onViewerDrag(event) {
        // Do your own custom drag handling here...

        event.preventDefaultAction = true;
    }

    function onViewerDragEnd(event) {
        event.preventDefaultAction = true;
    }

There's no mechanism in OpenSeadragon to define such a path, but if you have a path, you can move the viewport wherever you want via panTo(point, true) and zoomTo(zoom, null, true).

Thanks. I think panTo might do the trick.

That's how I did it, with jquery json polling.
fitBounds might be an option too.

authors: boskar iangilman msalsbery sharmaashish

repo: openseadragon/openseadragon


text:
imageToViewportRectangle data from db

I get a rect using selection plugin :
selection =viewer.selection({
onSelection: function(rect) {
console.log(rect.x);
console.log(rect.y);
console.log(rect.width);
console.log(rect.height);
var elt = document.createElement("div");
elt.id = "runtime-overlay";
elt.className = "highlight";
viewer.addOverlay({
element: elt,
location: viewer.viewport.imageToViewportRectangle(rect)
});
    input = "rect_x="+rect.x+"&rect_y="+rect.y+"&rect_w="+rect.width+"&rect_h="+rect.height+"&plink=<?=$plink?>&case_id=<?=$case_id?>&mode=add";
    $.ajax({
        url : "handlers/H_AnnotationHandler.php",
        data : input,
        type : "post",
        dataType : "json",
        success : function(response) {
            if (!response.error)
                alert("success");
            else
                alert("failed");
        }
    });
}

});
And i store the rect in mysql via H_AnnotationHandler.php. After that, when i try to use this data like that :
$.ajax({
url : "handlers/H_AnnotationHandler.php",
data : "case_id=&plink=&mode=get",
type : "post",
dataType : "json",
success : function (response) {
if (!response.error) {
for (var i = 0; i < response.annots.length; i++) {
var elt  = document.createElement("div");
elt.id = "runtime-overlay" + i;
elt.className = "highlight";
viewer.addOverlay({
element: elt,
location : viewer.viewport.imageToViewportRectangle(parseInt(response.annots[i].rect_x), parseInt(response.annots[i].rect_y), parseInt(response.annots[i].rect_w), parseInt(response.annots[i].rect_h))
});
}
}
}
});
it create a runtime-overlay division but left, top, height and width seems very wrong. (27094e+04 etc.)
note : i am sure, sql data is true. i compare it before take.

See my comment there: http://stackoverflow.com/questions/34744842/openseadragon-imagetoviewport

@iangilman This can be closed (resolved on SO).

authors: avandecreme ozgunlu


text:
getZoom for fabricjs overlay and zoom synchronization

Hi folks!
I am struggling with this (http://www.c-ev.com/osdragon/dragonfabric1.html) test of mine. What I am trying to do is to sync fabricjs canvas with openseadragon. If you scroll with a mouse you will see how both get zoomed in/out.

When I open browser console and run viewer.viewport.getZoom(), it returns 0.0013750000000000001 when the image is fully zoomed in and I am not understanding that number. In this example (http://msalsbery.github.io/openseadragonimaginghelper/index.html) it shows 15 when fully zoomed in. The reason why I was looking at the getZoom, is because I was hoping to sync both canvas.
Somewhere I read that it is possible to change some z value in order to make one or another element active. Under what class I should look for.
Any suggestions of how to achieve this would be highly + highly appreciated, otherwise it looks like I have to big of a bite.

Kindly

When I open browser console and run viewer.viewport.getZoom(), it returns 0.0013750000000000001 when the image is fully zoomed in and I am not understanding that number. In this example (http://msalsbery.github.io/openseadragonimaginghelper/index.html) it shows 15 when fully zoomed in. The reason why I was looking at the getZoom, is because I was hoping to sync both canvas.

The value is a viewport zoom. To convert to image zoom, see http://openseadragon.github.io/examples/viewport-coordinates/

Somewhere I read that it is possible to change some z value in order to make one or another element active. Under what class I should look for.

Are you refering to the z-index CSS property? It does not make an element active but defines which element should be on top.

Thank you avandecreme! You posted on issue #844 and that looks like is what I was trying to do my self. I'll research! Thank you sooo much!

@Kampii Are you using https://github.com/altert/OpenseadragonFabricjsOverlay by @altert? If not, it might be helpful...

@iangilman Ill test this https://github.com/altert/OpenseadragonFabricjsOverlay as it allows to select the element. Thanks.

authors: Kampii avandecreme iangilman


text:
[Searchable Snapshot] Cache path not deleted after index is deleted

Describe the bug
A path in the file system is created to store the cached blocks for a searchable snapshot index, such as:
data-2.x/nodes/0/cache/0/8hJW-teFRruFwKyEB4-_aA/0/RemoteLocalStore

When the index with ID 8hJW-teFRruFwKyEB4-_aA is deleted I expect the data-2.x/nodes/0/cache/0/8hJW-teFRruFwKyEB4-_aA path to be removed, but it is not. All the contents of that directory are deleted, but the directory itself is leaked.
To Reproduce
Create a searchable snapshot index, then delete it.
Expected behavior
Assuming there are no searchable snapshot indexes in my cluster, I expect the data-2.x/nodes/0/cache/0 directory to be empty.

@andrross Please assign this issue to me. I have started working on this issue.

@harshjain2 I have assigned the issue to you.

@kotwanikunal Thanks for assigning this issue. I have started working on this issue.

@andrross @kotwanikunal I am able to find the code fix. For testing the issue, I am not able to repro the issue. I tried the following steps:

https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/searchable_snapshot/ Followed this configuration.
https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/#shared-file-system
./gradlew run

After that I tried to test the OpenSearch service on local machine.
Request :
POST localhost:9200/_snapshot/my-fs-repository/
{
  "type": "fs",
  "settings": {
    "location": "/Users/harshjai/Desktop/opensearch/snapshot/snapshot2"
  }
}

Response :
{
	"error": {
		"root_cause": [
			{
				"type": "repository_exception",
				"reason": "[my-fs-repository] location [/Users/harshjai/Desktop/opensearch/snapshot/snapshot2] doesn't match any of the locations specified by path.repo"
			}
		],
		"type": "repository_exception",
		"reason": "[my-fs-repository] failed to create repository",
		"caused_by": {
			"type": "repository_exception",
			"reason": "[my-fs-repository] location [/Users/harshjai/Desktop/opensearch/snapshot/snapshot2] doesn't match any of the locations specified by path.repo"
		}
	},
	"status": 500
}

I have tried everything I can do, but find a solution to this issue. Can you please help me out with this issue.

@harshjain2 Can you try passing in the repo link as follows -
./gradlew run -Dtests.opensearch.path.repo=/Users/harshjai/Desktop/opensearch/snapshot/snapshot2

@anasalkouz I have the fix ready, but I couldn't test it out. I can attempt fixing it by Sunday, 25th March. In case someone else is interested in picking this up, feel free to re-assign to the interested ones.

@harshjain2 Thanks for testing it out. Can you check if the index is in a green state i.e the shards are assigned?
You can do that by using the following API - GET http://localhost:9200/_cat/shards
I think you might be missing the search role on the node that you are testing.
If you are testing locally, I would suggest using integration tests because of the required config by the nodes.
This class hosts the tests for searchable snapshots and you can extend this function to run your tests - https://github.com/opensearch-project/OpenSearch/blob/879ade758668c81d29476f38afb022dd338a1048/server/src/internalClusterTest/java/org/opensearch/snapshots/SearchableSnapshotIT.java#L544-L580
In the meanwhile, feel free to create a draft PR and we can work this out on the PR.

authors: andrross harshjain2 kotwanikunal

repo: opensearch-project/OpenSearch


text:
Add release notes for 1.3.14

Description
Add release notes for 1.3.14 up to commit 21940d8239b50285ef7f98a1762ef281a5b1c7ee
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[ ] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Not a code change - merging before checks


text:
Support dynamic node role

Signed-off-by: Yaliang Wu ylwu@amazon.com
Description
Currently OpenSearch only supports several built-in nodes like data node
role. If specify unknown node role, OpenSearch node will fail to start.
This limit how to extend OpenSearch to support specific tasks. For
example, user may prefer to run ML tasks on some dedicated node which
doesn't serve as any built-in node roles. So the ML tasks won't impact
OpenSearch core function. This PR removed the limitation. So user can
specify any node role and OpenSearch will start node correctly with that
unknown role. This opens the door for plugin developer to run specific
tasks on dedicated nodes.
Issues Resolved
https://github.com/opensearch-project/OpenSearch/issues/2877
Check List

[x] New functionality includes testing.

[x] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Is Unknown the right name? With this feature there's nothing unknown about it, so can we rename it? I think NamedRole would be closer to what we're trying to do.

:+1: , suggesting the DynamicRole (to complement builtin roles) but name is up to the debate  (NamedRole seems to be general since all roles do have name, imho)

Is Unknown the right name? With this feature there's nothing unknown about it, so can we rename it? I think NamedRole would be closer to what we're trying to do.

👍 , suggesting the DynamicRole (to complement builtin roles) but name is up to the debate (NamedRole seems to be general since all roles do have name, imho)

Thanks, I think DynamicRole or NamedRole is better than UnknownRole, like @dblock said, there will be no unknown role as we are going to support any custom role.
I'm ok for either. Considering the role is complementing builtin roles, maybe DynamicRole more reasonable?

I am good with this! @reta any objections?

LGTM, thanks @ylwu-amzn !

Since you added some lower case node name transform, please also add unit tests for the fact that these are now case-insensitive.

Sure, working on testing now

@reta @dblock Added tests for role name case insensitive change and Github workflow passed. Can you help review?

LGTM!

If no more comments, can anyone help merge this PR? I have no permission to merge.

authors: reta ylwu-amzn


text:
[Backport] [1.3] Update to Netty 4.1.80.Final (#4359)

Backport of https://github.com/opensearch-project/OpenSearch/pull/4359 to 2.x

Codecov Report

Merging #4379 (27dd72a) into 1.3 (1cf7ea2) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                1.3    #4379      +/-   ##
============================================
+ Coverage     77.80%   77.83%   +0.02%
- Complexity    63340    63371      +31
============================================
  Files          4453     4453
  Lines        274899   274899
  Branches      41165    41165
============================================
+ Hits         213884   213959      +75
+ Misses        44069    44023      -46
+ Partials      16946    16917      -29

Impacted Files
Coverage Δ

...nsearch/search/dfs/DfsPhaseExecutionException.java
0.00% <0.00%> (-66.67%)
:arrow_down:

...a/org/opensearch/client/cluster/ProxyModeInfo.java
0.00% <0.00%> (-57.90%)
:arrow_down:

...n/indices/upgrade/post/UpgradeSettingsRequest.java
30.00% <0.00%> (-45.00%)
:arrow_down:

...regations/metrics/AbstractHyperLogLogPlusPlus.java
51.72% <0.00%> (-44.83%)
:arrow_down:

...java/org/opensearch/threadpool/ThreadPoolInfo.java
56.25% <0.00%> (-37.50%)
:arrow_down:

.../opensearch/indices/InvalidAliasNameException.java
62.50% <0.00%> (-25.00%)
:arrow_down:

...pensearch/cluster/routing/PlainShardsIterator.java
75.00% <0.00%> (-25.00%)
:arrow_down:

.../opensearch/transport/ProxyConnectionStrategy.java
62.12% <0.00%> (-23.49%)
:arrow_down:

...g/opensearch/action/get/MultiGetShardResponse.java
76.92% <0.00%> (-23.08%)
:arrow_down:

...ain/java/org/opensearch/geometry/MultiPolygon.java
80.00% <0.00%> (-20.00%)
:arrow_down:

... and 484 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@kotwanikunal mind approving the last one in series please, thank you!

The WhiteSource Security Check is failing consistenly on 1.3 :(

authors: codecov-commenter reta


text:
Create shared parent class for HttpInputClient and DestinationHttpClient

 Issue by ann3431
Friday Jul 19, 2019 at 22:20 GMT
Originally opened as https://github.com/opendistro-for-elasticsearch/alerting/issues/87

HttpInputClient and DestinationHttpClient has duplicate functionalities. We should create a shared parent class for these two.

Closing this since HttpInput eventually became the LocalUriInput feature which is currently in PR and this issue no longer applies there.

authors: aditjind qreshi

repo: opensearch-project/alerting


text:
Generating CSV doesn’t include seconds and seconds fractions

Is your feature request related to a problem? Please describe.
Currently for the date field in csv report. In the code there are 2 steps to handle date value.

during data query stage, we added a format: date_hour_mintue as docValue(e.g. yyyy-MM-dd'T'HH:mm)
https://github.com/opensearch-project/dashboards-reports/blob/e5174537800c60b1bf3145a80c4a36ab4227b80b/dashboards-reports/server/routes/utils/savedSearchReportHelper.ts#L270
During csv rendering stage, we use moment.js to format it into 'MM/DD/YYYY h:mm:ss a', (e.g.  06/27/2021 9:59:00 pm)
https://github.com/opensearch-project/dashboards-reports/blob/e5174537800c60b1bf3145a80c4a36ab4227b80b/dashboards-reports/server/routes/utils/dataReportHelpers.ts#L173-L175
https://github.com/opensearch-project/dashboards-reports/blob/e5174537800c60b1bf3145a80c4a36ab4227b80b/dashboards-reports/server/routes/utils/constants.ts#L67

Notice is the first step above, we are cutting off the seconds and seconds fractions, comparing with what seems like the default date field format in advanced UI setting. And in step 2, the seconds fields will always be 00, becuase of the cut off in step 1.

Describe the solution you'd like
maybe cutting off seconds for date field is not a good choice. We have the following format as available options. Maybe we should use date_hour_mintue_seconds or date_hour_minute_second_fraction
Describe alternatives you've considered
Retrieve date format setting from Advanced UI setting and use that, but I feel like this will not only apply to date format, but also other settings, such as csv seperator, timezone, url prefix, etc. It's better to add it as a compete feature to support advanced UI setting loading
Additional context
This issue was originally raised from Opensearch forum https://discuss.opendistrocommunity.dev/t/generating-csv-doesnt-include-seconds-on-timestemp-fields/6413/6

We can use uiSettings.get('dateFormat') in frontend to get the date format in advanced settings.
The API request needs to be modified or a new route needs to be added to support all advanced settings (context menu and dashboards server doesn't have direct access to uiSettings). Also the default timeFormat MMM D, YYYY @ HH:mm:ss.SSS doesn't look very common. I'm thinking to just extend current time format to MM/DD/YYYY h:mm:ss.SSS a to fix this.

@joshuali925
uiSettings are available in CoreStart as this PR suggests, is it doable in our case? So overall we want to make sure at least for values,  what users see in Discover is the same as what they get in csv report.

we can do this step by step, no need to support everyting in advanced settings at once

@zhongnansu Yes this is doable, my concern is that 10/21/2021 6:55:21.748 am should be easier to parse and more standard than Oct 20, 2021 @ 23:55:21.748 in CSV in general. Also this way we don't introduce a big change to time fields in CSV.
But I'm not that familiar with csv processing tools, and if you feel it's ok to make the change then I can update the PR

@joshuali925 Found some reference. https://github.com/elastic/kibana/issues/56153
Instead of using some predefined format, we'll need to move to use UI settings anyway
@kgcreative  Hi Kevin, any thoughts?

@zhongnansu Got it, then it's better to let users decide. I updated the PR

Moving the discussion here https://github.com/opensearch-project/dashboards-reports/issues/208

repo: opensearch-project/dashboards-reports


text:
Add append option to add_entries processor

Description
Adds an append_if_key_exists option to add_entries processor as described in #4129
Issues Resolved
Resolves #4129
Check List

[x] New functionality includes testing.
[ ] New functionality has a documentation issue. Please link to it in this PR.

[ ] New functionality has javadoc added

[x] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Would it make sense to instead have something like this
action_when_key_exists: append //  can be one of `append`, `overwrite`, `skip`

Not a blocking comment just something to consider

action_when_key_exists: append //  can be one of append, overwrite, skip
@graytaylor0 I like it, making the config more concise. I can do the change if other reviewers think the same.

Would it make sense to instead have something like this
action_when_key_exists: append //  can be one of `append`, `overwrite`, `skip`

Not a blocking comment just something to consider

@graytaylor0 , I like this idea. But, I also think we should aim for consistency. Right now grok has overwrite_when_key_exists. Do you think grok could also move toward using this approach? I'd really like to see all the processors use similar behavior. And I think this enum could facilitate that.

These processors have similar overwrite_if_key_exists options:

copy_values
rename_keys
parse_json
parse_ion
key_value

grok processor has a keys_to_overwrite option that determines overwrite or append (default) for each grok generated fields.
I will open a follow-up issue for these and we can discuss further there.

authors: dlvenable graytaylor0 oeyh

repo: opensearch-project/data-prepper


text:
ByFieldRerank Processor (ReRankProcessor enhancement)

Description
{
    "response_processors": [
        {
            "rerank": {
                "by_field": {
                    "target_field": "ml_score",
                     "remove_target_field": true ## Default false
                }
            }
        }
    ]
}

The ByFieldRerank Processor applies a 2nd level re ranking on a search hit by specifying a targetField in the _source mapping of a search hit. Currently we support both shallow target and also a nested field target  such a.b.c given that this mapping exists.
When deleting a target field as specified by "remove_target_field": true the processor will delete any empty maps as a result of this action for example, when the targetField is ml_info.score_report. It will transform
{
   "my_field": "value"
   "ml_info" : {
       "score_report" : 27
   }
}

into the following
{
   "my_field": "value"
}

It was chosen this way to give users the flexibility to clean up their searchHit, however the default behavior was disabled to let new users adapt to the feature.
Related Issues
Resolves https://github.com/opensearch-project/neural-search/issues/926
Resolves https://github.com/opensearch-project/OpenSearch/issues/15631
Check List

[X] New functionality includes testing.
[ ] New functionality has been documented.
[ ] API changes companion pull request created.
[X] Commits are signed per the DCO using --signoff.
[ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

The functionality has been implemented all that is left is more UTs, hence I left the PR as a draft. I can't think of integration testing for it as it doesn't interact with a remote process.

The functionality has been implemented all that is left is more UTs, hence I left the PR as a draft. I can't think of integration testing for it as it doesn't interact with a remote process.

We can interact with local model, it should be fine for the sake of integ test, something similar to how it's done in existing test for rerank processor https://github.com/opensearch-project/neural-search/blob/main/src/test/java/org/opensearch/neuralsearch/processor/rerank/MLOpenSearchRerankProcessorIT.java

Will do the second round of review tonight

Add changelog in the PR.

@vibrantvarun why we're flipping this request from feature to enhancement? To me that's a clear feature, we're adding functionality and extending interface with new parameters

I thought it is the enhancement of rerank processor which already exist in the project. @martin-gaievski I am fine with either of the label.

authors: brianf-aws martin-gaievski vibrantvarun

repo: opensearch-project/neural-search


text:
Add bwc tests against the distribution

Signed-off-by: Zelin Hao zelinhao@amazon.com
Description
Add option to run BWC tests in distribution level, which is running tests in the test cluster with latest distribution bundle installed and all plugins (included in the latest manifest) present.
The command to run BWC tests in distribution level would be ./gradlew bwcTestSuite -Dtests.security.manager=false -PcustomDistributionDownloadType=bundle
If property customDistributionDownloadType is not set nor set to bundle, the BWC tests would be default to run in plugins level as previously configured.
Issues Resolved
Part of https://github.com/opensearch-project/opensearch-build/issues/2870
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1366 (e2a7ac2) into 2.x (d2a17a3) will increase coverage by 28.21%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##                2.x    #1366       +/-   ##
=============================================
+ Coverage     41.89%   70.10%   +28.21%
  Complexity      315      315
=============================================
  Files           302       46      -256
  Lines         17849     2482    -15367
  Branches       4332      253     -4079
=============================================
- Hits           7477     1740     -5737
+ Misses        10199      600     -9599
+ Partials        173      142       -31

Flag
Coverage Δ

dashboards-observability
?

opensearch-observability
70.10% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...nalytics/redux/slices/viualization_config_slice.ts

...config_panes/config_controls/config_text_input.tsx

...izations/config_panel/config_panes/json_editor.tsx

...rds-observability/common/constants/autocomplete.ts

...nfig_panes/config_controls/config_style_slider.tsx

...ublic/components/visualizations/charts/bar/bar.tsx

...tebooks/components/helpers/legacy_route_helpers.ts

...ponents/visualizations/charts/maps/treemap_type.ts

.../common/query_manager/ast/builder/stats_builder.ts

...isualizations/shared_components/toolbar_button.tsx

... and 246 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@joshuali925 Please help re-review this PR as I just rebase the branch to resolve conflicts. Thanks!

authors: codecov-commenter zelinh

repo: opensearch-project/observability


text:
Created 1.2.3 manifest.

Signed-off-by: dblock dblock@dblock.org
Description
Created 1.2.3 manifest.
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Didn't want to use the automation?

Current automation doesn't add to the cron, that's still a TODO :(

repo: opensearch-project/opensearch-build


text:
Field, Function and test fixes, adding missing struct to node stats

Description

Fix FLS role type
Fix json compare when response is null
Fix function where the wrong var is given to parse the response
Add missing node stats cache field

Issues Resolved
Closes https://github.com/opensearch-project/opensearch-go/issues/560
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Want to write some tests for this one? Looks like lots of tests failed.

Test are failing because of the omitempty tag causing the json compare to omit them and resulting in missing fields when comparing. I am currently thinking about a solution.

I am not totally happy with this solution as I would have liked to not duplicate the structs but there is not lib that compares a struct against a json blob and respects the omitempty tag. Or at least I did not find such a lib.
Therefore duplicating the structs but without omitemtpy is simplest and easiest way of solving this.

repo: opensearch-project/opensearch-go


text:
Task-able long running index operation enhancement and admin UI panel for composite index template.

Describe the blog post your would like to write

A trackable index operation mechanism applied to Split / Shrink / Force merge / Close operations.
A centralized notification management page to help manage notifications for default / specific operations.
Admin UI panel for composite index template.

What is the title of the blog post?
Enhancement on long running index operation and admin UI panel for composite index template.
Who are the authors?
lxuesong, suzhou, gbinlong, ihailong, zhichaog.
What is the proposed posting date?
2023.04.28

Synced with SMEs to establish workflow and writing schedule.

As the Notification Management won't be release on v2.7, the blog release date will be slipped to the release date of v2.8. Sorry for the inconvenience.

@SuZhou-Joe Please provide updated review and publish dates for this blog.

@SuZhou-Joe Please provide updated review and publish dates for this blog since 2.8 has been released.

Thanks for reminding, the estimated date for review will be 06.30 and the expected publish date will be 07.07

authors: SuZhou-Joe pajuric vagimeli

repo: opensearch-project/project-website


text:
Text Queries in OpenSearch

Describe the blog post your would like to write
Text Queries in OpenSearch
What is the title of the blog post?
Text Queries in OpenSearch - match_only text field optimization. Tied to OpenSearch 2.12 release on 2/20.
Who are the authors?
Saurabh Singh, Rishabh Kumar Maurya
What is the proposed posting date?
2/20/24

Closing this request. Please feel free to reopen if you plan to move this forward.

@getsaurabh02 - I am going to close this issue for now. When you are ready to move it forward, we'll reopen ing.


text:
Cms/blogs/try again what in the ml is going on around here

Description
Blog PR, Draft 1
Issues Resolved
#2001
Check List

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

Thanks for your help Nate. I'll make a few changes and then let you have at it.

@nateynateynate @dtaivpp - This is ready to post once the date and meta is updated. Let's get this shipped today, please.

authors: nateynateynate pajuric


text:
[CVE-2023-2976] Fix google-java-format-1.17.0.jar: 1 vulnerabilities

Description
Fix google-java-format-1.17.0.jar: 1 vulnerabilities
Issues Resolved
Resolves https://github.com/opensearch-project/security-analytics/issues/511
CVE: https://www.mend.io/vulnerability-database/CVE-2023-2976
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Only main (fix merged) and 2.x branch has this vulnerability. Please merge the backport https://github.com/opensearch-project/security-analytics/pull/527
Older branches 2.4-2.9 do not have this vulnerability. We can safely remove those backport tags since they are not required.

repo: opensearch-project/security-analytics


text:
JWT authentication: kibana rejects valid tokens after session expiry

Symptoms:

user successfully logs in using jwt access token. Token based user, roles all OK
access token refreshed on a 10 minute basis
new access token delivered to kibana, usually kibana responds without an issue.

However, sometimes:
4-a) kibana responds with 302 /kibana/customerror?type=sessionExpired#?_g=()
4-b) I can test the token delivered in step 3 with elastic and it authenticates fine.

clearing all cookies, starting in incognito session, trying in different browser, all produce same 302 error

Workaround:
Only restarting kibana seems to work. :-(
After restarting only kibana, and using same token, authentication works.
Comments:
Suspect, but can't validate that kibana server is somehow maintaining state associated with token or it's cookie ?
kibana log in verbose mode
kibana-service       | {"type":"log","@timestamp":"2019-06-04T22:24:11Z","tags":["plugin","debug"],"pid":1,"message":"Checking Elasticsearch version"}
kibana-service       | {"type":"log","@timestamp":"2019-06-04T22:24:13Z","tags":["debug","legacy-proxy"],"pid":1,"message":"Event is being forwarded: connection"}
kibana-service       | {"type":"log","@timestamp":"2019-06-04T22:24:13Z","tags":["debug","legacy-service"],"pid":1,"message":"Request will be handled by proxy POST:/elasticsearch/_msearch?rest_total_hits_as_int=true&ignore_throttled=true."}
kibana-service       | {"type":"response","@timestamp":"2019-06-04T22:24:13Z","tags":[],"pid":1,"method":"post","statusCode":302,"req":{"url":"/elasticsearch/_msearch?rest_total_hits_as_int=true&ignore_throttled=true","method":"post","headers":{"connection":"upgrade","host":"localhost","content-length":"1383","accept":"application/json, text/plain, */*","origin":"http://localhost","kbn-version":"6.7.1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","dnt":"1","content-type":"application/x-ndjson","referer":"http://localhost/kibana/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","securitytenant":"__user__"},"remoteAddress":"172.19.0.11","userAgent":"172.19.0.11","referer":"http://localhost/kibana/app/kibana"},"res":{"statusCode":302,"responseTime":8,"contentLength":9},"message":"POST /elasticsearch/_msearch?rest_total_hits_as_int=true&ignore_throttled=true 302 8ms - 9.0B"}
kibana-service       | {"type":"log","@timestamp":"2019-06-04T22:24:13Z","tags":["debug","legacy-proxy"],"pid":1,"message":"Event is being forwarded: connection"}
kibana-service       | {"type":"log","@timestamp":"2019-06-04T22:24:13Z","tags":["debug","legacy-service"],"pid":1,"message":"Request will be handled by proxy GET:/customerror?type=sessionExpired."}
kibana-service       | {"type":"response","@timestamp":"2019-06-04T22:24:13Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/customerror?type=sessionExpired","method":"get","headers":{"connection":"upgrade","host":"localhost","accept":"application/json, text/plain, */*","kbn-version":"6.7.1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36","dnt":"1","referer":"http://localhost/kibana/app/kibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"172.19.0.11","userAgent":"172.19.0.11","referer":"http://localhost/kibana/app/kibana"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"GET /customerror?type=sessionExpired 200 8ms - 9.0B"}
kibana-service       | {"type":"log","@timestamp":"2019-06-04T22:24:13Z","tags":["plugin","debug"],"pid":1,"message":"Checking Elasticsearch version"}

I've verified that a new token was sent.  However, kibana does not respond with it's own security_authentication cookie.

@hardik-k-shah
https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/blame/v0.9.0.0/lib/auth/types/AuthType.js#L149-L155
I'm suspecting this section.  My tokens always have the same header:
{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "XXXX_key_2019-03-08T05:28:18Z"
}

Would that cause the " no need to return new credentials" test to ignore the new token?

I suspect, but cannot prove, that the security_authentication cookie is out of step with the Authorization: Bearer token.
My brute force workaround seems to confirm this.
In nginx:
        # remove kibana's security_authentication cookie.
        set $sans_security_authentication $http_cookie;
        if ($http_cookie ~ "(.*)(?:^|;)\s*security_authentication=[^;]+(.*)") {
            set $sans_security_authentication $1$2;
        }
        proxy_set_header Cookie $sans_security_authentication;

...

        location /kibana {
            proxy_pass_request_headers off;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection 'upgrade';
            proxy_set_header Host $host;
            proxy_cache_bypass $http_upgrade;
            rewrite /kibana/(.*) /$1 break;
            proxy_set_header Authorization "$access_token";
            proxy_set_header kbn-version "$http_kbn_version";
            proxy_pass http://kibana-service:5601/;
        }

Not that I'm experiencing the same but I believe there is something wrong on how Kibana refreshes cookies' TTL when receiving requests. In my case I'm using SAML, not JWT, but user experience I believe is similar (session expiry is not updated even user is actively using kibana).
I'll create a separate issue for this (currently github is giving me 500 errors =:O) but here's the post I created sometime back:
https://discuss.opendistrocommunity.dev/t/kibana-session-keepalive-when-using-saml/724

any updates?

Any updates?

Try setting your cookie ttl to the same exact time as your access token expiration opendistro_security.cookie.ttl

Any updates?

I managed to get it working by adding the following configurations:
opendistro_security.cookie.ttl : [jwtExiprationTime]
opendistro_security.session.ttl: 2 * [jwtExiprationTime]
opendistro_security.session.keepalive: true

We are doing some "spring cleaning in the fall", and to make sure we focus our energies on the right issues and we get a better picture of the state of the repo, we are closing all issues that we are carrying over from the ODFE era (ODFE is no longer supported/maintained, see post here).
If you believe this issue should still be considered for current versions of OpenSearch, apologies! Please let us know by re-opening it.
Thanks!

authors: bwalsh davidlago horacimacias kakulukia maismail samoilenko shamsalmon


text:
Update integTest gradle scripts to run via remote cluster independently

Is your feature request related to a problem?
With the existing implementation of integration tests, opensearch plugins use the same test framework as local integration tests which spin up test clusters by itself. This is a hacky way to override the endpoint from local spun up test clusters (via opensearch.testclusters gradle plugin) with the remote cluster endpoint.
The solution doesn't work when we would like to test the opensearch bundle as part of the release. The test frameworks expect the artifact to be available before we could test them. To clean this up, plugins have to create gradle scripts for remote endpoints to run https://github.com/opensearch-project/opensearch-build#integration-tests for the release cycle.
What solution would you like?
Refer to: https://github.com/opensearch-project/anomaly-detection/pull/298 and make similar changes to the plugin.
Also if you have a custom name to invoke these tests, update the integtest.sh script in https://github.com/opensearch-project/opensearch-build/blob/main/scripts/default/integtest.sh

@saratvemulapalli Does this issue apply to frontend plugins?

This request doesn't apply to dashboards plugin. Ref: https://github.com/opensearch-project/opensearch-plugins/issues/103#issuecomment-1064458549

This request doesn't apply to dashboards plugin. Ref: https://github.com/opensearch-project/opensearch-plugins/issues/103#issuecomment-1064458549


text:
Add IP filtering to OpenSearch security

When securing a cluster, being able to filter connections based on IP addresses is a nice and simple feature.
By defining lists of IPs or ranges via the CIDR notation, it should be possible to restrict access to a node.
OpenSearch security already allows the restriction of connections based on distinguished names (https://opensearch.org/docs/latest/security-plugin/configuration/yaml/#nodes_dnyml). IP filtering would be a complementary feature of the same type.
IntraFind (my employer) has developed a proprietary IP filtering plugin with the features described above for Elasticsearch v1, which we have been upgrading and maintaining since then. We are interested in contributing the functionality to OpenSearch.

[Triage] This is a great feature request, we would welcome a contribution that adds this functionality.

I would like to do it a bit more granular: add ip whitelist for a role in opensearch. We have implemented this ourselfs so that a JWT with appid X can only access the cluster from specific CIDR's.
Currently you can apply a role based on ip-ranges, but this is different from that.

@peternied @cwperks I'm very interested in taking on this issue as a more influential project, but I'm very lost on where to start, where should I start referencing to get the basic information on features like this? Tysm!

This would enable us to drop a custom reverse proxy. If you need any input on the functional side: no problem. Happy to schedule a call.

@prabhask5 For where to start in the code during the life cycle checkAndAuthenticateRequest [1] is where the security plugin determines who is making the request, if we don't know we reject the request.  This seems like an 'easy' existing point in the process to allow/reject requests based on a CIDR range.
However, I'd caution, this is a feature of sizable scope, so rather than start with a only a draft pull request; I'd recommend creation an issue with this design template [2] that will help you plan out and get buy in from the maintainers of the direction you are thinking.

[1] Relevant code path for authentication https://github.com/opensearch-project/security/blob/905c97d4fed185362a12c4b46aa56ad8febabb4d/src/main/java/org/opensearch/security/filter/SecurityRestFilter.java#L254

[2] Design doc template https://github.com/opensearch-project/security/blob/main/.github/ISSUE_TEMPLATE/DESIGN_DOCUMENT_TEMPLATE.md?plain=1

repo: opensearch-project/security


text:
[BUG] simple joins on keywords don't seem to work

What is the bug?
Joins on keyword fields do not seem to work.
From the Query Workbench:
select
   q.query_id
from
   .ubi_log_queries q
join .ubi_log_events e
    on q.query_id = e.query_id

Yields this output:
: Bad Request, this query is not runnable.
Both indices have data and the following mapping:
        "query_id" : {
          "type" : "keyword"
        },

The OpenSearch logs have:
[2024-03-12T13:43:44,456][INFO ][o.o.s.l.p.RestSqlAction  ] [opensearch] [a77b58fa-bf13-4e8d-b9e7-c517f3793a96] Incoming request /_plugins/_sql
[2024-03-12T13:43:44,459][WARN ][stderr                   ] [opensearch] line 1:49 mismatched input 'join' expecting {<EOF>, ';'}
[2024-03-12T13:43:44,459][INFO ][o.o.s.l.p.RestSqlAction  ] [opensearch] [a77b58fa-bf13-4e8d-b9e7-c517f3793a96] Request SQLQueryRequest(jsonContent={"query":"select    q.query_id from     .ubi_log_queries q join .ubi_log_events e     on q.query_id = e.query_id"}, query=select    q.query_id from     .ubi_log_queries q join .ubi_log_events e     on q.query_id = e.query_id, path=/_plugins/_sql, format=jdbc, params={}, sanitize=true, cursor=Optional.empty) is not supported and falling back to old SQL engine
[2024-03-12T13:43:44,470][WARN ][o.o.s.l.u.QueryDataAnonymizer] [opensearch] Caught an exception when anonymizing sensitive data.
[2024-03-12T13:43:44,470][INFO ][o.o.s.l.p.RestSqlAction  ] [opensearch] Request Query: Failed to anonymize data.
[2024-03-12T13:43:44,470][ERROR][o.o.s.l.p.RestSqlAction  ] [opensearch] a77b58fa-bf13-4e8d-b9e7-c517f3793a96 Client side error during query execution
com.alibaba.druid.sql.parser.ParserException: Illegal SQL expression : select    q.query_id from     .ubi_log_queries q join .ubi_log_events e     on q.query_id = e.query_id
        at org.opensearch.sql.legacy.utils.Util.toSqlExpr(Util.java:279) ~[legacy-2.12.0.0.jar:?]
        at org.opensearch.sql.legacy.query.OpenSearchActionFactory.create(OpenSearchActionFactory.java:89) ~[legacy-2.12.0.0.jar:?]
        at org.opensearch.sql.legacy.plugin.SearchDao.explain(SearchDao.java:48) ~[legacy-2.12.0.0.jar:?]
        at org.opensearch.sql.legacy.plugin.RestSqlAction.explainRequest(RestSqlAction.java:243) [legacy-2.12.0.0.jar:?]
        at org.opensearch.sql.legacy.plugin.RestSqlAction.lambda$prepareRequest$1(RestSqlAction.java:172) [legacy-2.12.0.0.jar:?]
        at org.opensearch.sql.legacy.plugin.RestSQLQueryAction$1.onFailure(RestSQLQueryAction.java:130) [legacy-2.12.0.0.jar:?]
        at org.opensearch.sql.sql.SQLService.execute(SQLService.java:43) [sql-2.12.0.0.jar:?]
        at org.opensearch.sql.legacy.plugin.RestSQLQueryAction.lambda$prepareRequest$3(RestSQLQueryAction.java:107) [legacy-2.12.0.0.jar:?]
        at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) [opensearch-2.12.0.jar:2.12.0]
        at org.opensearch.rest.RestController.dispatchRequest(RestController.java:334) [opensearch-2.12.0.jar:2.12.0]
        at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:425) [opensearch-2.12.0.jar:2.12.0]
        at org.opensearch.rest.RestController.dispatchRequest(RestController.java:263) [opensearch-2.12.0.jar:2.12.0]
        at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:387) [opensearch-2.12.0.jar:2.12.0]
        at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:468) [opensearch-2.12.0.jar:2.12.0]
        at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:370) [opensearch-2.12.0.jar:2.12.0]
        at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:56) [transport-netty4-client-2.12.0.jar:2.12.0]
        at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:42) [transport-netty4-client-2.12.0.jar:2.12.0]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72) [transport-netty4-client-2.12.0.jar:2.12.0]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) [netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) [netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
[2024-03-12T13:47:17,192][INFO ][o.o.j.s.JobSweeper       ] [opensearch] Running full sweep

What is the expected behavior?
Rows of data should have been returned instead of an error.
What is your host/environment?

all versions are OS and plugins are 2.12

this is a bug when handing string with ., https://github.com/opensearch-project/sql/blob/main/legacy/src/main/java/org/opensearch/sql/legacy/utils/StringUtils.java#L93
to migrate the issue, is it possible to remove . from index name

@penghuo I'm happy to work this issue if you want to assign it to me. It will benefit our OpenSearch UBI work.

I suspect that you don't need it assigned to you to submit a PR ;-)

authors: RasonJ epugh jzonthemtn penghuo

repo: opensearch-project/sql


text:
Add STR_TO_DATE Function To The SQL Plugin

Description
Adds the str_to_date function to the SQL plugin. The function takes two arguments, an input string and a format string. The format string contains string format specifiers which describe how the characters in the input string are parsed to a DATETIME. Based off of MySQL, but differs in some areas due to the limitations of the plugin and Java's built-in DatetimeFormatter. Specifically, the format string used must exactly match the string being parsed, and any dates/times with 0 for the year/month/day will return NULL since these fields must be valid for Java LocalDate/LocalTime/LocalDatetime. The arguments must have enough information to build a DATE, TIME, or DATETIME.
Example:
SELECT str_to_date("May 1, 2013", "%M %d, %Y") -> 2013-05-01 00:00:00
SELECT str_to_date("9,23,11", "%h,%i,%s") -> 0001-01-01 09:23:11
Issues Resolved
#722
Check List

[X] New functionality includes testing.

[X] All tests pass, including unit test, integration test and doctest

[X] New functionality has been documented.

[X] New functionality has javadoc added
[X] New functionality has user manual doc added

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #1420 (3532f4c) into main (ef38389) will increase coverage by 0.01%.
The diff coverage is 100.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #1420      +/-   ##
============================================
+ Coverage     98.38%   98.40%   +0.01%
- Complexity     3698     3723      +25
============================================
  Files           343      343
  Lines          9121     9211      +90
  Branches        586      600      +14
============================================
+ Hits           8974     9064      +90
  Misses          142      142
  Partials          5        5

Flag
Coverage Δ

sql-engine
98.40% <100.00%> (+0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...c/main/java/org/opensearch/sql/expression/DSL.java
100.00% <100.00%> (ø)

...sql/expression/datetime/DateTimeFormatterUtil.java
100.00% <100.00%> (ø)

...arch/sql/expression/datetime/DateTimeFunction.java
100.00% <100.00%> (ø)

...h/sql/expression/function/BuiltinFunctionName.java
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


text:
Binding retries in stateless brokers

According to OSB Bindings spec:

200 OK: MUST be returned if the binding already exists and the requested parameters are identical to the existing binding. The expected response body is below.

How can this requirement be achieved in stateless brokers (see https://github.com/openservicebrokerapi/servicebroker/issues/203 and https://github.com/openservicebrokerapi/servicebroker/issues/225#issuecomment-311230413)? This requirement implicitly stands that:

Broker stores the request payload for each binding somewhere (i.e. broker is stateful).
Broker is able to re-retrieve the credentials for existing Binding (despite that there is no GET endpoint).

I'm pretty sure that existing stateless brokers don't respect this requirement, otherwise they are stateful 😕
Shall we remove this requirement from the spec?

This seems to be a subset of #225

@arschles it's related, but not a subset.
The question for this issue is whether this requirement is always respected in the current world of brokers. If not, we should remove it from the spec.
Also see related issue in Service Catalog: #1062
If this requirement is respected, there is no need for preventing duplicate requests from the Service Catalog side.

related #291
proposal: https://github.com/openservicebrokerapi/servicebroker/issues/291#issuecomment-323344056

I'm not sure what a good solution to this issue is. I understand the reasoning behind a stateless broker not being able to respond with a 200 OK here as it can't remember what parameters were used originally, but those brokers could always just respond with a 409 Conflict:

409 Conflict: MUST be returned if a Service Binding with the same id, for the same Service Instance, already exists or is being created but with different parameters.

What do you think @fmui ?

Looks like we solved that already by accident. See #528

You're right, that does technically solve this, but I still think that the spec doesn't offer much guidance as to what service brokers should do if they don't have state and they can't determine if the configuration parameters were the same as before. Should they return a 200 or 409 in that case?

Is this a problem for anyone, or can we close this for now and wait until it becomes a problem?

Closing due to inactivity. Please reopen if this becomes a problem for anyone.

authors: arschles duglin fmui mattmcneeney nilebox

repo: openservicebrokerapi/servicebroker


text:
Planning Milestone v1.1.0

This GitHub Issue is to be used for planning Open Service Mesh release v1.1.0
Please comment with recommendations on GitHub Issues or general areas that we should bundle with the v1.1.0 release.

Proposed tasks for v1.1.0:

#4351

Closing this as we will be planning v1.1.0 differently

authors: draychev snehachhabria

repo: openservicemesh/osm


text:
Fix MRC status

Signed-off-by: Keith Mattix II keithmattix2@gmail.com

Description:
Correctly sets the preset MRC's status on initial create and fixes a typo on the printer column of the MRC CRD

Testing done:
Ran the demo and checked the MRC manifest

Affected area:

Functional Area

Certificate Management
[X]

Please answer the following questions with yes/no.

Does this change contain code from or inspired by another project? no

Did you notify the maintainers and provide attribution? N/A

Is this a breaking change? no

Has documentation corresponding to this change been updated in the osm-docs repo (if applicable)? N/A

In osm-bootstrap_test.go we could add a check to TestCreateMeshRootCertificate to verify the MRC status has been set as expected.

Codecov Report

Merging #4856 (f42a565) into main (d970b24) will decrease coverage by 0.00%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##             main    #4856      +/-   ##
==========================================
- Coverage   69.51%   69.50%   -0.01%
==========================================
  Files         219      219
  Lines       16033    16041       +8
==========================================
+ Hits        11146    11150       +4
- Misses       4833     4837       +4
  Partials       54       54

Flag
Coverage Δ

unittests
69.50% <66.66%> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

cmd/osm-bootstrap/osm-bootstrap.go
47.93% <66.66%> (+0.05%)
:arrow_up:

pkg/messaging/workqueue.go
89.28% <0.00%> (-10.72%)
:arrow_down:

pkg/ticker/ticker.go
87.17% <0.00%> (+3.84%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d970b24...f42a565. Read the comment docs.

authors: codecov-commenter jaellio keithmattix


text:
Add ibgu package

Add ibgu package for ImageBasedGroupUpgrades custom resource.

cc @josclark42

@mcornea Can you please split this PR into 2 different PRs. 1 PR for sync and scheme update 2 PR for ibgu package

Sure, I opened https://github.com/openshift-kni/eco-goinfra/pull/663 for the sync and schema.

I'll rebase once https://github.com/openshift-kni/eco-goinfra/pull/663 gets merged.

repo: openshift-kni/eco-goinfra


text:
Fix RBAC role for ImageBasedUpgrade

This adds the missing RBAC role for provisioning request controller to access and modify IBGUs.

/lgtm

retested on my system.  it works now.

/approve

authors: Missxiaoguo alegacy sudomakeinstall2


text:
Update manager config to use postgresql-16-c9s image

Replaced the postgresql image with one that is publicly available in order to avoid requiring a pull secret in deployment automation being contributed to the ORAN-SC community.

/lgtm

/approve

authors: clwheel donpenney mlguerrero12


text:
Use custom service account for metrics deployment

Local run:
[bnr@bnr pipeline-service]$ k -n openshift-pipelines get pods
NAME                                                 READY   STATUS        RESTARTS   AGE
pipeline-metrics-exporter-7c87c975bc-t2r5r           1/1     Running       0          33s

# HELP pipeline_service_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which pipeline_service_exporter was built.
# TYPE pipeline_service_exporter_build_info gauge
pipeline_service_exporter_build_info{branch="",goversion="go1.19.5",revision="",version=""} 1
# HELP pipelinerun_duration_completed_seconds Duration in seconds for a PipelineRun to complete.
# TYPE pipelinerun_duration_completed_seconds gauge
pipelinerun_duration_completed_seconds{name="pipelinerun-echo-greetings",uid="da686962-80de-413e-8ee8-34987b60c0ac"} 10
# HELP pipelinerun_duration_scheduled_seconds Duration in seconds for a PipelineRun to be scheduled.
# TYPE pipelinerun_duration_scheduled_seconds gauge
pipelinerun_duration_scheduled_seconds{name="pipelinerun-echo-greetings",uid="da686962-80de-413e-8ee8-34987b60c0ac"} 0

/lgtm

authors: bnallapeta xinredhat

repo: openshift-pipelines/pipeline-service


text:
Refine Dependency Checking and Improve Error Messaging in Development Environment Script

Changes

The pull request introduces enhancements to the development environment script, focusing on improving the accuracy of dependency checks and providing informative error messages for a smoother onboarding experience.
Submitter Checklist

[ ] 📝 A good commit message is important for other reviewers to understand the context of your change. Please refer to How to Write a Git Commit Message for more details how to write beautiful commit messages. We rather have the commit message in the PR body and the commit message instead of an external website.
[ ] ♽ Run make test before submitting a PR (ie: with pre-commit, no need to waste CPU cycle on CI. (or even better install pre-commit and do pre-commit install in the root of this repo).
[ ] ✨ We heavily rely on linters to get our code clean and consistent, please ensure that you have run make lint before submitting a PR. The markdownlint error can get usually fixed by running make fix-markdownlint (make sure it's installed first)
[ ] 📖 If you are adding a user facing feature or make a change of the behavior, please verify that you have documented it
[ ] 🧪 100% coverage is not a target but most of the time we would rather have a unit test if you make a code change.
[ ] 🎁 If that's something that is possible to do please ensure to check if we can add a e2e test.
[ ] 🔎 If there is a flakiness in the CI tests then don't necessary ignore it, better get the flakyness fixed before merging or if that's not possible there is a good reason to bypass it. (token rate limitation may be a good reason to skip).

Fixes #1531
/cc @chmouel @vdemeester

/ok-to-test

This looks good, just a comment! since we use this on our e2e tests which is run by github actions we should test this since i don't know why GHA don't let me run this on external contributor via an approval process like the pac's /ok-to-test (maybe i have tighten down too much the security there)

it works for me locally on my laptop (TM)
Let me know if you want to address the suggestions but i am fine to merge this as is

authors: chmouel roman-kiselenko

repo: openshift-pipelines/pipelines-as-code