text:
detect.php says that An unexpected URL was returned

I do not know if it is really bug, but when I open example/detect.php I am getting
HTTP Fetching

This PHP installation has support for libcurl. Good.

An HTTP request was completed. An unexpected URL was returned: https://gist.githubusercontent.com/chuckpearce/6810258/raw/c57eff55ebc0c54973903af5f72bac72762cf4f4/gistfile1.txt.

Your PHP installation appears to support SSL, so it will be able to process HTTPS identity URLs and server URLs.

But if I replace in examples/detect.php
$fetch_url = 'https://gist.github.com/chuckpearce/6810258/raw/c57eff55ebc0c54973903af5f72bac72762cf4f4/gistfile1.txt';
with
$fetch_url = 'https://gist.githubusercontent.com/chuckpearce/6810258/raw/c57eff55ebc0c54973903af5f72bac72762cf4f4/gistfile1.txt';
all works fine and I get Setup Complete!
I think maybe github changed their infrastructure (redirecting from gist.github.com to gist.githubusercontent.com) or maybe my setup is some kind wrond.
Any suggestions?

This repo is being archived. Closing issue.

authors: laptander timcappalli

repo: openid/php-openid


text:
openjfx.io site: wrong link to reference documentation on "Getting Started" page

Hello,
The link titled "detailed reference documentation for JavaFX" on the page https://openjfx.io/openjfx-docs/ points to https://docs.oracle.com/javase/8/javase-clienttechnologies.htm instead of wherever it should be pointing to.

Thanks for reporting.
I discussed this with a number of people, and the main issue is that the "wherever it should be pointing to" doesn't exist. The old overview (JavaFX 8 based) on the Oracle site is still, from a conceptual point, a good overview.
However, many internal parts need to be updated, but we don't have access to the old site.
It would make sense to have an updated version of that 8-based webpage, and we're looking for contributors who want to help with that.

Closing as this is not an issue.

source: gharchive/issue

authors: abhinayagarwal badscrew johanvos

repo: openjfx/openjfx-docs

license_type: permissive

license_source: github-api


text:
0.10.3以上版本对bundle中有中文会报错

使用的 Kraken 版本 ｜ What version of kraken are you using

重现步骤 ｜ Steps To Reproduce

重现代码 | Code example:
见上
预期结果 ｜ Expected results:
预期与0.10.2一致可以打印
实际结果 ｜ Actual results:
报错

我看一下

@wssgcg1213 你好，这个是不是跟bundle.dart修改了有关？

bundle: KrakenBundle.fromContent('let a = "折";console.log(a);'),

是的, fromString 把字符串按照 utf16 编码了, 解析的时候是按照 utf8 解析的, 所以中文部分出错了, 我修复一下

@zelllez 你可以先这样:
bundle: DataBundle(
  Uint8List.fromList(utf8.encode(content)),
  DEFAULT_URL,
  contentType: ContentType('text', 'javascript', charset: 'utf-8')
)

repo: openkraken/kraken


text:
Feature/collaborations

This is hubble.

Quick high level feedback.

Rename projected table to just table or contributed table.
Don't add collaboration to AssemblerConnectionManager (should be separate self contained logic)
Organizations should have to accept an invitation to a collaboration (create -> invite -> contribute)
Should probably track what users from organizations are a member of the collaboration (eventually authorization within the context of a collaboration should be limited to users of a collaboration).
Needs standard representation of metadata as we are introducing a new wrapper object for metadata.

source: gharchive/pull-request

authors: geekbeast katherinebernstein

repo: openlattice/openlattice


text:
Proposals: calculate all off-chain vote results and pass down to cards

Like we've done for GovernanceProposalsList, we can pass SnapshotProposal[] to useOffchainVotingResults to get the tallies for off-chain voting (e.g. % Yes, % No) results. We can then provide this information as a prop to the proposal cards, and they won't re-fetch the same information, again.
This will also allow us to be more specific about how we establish proposal categories, because we now have access to the voting tallies. This information in-turn could be used to be passed to other not-yet-implemented callback props where the user can control how they want Proposals to be filtered, for example.

Note for Muse0 we will need to copy-in the same logic once implemented, as it uses a custom listing component which we will eventually make re-usable for other NFT projects. cc: @jdville03

Fixed in #420

repo: openlawteam/tribute-ui


text:
Add a Makefile section to .editorconfig

make requires tabs.

Please merge, Éric.

authors: elemoine marcjansen


text:
custom (my handle) tile in version 6-4-3 is not working

I have create tile source
var kmlSource = new ol.source.VectorTile(
{
//format: new ol.format.GeoJSON(),
tileLoadFunction: myTileFunction,
url: 'kml_' + feature.idLocation // arbitrary url, we don't use it in the tileLoadFunction
}
);
and in my tile Function i set features in tile. Features i take from my base. Can to answer - Whey tile function work only one time,

Hey, this kind of question might be better suited for stackoverflow, but did might need to call setLoader inside of your tile to have it called multiple times. If you are sure this is a bug, please provide some more code and idealy a fiddle on codesandbox. You can use https://openlayers.org/en/latest/examples/mapbox-vector-tiles-advanced.html as a starting point and click on edit in the upper right corner to immediately start a new codesandbox.

it happens when function  parameter url not change/ If url parameter change all working.  An it provide other problem —  not enouf memory/ In version 4 openlayers such not been

Вторник, 6 октября 2020, 19:17 +03:00 от Simon Seyock notifications@github.com:

Hey, this kind of question might be better suited for stackoverflow, but did might need to call setLoader inside of your tile to have it called multiple times. If you are sure this is a bug, please provide some more code and idealy a fiddle on codesandbox. You can use  https://openlayers.org/en/latest/examples/mapbox-vector-tiles-advanced.html as a starting point and click on edit in the upper right corner to immediately start a new codesandbox.
—
You are receiving this because you authored the thread.
Reply to this email directly,  view it on GitHub , or  unsubscribe .

Can you provide a reproduction of the bug on https://codesandbox.io ?

I can connect you to my computer via anydesc remote viewer And via whatsup you will show where openlayers have problem.
I have a lot data and code for sandbox

Среда, 7 октября 2020, 13:05 +03:00 от Simon Seyock notifications@github.com:

Can you provide a reproduction of the bug on  https://codesandbox.io ?
—
You are receiving this because you authored the thread.
Reply to this email directly,  view it on GitHub , or  unsubscribe .

I am sorry but I would not want to do this. Can you provide just a minimal example where this problem occurs? That means only the code that is necessary to reproduce this. No extra code around it. You can use https://openlayers.org/en/latest/examples/mapbox-vector-tiles-advanced.html as a starting point and click on edit in the upper right corner to immediately start a new codesandbox.

repo: openlayers/openlayers


text:
Jsdoc types syntax

With these commits the check-types and check-syntax rules of eslint-plugin-jsdoc pass.
The first commit contains some type changes and second commit only replaces the {type=} name syntax with {type} [name].

Thanks for the continued work on this, @simonseyock.

authors: simonseyock tschaub


text:
Remove absolute position on ol-overlaycontainer divs

work in progress, trying to fix #9467 and #9139
The absolute position have been added in 898c349fbfed798d68bb570f4be55cc8c6c290d3

not working, I'll try to come up with a cleaner solution


text:
OAI PMH interface is not delivering providedPlatforms

Discovered and documented as LAS:eR-issue ERMS-1573:
Comparing these two records:
https://phaeton.hbz-nrw.de/gokb/oai/packages?verb=getRecord&metadataPrefix=gokb&identifier=b2e9d143-1721-4a8a-a048-0723b6d425d7
with
https://phaeton.hbz-nrw.de/gokb/resource/show/b2e9d143-1721-4a8a-a048-0723b6d425d7
The LAS:eR-sync would expect a tag providedPlatforms which is inexistent, so no package - provider connection is being established even if there is one.

Fixed in #82

repo: openlibraryenvironment/gokb


text:
fix jeff33 chain generation

As mentioned here, the generate_jeff_chain script fails when loading the NFY information from JEFF-33's file available at this adress
The reason for the failure is that this file is malformed and lacks a proper header line. This PR adds a few lines in the generate_jeff_chain script to fix the file before loading neutron fission yields.

Awesome thanks for this fix. I've triggered the tests

authors: nplinden shimwell

license: MIT


text:
As a developer I want to know about p2p tests status

Work:

Tests for incoming/outgoing connections that test p2p up to noise handshake (including noise handshake).
Bootstrap test that tests outgoing RPC requests. It also tests yamux indirectly.
Tests for peer discovery (kademlia). But they are disabled pending fixes related to kademlia.

Don't work (TODO):

Incoming RPC requests (incoming bootstrap), should port the test to the github CI.
Don't have any tests for yamux. But other high level tests can also detect problems in yamux.
Don't have a test for Identify /ipfs/id/1.0.0.
I should also port test for pubsub (meshsub) in the CI.

Progress:
Merged pubsub branch. Ported pubsub test, running in the CI. Ensure stack is not overflow in tests.
Status:
Need tests for /ipfs/id/1.0.0, yamux, and incoming bootstrap.

repo: openmina/openmina


text:
[MLS Spec change] Introduce GroupContext in terminology. (#683)

Link to the exact changes
https://github.com/mlswg/mls-protocol/pull/683
Description of the changes

Closes #668

Editorial.

repo: openmls/openmls


text:
Reuse dropbox link

Builds on top of #993 to see if reusing the dropbox links is enough for CI to pass

The answer is "no"
~/work/openpathsampling/openpathsampling/openpathsampling/netcdfplus/dictify.py in callable_from_dict(c_dict)
    471             if '_marshal' in c_dict:
    472                 if ObjectJSON.allow_marshal:
--> 473                     code = marshal.loads(base64.b64decode(c_dict['_marshal']))
    474                     c = types.FunctionType(code, globals(), code.co_name)
    475
ValueError: bad marshal data (unknown type code)

Yeah, this is what I expected.

authors: dwhswenson sroet

repo: openpathsampling/openpathsampling


text:
Add ability to remove EON from list

Am I missing something? Can't seem to remove other EONs from the list

I accidentally registered Chffr+ and would love to remove the instance from all my dashboards.

repo: openpilot-community/workbench


text:
2015-16 Genesis min steer speed update

This change enables lateral control to start @ 34 mph and also removes the annoying below steer speed warning.

done

repo: openpilotkr/openpilot


text:
上架应用是否可以再发起上架审核？

目前按后端接口中的逻辑，“已下架”应用只能管理员进行“恢复”操作变成“已上架”。
后端早期的有关于vesion操作的文档说明，下面是操作流程图，详见(https://github.com/openpitrix/openpitrix/issues/564#issuecomment-415313018)。备注：部分逻辑在后面开发测试中有所调整。

Is this issue still in discuss?

authors: iwisunny wangxhsoft whDongRui

repo: openpitrix/dashboard


text:
应用显示

这个现在可以显示服务商的名称了，效果如下截图。 修复代码请参见PR

authors: wangxhsoft whDongRui


text:
refactor(cluster detail): Display env params as json for vmbased

Display env params as yaml for helm

fix #788


text:
Website cleanup

@LuckyMD do you mind taking a look over the website and commenting on any text you think needs changing?

First observations for text I would like to add/change;
At the end of “Our inspiration”:

Of course, computational single-cell methods have seen a lot of innovation in the past years. This is partly driven by benchmarking efforts, that quantify progress and set goals. Yet, we noticed that many successful methods draw from innovations in computer vision and other areas of machine learning that are more mature. We want to promote this cross-fertilization to increase the pace of innovation for single-cell analysis, and enable machine learning scientists to use their skills to address relevant tasks in digital healthcare research.

Before “join us”

Who are we?
We are machine learning scientists, computational biologists, single-cell data analysts who formalize computational tasks in single-cell analysis and collaborate with molecular biologists to generate benchmarking datasets that challenge methods and method developers to perform ever better.

Denoising blurb:

“Removing noise in sparse single-cell RNA-sequencing count data”

Label projection:

“Automated cell type annotation from rich, labeled reference data”

Multimodal Data Integration:

“Jointly embedding single-cell omics datasets across modalities and batches”

Spatial decomposition:

“Calling cell-type compositions for spot-based spatial transcriptomics data”

Denoising task description:

“detect a small fraction” -> “detect a fraction”

@dburkhardt Please check the changes to the main website I suggested above to see if you agree.

Round 2:
Dimensionality reduction task:

link shendure fetal expression atlas and not 10X dataset: https://www.science.org/doi/full/10.1126/science.aba7721?casa_token=XhdLEBpeayMAAAAA%3AhD1s6Aoc5aQhmQWRBygkKen4yexUuiDSmtFmkeF-X4Bsc-In-z23-YqeLbiC5GIQ3zTVGKNH6GB-pZXQ

Second paragraph, change to:

While each of the ~30,000 dimensions measured in each cell contribute to an underlying data structure, the overall structure of the data is challenging to display in few dimensions due to data sparsity and the “curse of dimensionality” (distances in high dimensional data don’t distinguish data points well). Thus, we need to find a way to dimensionally reduce the data for visualization and interpretation.

is this task description too long? -> can we make it shorter? @lazappi

move method description to code documentation?

Multimodal data integration page:
Change text to:

Cellular function is regulated by the complex interplay of different types of biological molecules (DNA, RNA, proteins, etc.), which determine the state of a cell. Several recently described technologies allow for simultaneous measurement of different aspects of cellular state. For example, sci-CAR jointly profiles RNA expression and chromatin accessibility on the same cell and CITE-seq measures surface protein abundance and RNA expression from each cell. These technologies enable us to better understand cellular function, however datasets are still rare and there are tradeoffs that these measurements make for to profile multiple modalities.

Joint methods can be more expensive or lower throughput or more noisy than measuring a single modality at a time. Therefore it is useful to develop methods that are capable of integrating measurements of the same biological system but obtained using different technologies on different cells.

Here the goal is to learn a latent space cells profiled by different technologies in different modalities are matched if they have the same state. We use jointly profiled data as ground truth so that we can evaluate when the observations from the same cell acquired using different modalities are similar. A perfect result has each of the paired observations sharing the same coordinates in the latent space.

Metrics: can we make higher better for both? And potentially normalize these to within 0-1? Atm is says that lower is better for one, but higher is better for the other

Reg effect prediction looks bad... can we remove it from the website for now?

Dimensionality reduction task:

link shendure fetal expression atlas and not 10X dataset: science.org/doi/full/10.1126/science.aba7721?casa_token=XhdLEBpeayMAAAAA%3AhD1s6Aoc5aQhmQWRBygkKen4yexUuiDSmtFmkeF-X4Bsc-In-z23-YqeLbiC5GIQ3zTVGKNH6GB-pZXQ

Not sure what this is referring to

Second paragraph, change to:

While each of the ~30,000 dimensions measured in each cell contribute to an underlying data structure, the overall structure of the data is challenging to display in few dimensions due to data sparsity and the “curse of dimensionality” (distances in high dimensional data don’t distinguish data points well). Thus, we need to find a way to dimensionally reduce the data for visualization and interpretation.

Is this in the task README? Not sure what the change is here but I didn't write this so happy for it to be changed.

is this task description too long? -> can we make it shorter? @lazappi

The paragraph above or something else? Happy to do whatever is consistent with the other tasks.

move method description to code documentation?

I think I put things in the right place last time I looked at this but things can be moved wherever they are supposed to be.

This was indeed in the task description. I've adapted a bit, but not in this PR yet. I don't think methods are described in other tasks... are they @scottgigante-immunai ?

Methods are not described in other tasks, no. By moving API above the methods, we take the methods section off the website but leave it on the github readme.

Reg effect prediction looks bad... can we remove it from the website for now?

I can set a filter in the processing script to only publish tasks that meet the criteria for a stub -- one dataset, three methods, one metric.

Metrics: can we make higher better for both? And potentially normalize these to within 0-1? Atm is says that lower is better for one, but higher is better for the other

Unfortunately I don't think there's a clean way to do this. I personally think it's better to have interpretable metrics (AUC instead of 1-normalized AUC) over metrics that always range from 0 (worst) to 1 (best)

I can set a filter in the processing script to only publish tasks that meet the criteria for a stub -- one dataset, three methods, one metric.

That sounds perfect!

Unfortunately I don't think there's a clean way to do this. I personally think it's better to have interpretable metrics (AUC instead of 1-normalized AUC) over metrics that always range from 0 (worst) to 1 (best)

This would clash with taking the mean to rank the methods though... making it uninterpretable again.

We take the rank of the mean rank of the methods, not the rank of the mean of the metric values. Each method has a flag maximize: bool which means they are either ranked top to bottom or bottom to top.

I see. Taking the average of ranks is advised against for biomedical image analysis competitions (https://doi.org/10.1038/s41467-018-07619-7). From the paper:

According to bootstrapping experiments (Figs.3 and 4), single-metric rankings are statistically highly significantly more robust when (1) the mean rather than the median is used for aggregation and (2) the ranking is performed after the aggregation.

Ah, interesting. If we're going to take the average before ranking, we need to define bounds for each metric -- perhaps we could do this by requiring tasks to provide sample methods with a) perfect performance and b) random performance, and then we scale all results linearly based on perfect == 1 and random == 0?

Certain metrics could still be arbitrarily worse than random (e.g. R2) which could lead to those metrics dominating everything else if we take the mean of the scaled score

I think that would be cool, but might be a bit of overhead to define random for unsupervised tasks... Also, I don't like R^2 for that reason... mentioned this issue to @giovp yesterday as well.

might be a bit of overhead to define random for unsupervised tasks

I think we can handle this. E.g. for dimensionality reduction, assign each point a 2D value drawn from a uniform distribution. It doesn't have to be a perfect random baseline but it gives some sense of what bad performance is.

That works for dimensionality reduction. The difficulty is more for data integration, where random is actually perfect batch integration and terrible bio conservation. For the batch removal metrics, this is tough. There maybe the baseline could be unintegrated data for the batch removal metrics, and random for the bio conservation metrics? Usually this is 0-1 already though by design.
Also, if we do this it would need to go into the READMEs...

Honestly I think random performance in data integration should be bad on both. For batch_integration_feature / embedding, this means scramble the features per batch. For batch_integration_graph, you scramble the graph edges within a batch.

Right, hence random connections within each batch. All connections are
within-batch, so no batch mixing.
On Thu, 15 Sept 2022, 12:42 pm MalteDLuecken, @.***>
wrote:

In my mind: Random connections -> random neighbors -> really good batch
mixing.
—
Reply to this email directly, view it on GitHub
https://github.com/openproblems-bio/openproblems/pull/566#issuecomment-1248352982,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA3DX46UK2WEL5W4RXBZULV6NGXNANCNFSM6AAAAAAQHYJ4BM
.
You are receiving this because you authored the thread.Message ID:
@.***>

In any case, this is a separate discussion. Happy for me to merge these changes?

Ah, I didn't read that properly. Good point. And I also didn't see you had addressed my comments ^^. Will check

A few small comments... and I do think the dimensionality reduction README still needs to be shortened. But we can do that in another PR.

authors: LuckyMD lazappi scottgigante scottgigante-immunai

repo: openproblems-bio/openproblems


text:
Recipes having a description should be enforced by tests

Part of our test process should be to load recipes all recipes on the classpath and ensure they have a description which matches our expectations (not blank, ends in a period).
rewrite-recipe-starter could come with a default test which does this. Creators of recipe modules can remove the test, but generating it will at least alert them of the expectation.

This has been completed

authors: sambsnyd tkvangorder

repo: openrewrite/rewrite


text:
rename fieldAccess when variable is renamed

Currently there is a compilation error, when a variable with field access is renamed.
With this pull request the variables with field access will be renamed as expected.

Very much appreciate the effort @Guliver !

authors: Guliver jkschneider


text:
Fixed a bug  about capacity unit

What this PR does / why we need it:

Using cmd vgs instead of vgdisplay.
Fixed a bug about capacity unit.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

Coverage decreased (-0.03%) to 43.919% when pulling 88c66fb1ff5bef206e70560672c41a2cb4dfdf2b on xxwjj:lvm_pool_capacity_bug into dd199c969f56b00a1d2d08be42b3be2ba949ef43 on opensds:development.

authors: coveralls xxwjj

repo: opensds/opensds


text:
Services APIs should have tenant ID in it

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:
/kind bug
/kind feature

What happened: Each APIs should have the tenant ID in the URL
What you expected to happen: tenant id in URL
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:

Orchestration(release/branch) version:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

/assing @kumarashit

Fixed

repo: opensds/orchestration


text:
Filenames with special characters interpreted as JSON?

I'm trying to open a file with the path:
/fcgi-bin/iipsrv.fcgi?DeepZoom=/data/images/EXxx-xxxx-x-CKXXX-[Boon's]-AV.svs.dzi
When I pass this in the viewer.open() function I get the following error:
osdCamicroscope.php?tissueId=test:1 Uncaught SyntaxError: Unexpected token / in JSON at position 0
    at Function.parse [as parseJSON] (<anonymous>)
    at Function.parseJSON (openseadragon.js:2264)
    at getTileSourceImplementation (openseadragon.js:8749)
    at $.Viewer.addTiledImage (openseadragon.js:8010)
    at doOne (openseadragon.js:7346)
    at $.Viewer.open (openseadragon.js:7351)

I think this is due to the match statement tileSource.match( /\s*[\{\[].*/ ) [Source]
Which returns a matched JSON for the file location I'm using. Thus it tries to parse this file location as a JSON leading to the error.

Fixed in: https://github.com/openseadragon/openseadragon/pull/1189

repo: openseadragon/openseadragon

license_source: bigquery


text:
Viewer tries to load zero-dimension images

This is with version 2.2.1. Here is my invocation code:
var viewer = OpenSeadragon({
    id: "openseadragon1",
    prefixUrl: "images/",
    showNavigator: true,
    tileSources: "http://localhost:8182/iiif/2/andromeda.jp2"
});

When I load this, OpenSeadragon hits the following IIIF URL:
http://localhost:8182/iiif/2/andromeda.jp2/full/2,/0/default.jpg
Because the image is very narrow in landscape orientation, the height, at a width of 2 pixels, rounds down to 0 pixels. Thus, there is nothing to return. (The source image is very large, FWIW.) My web service responds with an HTTP 400 Bad Request. After receiving this, OpenSeadragon terminates the whole tile-loading process, leaving the following message in my JavaScript console:
"Tile 0/0_0 failed to load: http://localhost:8182/iiif/2/andromeda.jp2/full/2,/0/default.jpg - error: Image load aborted"

Incidentally it seems that other unnecessary requests are made as well, for widths of 3, 5, 9...

I seem to be able to work around this by commenting out the return in the if (!image) conditional in onTileLoad() in tiledimage.js. I don't know if that's a good solution, though.

Interesting. I guess maybe the IIIFTileSource needs to be fixed to never round down all the way to zero? That seems like the cleanest solution. We wouldn't want to comment out the return in the conditional in onTileLoad as we don't want load errors to be ignored.
Any thoughts, @azaroth42 @jpstroop?
@adolski Regarding the additional "unnecessary" requests, it sounds like you don't want OSD to go all the way to the tip of the pyramid. OSD attempts to load the biggest tile that covers the entire image, so it should be doing the right thing, but I don't know enough about the IIIFTileSource to know if it's working properly with it. The code that handles this is here if you want to take a look:
https://github.com/openseadragon/openseadragon/blob/master/src/tiledimage.js#L870-L888
The places where it gets data from tiledImage.source are where it's interacting with the IIIFTileSource.
Another option, to short circuit the loading of lower level tiles entirely would be to set immediateRender: true in your options when creating your viewer.
At any rate, I think probably the best fix for OSD is to institute a minimum width and height of 1 pixel in the IIIFTileSource when picking a tile size to load. @adolski would you be up for looking into making that fix? @azaroth42 or @jpstroop might be able to provide some pointers as to where in https://github.com/openseadragon/openseadragon/blob/master/src/iiiftilesource.js to make the fix.

I think the fix would be to take the full height and width (from info.json) and then assuming floor() as the (worst case) rounding algorithm, check that the height of the image is not zero after halving the width.
However ... it should only try to load images according to the scaleFactors set in tiles. Have you checked your implementation, that it's not declaring a scaleFactor as being available that would generate a width of 2 pixels, @adolski?  Is the endpoint available for us to test fixes against?

Thanks, @iangilman -- immediateRender: true is working well for me as a workaround.
@azaroth Here is an endpoint: http://iiif.library.illinois.edu/test/iiif/2/andromeda-pyramidal-tiled.tif/info.json
(Note that my resample filter is incapable of outputting less than 3 pixels on a side, so that is the smallest dimension it will return.)
Whether or not the scales come from the scaleFactors, I'd actually go further and set the minimum dimension quite a bit larger than zero, like say, 1/4 of the initial dimensions, unless there is some reason I'm not understanding for making requests for such tiny images as e.g. 5x2, 9x3, etc. (If the expectation is that tiny images can be delivered faster in order to give the client something [anything!] to chew on while they wait for some more usable levels, it seems to me that they also create more work to do on both the server and client.)

The info.json response says that you support tiles down to 1/256th of the full resolution, which is the 272,87 image.  So why OSD is then going another factor of 100 smaller and asking for 2,0 seems like a bug in the IIIF TileSource to me.

That's how it's designed to work right now. Scale factors only apply to tiles:
  "tiles": [
    {"width" : 512, "scaleFactors" : [1,2,4,8,16]}
  ]

So when you're dealing with smaller images whose long dimension is less than that of the tiles, the region is full and scaled sizes are calculated and requested by OSd. The most semantically correct fix, from IIIF's perspective, would be to make OSd respect the optional sizes list in info.json:
"sizes" : [
  { "width" : 41, "height": 57 },
  { "width" : 82, "height": 113 },
  { "width" : 163, "height": 225 },
  { "width" : 326, "height": 450 },
  { "width" : 651, "height": 900 },
  { "width" : 1301, "height": 1800 },
  { "width" : 2602, "height": 3600 },
  { "width" : 5204, "height": 7200 }
]

I like that idea, though it might be a bit of a burden for less complete IIIF image server implementations.

We have objects that look like this:
{
@context: "http://iiif.io/api/image/2/context.json",
@id: "https://stacks.stanford.edu/image/iiif/example",
protocol: "http://iiif.io/api/image",
width: 31343,
height: 47104,
sizes: [
{
width: 122,
height: 184
},
{
width: 245,
height: 368
},
{
width: 490,
height: 736
},
{
width: 979,
height: 1472
},
{
width: 1959,
height: 2944
},
{
width: 3918,
height: 5888
},
{
width: 7836,
height: 11776
},
{
width: 15672,
height: 23552
},
{
width: 31343,
height: 47104
}
],
tiles: [
{
width: 980,
height: 1472,
scaleFactors: [
1,
2,
4,
8,
16,
32,
64,
128,
256
]
}
],
profile: "http://iiif.io/api/image/2/level2"
}

which cause OSD to request https://stacks.stanford.edu/image/iiif/example/full/1,/0/default.jpg

Plugging some of these failing images into the test/demo/iiif.html test page, it seems they don't fail or cause this blank canvas effect when passed as tilesource urls on initialisation. It might only be the case where viewer.addTiledImage() is being used. I will test that shortly.

I think there is an additional question of why this smallest tile is being requested the viewer initialisation. Perhaps there is an opportunity to short-circuit this and only get the full image at the appropriate size for immediate display. I see immediateRender: true(definition) is an option from the comment above, but for some reason this feels like it should be the default (though to be honest I don't understand the other rendering constraints or optimisations).

Interesting... it shouldn't be asking for the smallest tile. It should be asking for the largest tile that covers the whole image. This is different from immediateRender, which loads the tiles for the current level, which gives a patchwork effect (you can see holes in the image where tiles haven't loaded yet). The default behavior of loading the largest full image tile gives you the entire image immediately in blurry form and then it gets progressively sharper from there, which is a better UX. It shouldn't be asking for tiny tiles though.

authors: adolski aeschylus azaroth42 iangilman jpstroop


text:
Add an AJAX concurrency limiter for tile source requests

This is a proof-of-concept for a new viewer option which attempts to address #2241
For very large sessions it's easy to hit a browser's concurrent network requests limit, causing it to cancel a bunch of tile source ajax requests, it would be nice to have an option similar to imageLoaderLimit to avoid this case.
I'm opening this as a draft now; I added this feature without much care for style or the existing project structure, but thought it would be a good jumping off point for discussion about how to properly implement this!
This is a quick-and-dirty addition based on this gist: https://gist.github.com/Terrance/158a4c436baa64c4324803467844b00f

I think renaming the feature to be more generic makes a lot of sense! Something just like ajaxConcurrencyLimit, or maxConcurrentAjaxRequests?
Yeah I'm also not sure what the best way to set this value would be, having it be a member of the viewer would be nice because it lines up with all of the existing configuration options. But having it at the viewer level could still get us into trouble with the browser limits in the case that there are multiple viewers on the same page; from the perspective of making OSD as a whole not run up against browser limits (regardless of how many viewers are instantiated) it does make sense to make a global option, but we might have to invent a new way to set it 🤔
I'm willing to put in the work to implement it if we decide on a good way forward! Although this is my first time approaching the source so I'm not too familiar with the style or structure yet

I agree that having it global makes sense, and maybe it's about time we introduce global options somehow.
I think I like moving towards having functions for getting and setting properties, so maybe it would be something like OpenSeadragon.setAjaxLoaderLimit()? As for the name, the "loaderLimit" language already exists, so it might be good to stick with that. Your other name suggestions seem reasonable though.
You seem to be doing a fine job of fitting into the source style/structure!

@ChiliJohnson Are you interested in continuing work on this?

Hello, yes I'm still interested! Sorry I was out on vacation for a couple weeks there but am back now and able to push this forward.
I've been running this branch in production for the last few weeks without issue (churning through hundreds of thousands of AJAX requests) so I've at least convinced myself that the core logic works; now I just have to refactor it into something prettier

@ChiliJohnson Wonderful! There's no rush... Just making sure you haven't walked away :)

I've just pushed some updates based on the feedback! Still not sure if this is exactly the right approach to global settings but it's at least less of a draft now so I converted this to a real PR

authors: ChiliJohnson iangilman


text:
Flexible support for drawers to modify cache

Fixing reference to the webgl drawer fallback strategy.

I noticed one problem here: current cache design does not support well this implementation as I thought, the reason is that the data on the internal cache is getting destroyed once the texture loading handler exits, and there is no way now to keep the data. @pearcetm what was the second idea of having the support for custom conversions on the drawer level?
I would still prefer for drawers not to touch types and rely on the system if possible. It might create problems for interacting plugins, or other optimizations - I was thinking about experimenting with threads for data processing, this would bypass such logics..
Interesting that current tests did not find out it was broken 🥲

I fixed a typo when accessing the 'cpu' data variable - it was being accessed wrong. And then I realized it does not work as intended, because conversion routine frees the data and only keeps the last item. E.g. I cannot keep both the texture and the original context object.  So the data to be used in the fallback canvas drawer is already destroyed. That's why the test says 'accessing data with width/height of 0'..
So it seems the fallback drawer usage is not well supported after all. I would like to discuss what is the best design. We could try to change the API to support also drawer re-use. We could allow drawers to further modify the type they need to use before rendering. We could support general fallback  logics where regardless of a drawer type, the library catches a specific exception in a drawer and employs replacement with a fallback type automatically

Thinking specifically about WebGLDrawer and falling back to CanvasDrawer, here's how I see it.

WebGLDrawer should indicate that it accepts data in certain formats, like context2d currently, or with a bit of refactoring we could accept anything that texImage2D takes: https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/texImage2D#pixels

The data in this format should stay in the cache for reuse.

The WebGLDrawer should create the texture and upload data to the GPU within its own context.

It doesn't make sense to "cache" a glTexture since the texture is specific to one webgl context, and can't be reused outside of this context.
Since a glTexture is purely internal to a single drawer, exposing it on the cache item doesn't make sense. It can be kept in a Map within the drawer that maps the cache item to the texture to tell the drawer which texture goes in which location when drawing.

To allow the cache to manage lifecycle of the data, we should have an API on the cache item for the drawer to register a cleanup function, such that when the cache item is unloaded, the texture is appropriately cleaned up.
When the drawer is destroyed, any remaining textures should be cleaned up at that time. The cleanup function should either be unregistered from the cache, or should gracefully handle already-destroyed textures.

Destroying a drawer should not affect what is or is not in the cache.
The internal Map will be useful here.

With this strategy, data in "CPU" format will always be available for CanvasDrawer, whether that is through automatic fallback or changing drawer entirely, or even for changing to a new WebGLDrawer for whatever reason.

For the case where data in the cache is modified/invalidated (which you know I think is a bad idea, but we don't have to rehash that), the cache item should call the cleanup function which would unload the old texture and remove it from the drawer's internal Map. Then, the next time the data is supposed to be drawn, the drawer's normal mechanism would automatically know to create and upload a new texture to the GPU and register a new cleanup function.

authors: Aiosa pearcetm


text:
[Timestamp Pinning in Remote Store] Remote translog garbage collection changes

Is your feature request related to a problem? Please describe
As part of shallow snapshot optimisations, we are introducing timestamp based implicit locking support. This feature will be called Timestamp Pinning in remote backed storage. When a timestamp is pinned, remote store garbage collectors will skip deleting data corresponding to the timestamp.
Describe the solution you'd like
As part of this solution, we need following changes to segment store garbage collection

Each data node keeps an in-memory data structure remote_store_pinned_timestamps (this is updated regularly as part of https://github.com/opensearch-project/OpenSearch/issues/15062)
If the last update time of remote_store_pinned_timestamps is > X mins, skip garbage collection.
Fetch metadata files in sorted order (which generally is reverse-chronological order but not always)
If timestamp of a metadata file md1 is > pinned_timestamp_a and the timestamp of next metadata file md2 <= pinned_timestamp_a add md2 to pinned_metadata_files
Skip deletion of metadata files in pinned_metadata_files and corresponding data files

Related component
Storage:Remote

We also need to handle garbage collection on index delete.

repo: opensearch-project/OpenSearch


text:
[Feature/Extensions] getNamedWriteables - Add support for extensions

Add extension framework support for getNamedWriteables extension point.

Design for default extension point support for getNamedWriteables.
Extension Point Purpose :

When plugins override getNamedWriteables() with their own implementation, they are able to register custom parsers for Named Objects that implement the Writeable class within the NamedWriteableRegistry. Similar to the NamedXContent registry, the Named Writeable Registry entries consist of a category class, a unique name associated with the writeable object, and a parser. The category class and name is used to validate that the parser has been registered during OpenSearch bootstrap.
The main difference between the two default extension points is that the NamedWriteableRegistry is used to deserialize named objects between nodes, allowing nodes to transfer and understand objects passed across the wire using the OpenSearch inter-node communication protocol outlined by StreamInput/StreamOutput, while the NamedXContentRegistry is used to parse xcontent (ie json, yaml, cbor, smile formatted data) that is sent across the wire.

Previous workflow:

Traditionally installed plugins override the getNamedWriteableRegistry() extension point and return a collection of NamedWriteableRegistry.Entry's
During OpenSearch bootstrap within Node.java, Named Writeable Registries from various modules (ie: search, index, network, cluster) and all installed plugins are consolidated into a central NamedWriteableRegistry.
This registry is then used to create components for all installed plugins, indices service, network module, discovery module, and a module builder used for dependency injection.

Extensibility workflow:

In order to enable independent plugins to register xcontent registry entries across the wire using the inter-node communication protocol, it is necessary for these xcontent entry category classes to be first registered within the NamedWriteableRegistry
This results in a conundrum since both extension point architectures are quite similar and thus the category class of the NamedWriteableRegistry is not able to be deserialized by opensearch after it is sent across the wire
In order to circumvent this issue, modifications to the NamedWriteableRegistry entry category class field must be made such that only the string form of the class name is sent across the wire and registered in the central named writeable registry within Node.java
This is possible since the named writeable registry only uses the class name for registration verification, while the named xcontent registry verification also uses the category class of the entry to cast the object sent across the wire, requiring that particular entry category class to already be class loaded within the jvm that opensearch is running on
after all installed plugin writeable registries are collected, extensions orchestrator will send out requests to all extensions that have been previously discovered for any entries they want to register
Upon receiving the request, the independent plugins would return a response consisting of all the entries that it wants to register, however, instead of providing the custom parser within the entry, the reader will be replaced with a callback method to send a request from opensearch to the extension for object deserialization
The work done to handle object deserialization will be done by the extension in order to maintain the separation of concerns between it and OpenSearch. In the case that the extension goes down, the entries corresponding to the extension must then be deregistered from the NamedWriteableRegistry
Once all NamedWriteable registry entries from the extensions have been collected, the registry will then be passed onto various services that are created during bootstrap
When xcontent registry entries are then collected, the extension entries sent across the wire can then be identified by the named writeable registry and then parsed into the correct object

Love the details @joshpalis . TY

Thanks @joshpalis for detailed notes.
To give out a summary, we are trying to run plugins as a standalone process.
getNamedWriteables adds support to understand messages transported by plugins.
Unfortunately when we'd like to run them as extensions, we do not have the class definitions to deserialize the messages (Because OpenSearch wouldn't not understand it or has the capability to decrypt it).
@nknize @rmuir  What are your thoughts on solving this problem of running extensions independently yet still enable OpenSearch to decrypt it?

#4067

authors: joshpalis minalsha saratvemulapalli


text:
[Performance] Potential regression in 1.0.0-Beta1

Describe the issue
Benchmarking has been set up and the following numbers captured today:

Product/ Version
Architecture
Description
Instance Type
Workload Details
Index Latency (ms) - p50
Index Latency (ms) - p90
Index Latency (ms) - p99
Index Latency (ms) - p100
Index Throughput (docs/s) - p0
Index Throughput (docs/s) - p50
Index Throughput (docs/s) - p100
Ops Count
Op Error Count
Error Rate
Query Latency (ms) - p50
Query Latency (ms) - p90
Query Latency (ms) - p99
Query Latency (ms) - p100
Query Throughput (docs/s) - p0
Query Throughput (docs/s) - p50
Query Throughput (docs/s) - p100
Ops Count
Op Error Count
Error Rate
CPU (%) - p50
CPU (%) - p90
CPU (%) - p99
CPU (%) - p100
Memory (%) - p50
Memory (%) - p90
Memory (%) - p99
Memory (%) - p100
GC (ms) - Old
GC (ms) - Young

ODFE 1.13.2
X64
With security
m5.xlarge
nyc_taxis/ 2 warmupIterations/ 3 testIterations
2,371.4
3,077.4
4,294.3
9,076.4
32,124.7
33,120.6
38,174.6
46,871
-
-
175.2
203.5
216.6
273.3
1.748
1.753
1.767
1,510
-
-
97
97
98
98
50
64
67
70
-
1,678,812

OpenSearch-1.0.0-beta1
X64
With security
m5.xlarge
nyc_taxis/ 2 warmupIterations/ 3 testIterations
3,198.3
4,432.5
7,910.1
10,011.5
22,784.5
23,332.2
27,295.6
47,649
58
0.10%
261
305.1
381.9
411.6
2.009
2.015
2.031
906
-
-
98
98
98
98
51
64
68
70
-
1,794,217

ODFE 1.13.2
X64
Without security
m5.xlarge
nyc_taxis/ 2 warmupIterations/ 3 testIterations
1,915.7
2,592.7
3,620.7
6,655.2
39,218.4
40,990.2
47,736.1
46,193
-
-
266.5
280.5
289.5
294
1.747
1.751
1.764
1,510
-
-
98
98
99
100
48
62
67
75
-
1,281,143

OpenSearch-1.0.0-beta1
X64
Without security
m5.xlarge
nyc_taxis/ 2 warmupIterations/ 3 testIterations
2,986
4,170.5
7,673.3
13,117.2
24,271.4
25,024.7
29,637.7
47,475
-
-
316.7
335.4
424.2
466.2
2.002
2.017
2.033
906
-
-
99
99
99
99
49
62
67
69
-
1,543,353

Initial results indicate a possible regression. Further root cause analysis, from benchmark server configuration to investigating dependency performance, needs to be done to first confirm these as valid performance regression indicators. If the behavior is confirmed the regression will be tracked down and a patch will be opened and merged in advance of the Beta 2 release. This issue is to track the progress and transparently communicate the update as it becomes available.

Tl;Dr  The original large performance degradation we saw was caused by an issue in the performance tool.  Now that we've cleared that out, we're still seeing a (smaller) performance degradation, but we've got a smoking gun that the version changes are causing the issue.   So we believe this is ultimately related to #693, and should be resolved with that issue.
We've been digging into this, and here's what we've found:
The results above are from an internal tool based on Rally.  We had previously run Rally against OpenSearch Alpha, so in order to get an Apples to Apples comparison, we wanted to rerun Rally against OpenSearch Beta.   We did that and saw similar results:
osbeta1_0_0_vs_es_&_10_2_nyc_taxi.pdf
BUT!  (and the plot thickens!)
Then we figure out what was going on with Rally and the internal tool.  Rally picks the branch of the track based on the version reported by the cluster, for ODFE that’s ‘7.10.2’, for OpenSearch that’s ‘1.0.0-beta1’.  This means that Rally used the 5.0 branch of nyc-taxis when running OpenSearch instead of the 7.1 branch that it used for ODFE.  So from out POV, that was (relatively) good news since it meant that these results have no meaning, since the track and operations were different.
BUT!  (more plot!)
Once we'd figure out a way to get Rally to use the same tracks we ran again... and we still seeing some degradation.  But not nearly as bad.

Product/ Version | Architecture | Description | Instance Type | Workload Details | Index | Query | CPU (%) |   |   |   | Memory (%) |   |   |   | GC (ms) |
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
Latency (ms) | Throughput (docs/s) | Operation Counts | Latency (ms) | Throughput (docs/s) |   |   | Operation Counts |   |   |   |   |   |   |   |   |   |   |   |
p50 | p90 | p99 | p100 | p0 | p50 | p100 | Ops Count | Op Error Count | Error Rate | p50 | p90 | p99 | p100 | p0 | p50 | p100 | Ops Count | Op Error Count | Error Rate | p50 | p90 | p99 | p100 | p50 | p90 | p99 | p100 | Old | Young
ODFE 1.13.2 | X64 | With security | m5.xlarge | nyc_taxis/ 2 warmupIterations/ 3 testIterations | 2,371.4 | 3,077.4 | 4,294.3 | 9,076.4 | 32,124.7 | 33,120.6 | 38,174.6 | 46,871 | - | - | 175.2 | 203.5 | 216.6 | 273.3 | 1.748 | 1.753 | 1.767 | 1,510 | - | - | 97 | 97 | 98 | 98 | 50 | 64 | 67 | 70 | - | 1,678,812
OpenSearch-1.0.0-beta1 | X64 | With security | m5.xlarge | nyc_taxis/ 2 warmupIterations/ 3 testIterations | 2,530.2 | 3,251.7 | 4,833.3 | 7,956.3 | 30,336.8 | 31,283.1 | 36,216.9 | 47,046 | - | - | 178.3 | 206.9 | 221.9 | 225.2 | 1.749 | 1.754 | 1.767 | 1,510 | - | - | 97 | 97 | 98 | 98 | 53.5 | 64 | 68 | 70 | - | 1,781,648
Comparison |   |   |   |   | -6.70% | -5.66% | -12.55% | 12.34% | -5.57% | 5.55% | 5.13% |   |   |   | -1.77% | -1.67% | -2.45% | 17.60% | 0.06% | 0.06% | 0.00% |   |   |   |   |   |   |   |   |   |   |   |   |
ODFE 1.13.2 | X64 | Without security | m5.xlarge | nyc_taxis/ 2 warmupIterations/ 3 testIterations | 1,915.7 | 2,592.7 | 3,620.7 | 6,655.2 | 39,218.4 | 40,990.2 | 47,736.1 | 46,193 | - | - | 266.5 | 280.5 | 289.5 | 294 | 1.747 | 1.751 | 1.764 | 1,510 | - | - | 98 | 98 | 99 | 100 | 48 | 62 | 67 | 75 | - | 1,281,143
OpenSearch-1.0.0-beta1 | X64 | Without security | m5.xlarge | nyc_taxis/ 2 warmupIterations/ 3 testIterations | 2,252 | 2,984 | 4,472.3 | 7,873 | 33,929.4 | 35,021.5 | 41,034.1 | 46,697 | - | - | 204.4 | 217.9 | 226.7 | 235 | 1.749 | 1.754 | 1.768 | 1,510 | - | - | 98 | 98 | 99 | 99 | 48 | 62 | 66 | 68 | - | 1,328,941
Comparison |   |   |   |   | -17.55% | -15.09% | -23.52% | -18.30% | -13.49% | -14.56% | -14.04% |   |   |   | 23.30% | 22.32% | 21.69% | 20.07% | 0.11% | 0.17% | 0.23% |   |   |   |   |   |   |   |   |   |   |   |   |

BUT! (still more!)
So if we believe (and we do) that the original performance impact was because we was using different tracks, where was the new issue coming from?  @harold-wang has been digging in, running tests in rally with different builds, removing different PRs.  We have definitely gotten a smoking gun that the issue started with our version changes.   We have a couple of theories on what's going on that's causing the degradation, but I'll let @harold-wang post more about it.
Thanks very much to @harold-wang, @cmanning09, @nknize  and Phil Treddenick for digging in.

Initial PR has been opened to fix BWC for legacy transport clients.

Performance result is gone after I change the version id from 1.0.0 to 7.10.2 in ESRally test.

Min Throughput
Mean Throughput
Median Throughput
Max Throughput
50th percentile latency
90th percentile latency
99th percentile latency
99.9th percentile latency
99.99th percentile latency
100th percentile latency
percentile service time
percentile service time
percentile service time
percentile service time
percentile service time
percentile service time

OpenSearch100_main_vs_elasticsearch_7_10_3_performanceIssue_fakeversionfix.pdf

Re-open this issue, we are running the same test by Mensor against the cluster with opensearch BWC enabled. will update.
https://prod.mensor.searchservices.aws.dev/test/26010c94-647f-466b-a386-e23c370f1345
https://search-mensor-metrics-jz77bwnyqqiekf7g2fmkfmfncu.eu-west-1.es.amazonaws.com/_plugin/kibana/app/kibana?security_tenant=global#/dashboard/2366e970-419d-11eb-ac43-fb82b7859827?_g=(filters:!(),refreshInterval:(display:'1 minute',pause:!f,value:60000),time:(from:now-24h,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!f),panels:!((embeddableConfig:(title:''),gridData:(h:15,i:'5ef0a36f-ec97-4d52-96d1-6ccee5c2adce',w:48,x:0,y:0),id:ccb1ec30-419f-11eb-9fed-f3dfeb55b7ea,panelIndex:'5ef0a36f-ec97-4d52-96d1-6ccee5c2adce',type:visualization,version:'7.7.0'),(embeddableConfig:(title:''),gridData:(h:8,i:'70420d98-9111-4f74-ab57-48dad5a4200a',w:32,x:0,y:15),id:'27b72960-41a0-11eb-94ff-69ed4c039cfc',panelIndex:'70420d98-9111-4f74-ab57-48dad5a4200a',type:visualization,version:'7.7.0'),(embeddableConfig:(title:''),gridData:(h:8,i:'1b441e91-d728-4344-b5d5-e48d7b482328',w:16,x:32,y:15),id:'9df6dd90-43c2-11eb-94ff-69ed4c039cfc',panelIndex:'1b441e91-d728-4344-b5d5-e48d7b482328',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:15,i:b82abd45-c77e-42c2-a138-e049aed07172,w:48,x:0,y:23),id:bf5343d0-5528-11eb-b91d-d9b60acd3a47,panelIndex:b82abd45-c77e-42c2-a138-e049aed07172,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:fa7c7977-37a2-4bf7-b9f4-73711e5243cd,w:48,x:0,y:38),id:d07659f0-416d-11eb-9fed-f3dfeb55b7ea,panelIndex:fa7c7977-37a2-4bf7-b9f4-73711e5243cd,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'9df12f96-8cb2-4329-837b-33dbc31e2d14',w:48,x:0,y:49),id:f2231c00-416d-11eb-9fed-f3dfeb55b7ea,panelIndex:'9df12f96-8cb2-4329-837b-33dbc31e2d14',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'1d998eca-9033-48da-b383-944cd9c9f0f7',w:48,x:0,y:60),id:'2061ab60-5a99-11eb-94ff-69ed4c039cfc',panelIndex:'1d998eca-9033-48da-b383-944cd9c9f0f7',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:b59878b5-418e-4e62-854a-182d3f527ef9,w:24,x:0,y:71),id:'5eeadb70-416e-11eb-9fed-f3dfeb55b7ea',panelIndex:b59878b5-418e-4e62-854a-182d3f527ef9,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'337414c5-61ce-496f-8f4d-b43f40aaffbb',w:24,x:24,y:71),id:a4b02f70-416e-11eb-abea-1752399bd867,panelIndex:'337414c5-61ce-496f-8f4d-b43f40aaffbb',type:visualization,version:'7.7.0'),(embeddableConfig:(title:''),gridData:(h:7,i:'8d58abfe-ff93-431c-923b-0db04ea9ec3a',w:48,x:0,y:82),id:c8324220-5aa6-11eb-abea-1752399bd867,panelIndex:'8d58abfe-ff93-431c-923b-0db04ea9ec3a',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:fd9ce5e4-495c-4ed3-b0e4-f5e40c234b14,w:24,x:0,y:89),id:'9b6e1000-4176-11eb-abea-1752399bd867',panelIndex:fd9ce5e4-495c-4ed3-b0e4-f5e40c234b14,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'4870dc1c-5da5-4a3e-8d1c-e43a23c54811',w:24,x:24,y:89),id:e7cb8a60-4174-11eb-b91d-d9b60acd3a47,panelIndex:'4870dc1c-5da5-4a3e-8d1c-e43a23c54811',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:cb980b53-f209-487e-839e-d39ffca937a6,w:24,x:0,y:100),id:'6acd06e0-4176-11eb-b91d-d9b60acd3a47',panelIndex:cb980b53-f209-487e-839e-d39ffca937a6,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'76f0ec10-627b-4ace-9eb7-9f02bf05d8cc',w:24,x:24,y:100),id:'7f9ac2c0-4175-11eb-94ff-69ed4c039cfc',panelIndex:'76f0ec10-627b-4ace-9eb7-9f02bf05d8cc',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'4d7508d5-245d-482f-ab6b-df44a3aa2f11',w:24,x:0,y:111),id:abd0a990-4175-11eb-9fed-f3dfeb55b7ea,panelIndex:'4d7508d5-245d-482f-ab6b-df44a3aa2f11',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:b4133175-32ca-4703-9824-382c6d2b8758,w:24,x:24,y:111),id:'1530acf0-4176-11eb-94ff-69ed4c039cfc',panelIndex:b4133175-32ca-4703-9824-382c6d2b8758,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:ee172f65-1d1a-4e43-b391-3b38b0d74955,w:24,x:0,y:122),id:'3f86d770-419b-11eb-b91d-d9b60acd3a47',panelIndex:ee172f65-1d1a-4e43-b391-3b38b0d74955,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:f534741a-b7cc-4454-ac81-41e3a2aa4f76,w:24,x:24,y:122),id:'476cef80-4176-11eb-9fed-f3dfeb55b7ea',panelIndex:f534741a-b7cc-4454-ac81-41e3a2aa4f76,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'6fd55884-b028-4e27-b637-a7daab98848e',w:24,x:0,y:133),id:a76bac80-419b-11eb-b91d-d9b60acd3a47,panelIndex:'6fd55884-b028-4e27-b637-a7daab98848e',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'1efa0a4f-96fe-4fab-8d88-86897bb52851',w:24,x:24,y:133),id:'64a91130-419b-11eb-ac43-fb82b7859827',panelIndex:'1efa0a4f-96fe-4fab-8d88-86897bb52851',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'19b7c63b-ff40-4de4-99ce-7a1f42aef388',w:24,x:0,y:144),id:d7e85a30-419a-11eb-b91d-d9b60acd3a47,panelIndex:'19b7c63b-ff40-4de4-99ce-7a1f42aef388',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:cfc5116f-a02b-4e29-8e96-b31c95cdc52b,w:24,x:24,y:144),id:'16eed7e0-419b-11eb-94ff-69ed4c039cfc',panelIndex:cfc5116f-a02b-4e29-8e96-b31c95cdc52b,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'1d9b208e-635d-4ef3-8dff-589d0a8094e8',w:24,x:0,y:155),id:'67fdf570-4183-11eb-ac43-fb82b7859827',panelIndex:'1d9b208e-635d-4ef3-8dff-589d0a8094e8',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:f9feeb80-8600-4788-a9d3-5fbc1bd77a40,w:24,x:24,y:155),id:'83bf70f0-419b-11eb-abea-1752399bd867',panelIndex:f9feeb80-8600-4788-a9d3-5fbc1bd77a40,type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'4b0e744c-d5da-4fcc-9a69-31a23e196757',w:24,x:0,y:166),id:'85cb1f40-4176-11eb-abea-1752399bd867',panelIndex:'4b0e744c-d5da-4fcc-9a69-31a23e196757',type:visualization,version:'7.7.0'),(embeddableConfig:(),gridData:(h:11,i:'9aa13877-a0e3-4f39-a582-b356b75014ce',w:24,x:24,y:166),id:c7a68f10-419b-11eb-94ff-69ed4c039cfc,panelIndex:'9aa13877-a0e3-4f39-a582-b356b75014ce',type:visualization,version:'7.7.0')),query:(language:kuery,query:%2226010c94-647f-466b-a386-e23c370f1345%22),timeRestore:!f,title:'Mensor%20Test%20Run',viewMode:view)

Summary:

OpenSearch shows higher CPU usage in P50, P90 and P99 (23% to 25%)
OpenSearch shows lower memory usage(-15% to -25%)
Query throughput is almost same
OpenSearch has 16% higher index throughput
OpenSearch has 50% higher Query latency in P50 and P99, P99, P100

@harold-wang what was the resolution here?

I think this was resolved as a versioning discrepancy due to the versioning framework using old versions of elasticsearch? bumping to v.1.2.0 but I think this can be closed?

authors: CEHENKLE dblock harold-wang nknize


text:
Fix flakyness in RemoteStoreRefreshListenerTests

Description
Fix flakyness in RemoteStoreRefreshListenerTests -> testRefreshSuccessOnThirdAttemptAttempt
Related Issues
Resolves #8549
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

I will need to rebase after #8547 is merged.

I will need to rebase after #8547 is merged.

https://github.com/opensearch-project/OpenSearch/pull/8547/ is merged. Please rebase.

Re-triggering precommit


text:
Fix event analytics field blank margin

Signed-off-by: Joshua Li joshuali925@gmail.com
Description
closes https://github.com/opensearch-project/dashboards-observability/issues/165

The style was added by commit db674583bf026df783dad2553699ea432259354c. I checked doesn't seem to break other places but not sure why it was added
Issues Resolved
#165
Check List

[ ] New functionality includes testing.

[ ] All tests pass, including unit test, integration test and doctest

[ ] New functionality has been documented.

[ ] New functionality has javadoc added
[ ] New functionality has user manual doc added

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

LGTM - can you backport this to 2.x/1.x - not sure if main is the right branch for this

ported to 2.x. 1.x didn't have problem since the commit introduced it was added at 2.2.0


text:
[FEATURE] Sort Index Dropdown and Implement ComboBox

Description
Describe what this change achieves.
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Saw this on #342. Could we try implementing the DataSourceSelectable component instead?
cc: @ashwin-pc in case more context is needed


text:
[FEATURE] Enable test with-security for geospatial in opensearch-build

Is your feature request related to a problem?
Geospatial plugin is not enabled for test with-security in opensearch-build
What solution would you like?
Enable test with-security for Geospatial  in opensearch-build when opensearch-2.4.0-test.yml is ready
See an example for opensearch-2.3.0-test.yml
Do you have any additional context?
Support for test with-security in Geospatial was added recently. https://github.com/opensearch-project/geospatial/pull/138
During 2.4 release, we need to enable it in opensearch-build package.

@heemin32 can this be closed?

Not yet. opensearch-2.4.0-test.yml is not ready.
@peterzhuamazon when will the file be ready?

Not yet. opensearch-2.4.0-test.yml is not there yet. @peterzhuamazon when will the file be ready?

I will add this next week as our build just get stabled a bit. Not fully ready.

It is enabled. https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.4.0/opensearch-2.4.0-test.yml#L102

authors: VijayanB heemin32 peterzhuamazon

repo: opensearch-project/geospatial


text:
[BUG] testReDeployModel

What is the bug?
We are trying to run the test org.opensearch.ml.rest.RestMLDeployModelActionIT > testReDeployModel , which uses the Dlj pytorch native binaries. Djl usually downloads the pytorch binaries from its repository, but it can be superseded to use our own version of the pytorch binaries using environment variables as described here (https://github.com/deepjavalibrary/djl/tree/master/engines/pytorch/pytorch-engine#load-your-own-pytorch-native-library)
We have tested this mechanism with the Djl library tests and they are being recognized correctly.
When run from testReDeployModel, the code is apparently not able to read the three environment variables even when they are set for the shell. Is there anything in the ml-commons code that might be restricting this?
How can one reproduce the bug?
e.g.:
export PYTORCH_LIBRARY_PATH=$HOME/conda/lib/python3.9/site-packages/torch/lib
export PYTORCH_VERSION=1.13.1
export PYTORCH_FLAVOR=cpu
./gradlew integTest -Dtests.heap.size=4096m --info
What is the expected behavior?
The pytorch native binaries should be picked as specified in the environment variables
What is your host/environment?

OS: [e.g. iOS] RHEL 8.7 docker
Version [e.g. 22] 2.11.1.0
Plugins

cc: @seth-priya

Looks like the env var gets picked up in this method. Any chance the OpenSearch security manager has problems with reading env vars?

@HenryL27 Its read as null by that method, but I am not sure about the SecurityException. I tried
permission java.lang.RuntimePermission "getenv.*";
in the .policy file, but it still didn't work.

What logs do you get near the 'downloading pytorch' bit? I also notice your OS is RHEL docker - are you running in a docker container?

repo: opensearch-project/ml-commons


text:
use small model to run integ test

Signed-off-by: Yaliang Wu ylwu@amazon.com
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #509 (f93f0c0) into 2.x (711031d) will increase coverage by 7.91%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                2.x     #509      +/-   ##
============================================
+ Coverage     72.38%   80.30%   +7.91%
- Complexity      838      914      +76
============================================
  Files            92       92
  Lines          3462     3462
  Branches        314      314
============================================
+ Hits           2506     2780     +274
+ Misses          837      540     -297
- Partials        119      142      +23

Flag
Coverage Δ

ml-commons
80.30% <ø> (+7.91%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...va/org/opensearch/ml/model/MLModelCacheHelper.java
100.00% <0.00%> (+0.90%)
:arrow_up:

...earch/ml/action/load/TransportLoadModelAction.java
90.29% <0.00%> (+0.97%)
:arrow_up:

...ain/java/org/opensearch/ml/task/MLTaskManager.java
62.82% <0.00%> (+2.61%)
:arrow_up:

...java/org/opensearch/ml/profile/MLProfileInput.java
95.77% <0.00%> (+4.22%)
:arrow_up:

...va/org/opensearch/ml/task/MLPredictTaskRunner.java
86.33% <0.00%> (+4.31%)
:arrow_up:

...org/opensearch/ml/cluster/DiscoveryNodeHelper.java
53.03% <0.00%> (+9.09%)
:arrow_up:

...ch/ml/action/profile/MLProfileTransportAction.java
93.75% <0.00%> (+12.50%)
:arrow_up:

...n/java/org/opensearch/ml/model/MLModelManager.java
74.55% <0.00%> (+12.72%)
:arrow_up:

...g/opensearch/ml/profile/MLPredictRequestStats.java
77.27% <0.00%> (+20.45%)
:arrow_up:

...java/org/opensearch/ml/profile/MLModelProfile.java
89.74% <0.00%> (+35.89%)
:arrow_up:

... and 5 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

authors: codecov-commenter ylwu-amzn


text:
[BUG] Text chunking max_chunk_limit error

What is the bug?
Text chunking processor will produce the following exception if the number of produced chunks exceed max_chunk_limit. This would lead to data loss.
The number of chunks produced by text_chunking processor has exceeded the allowed maximum of [100]

How can one reproduce the bug?
Use text chunking processor and produce more chunks thank max_chunk_limit.
What is the expected behavior?
There should be no data loss. Suppose parameter max_chunk_limit takes the default value 100. if the generated chunks exceed 100, we only keep the first 100 passages into the results.
What is your host/environment?
Linux and Mac.
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.

Raised a PR to fix the bug: https://github.com/opensearch-project/neural-search/pull/717

Please elaborate the truncating logic and overall background description.

Expected behavior when we are perform chunking with delimiter algorithm with " " as delimiter and ["a b c", "", "", "d e f"] as the input.

When max_chunk_limit >= 6 or max_chunk_limit == -1, the output should be ["a ", "b ", "c", "d ", "e ", "f"].
When max_chunk_limit <= 2 and max_chunk_limit != -1, the output should be ["a b c", "d e f"].
When max_chunk_limit = 5, the output should be ["a ", "b ", "c", "d ", "e f"]
When max_chunk_limit = 4, the output should be ["a ", "b ", "c", "d e f"]
When max_chunk_limit = 3, the output should be ["a ", "b c", "d e f"].

@yuye-aws if PR has fixed the issue can we close it?

Sure. I am closing the issue.

authors: martin-gaievski model-collapse yuye-aws

repo: opensearch-project/neural-search


text:
[BUG] Corrupt benchmark data

Describe the bug
I followed https://github.com/opensearch-project/OpenSearch-Benchmark/blob/main/DEVELOPER_GUIDE.md to run the benchmark on Amazon Linux Developer Desktop and it failed with following error (logs attached):
[ERROR] Cannot execute_test. Error in task executor
	Download of [/home/ankikala/.benchmark/benchmarks/data/eventdata/eventdata.json.bz2] is corrupt. Downloaded [1159168] bytes but [792768300] bytes are expected. Please retry.

To Reproduce

Checkout opensearch-benchmark.
Setup all the prereq in Developer Guide linked above.
execute make prereq
execute make install
source .venv/bin/activate
Run the benchmark test. sample command: opensearch-benchmark execute_test --workload=eventdata --target-hosts=endpoint --pipeline=benchmark-only --workload-params="number_of_replicas:1" --client-options="use_ssl:true,basic_auth_user:USERNAME,basic_auth_password:PASSWORD,verify_certs:false" --kill-running-processes

Expected behavior
Benchmark test should execute successfully.
Logs
log-devdesktop.txt
Traceback (most recent call last):

 File "/home/ankikala/.pyenv/versions/3.8.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
   result = self.fn(*self.args, **self.kwargs)

 File "/local/home/ankikala/ws/opensearch-benchmark/osbenchmark/workload/loader.py", line 418, in prepare_docs
   preparator.prepare_document_set(document_set, data_root[0])

 File "/local/home/ankikala/ws/opensearch-benchmark/osbenchmark/workload/loader.py", line 575, in prepare_document_set
   self.downloader.download(document_set.base_url, target_path, expected_size)

 File "/local/home/ankikala/ws/opensearch-benchmark/osbenchmark/workload/loader.py", line 490, in download
   net.download(data_url, target_path, size_in_bytes, progress_indicator=progress)

 File "/local/home/ankikala/ws/opensearch-benchmark/osbenchmark/utils/net.py", line 239, in download
   raise exceptions.DataError("Download of [%s] is corrupt. Downloaded [%d] bytes but [%d] bytes are expected. Please retry." %

osbenchmark.exceptions.DataError: Download of [/home/ankikala/.benchmark/benchmarks/data/eventdata/eventdata.json.bz2] is corrupt. Downloaded [1159168] bytes but [792768300] bytes are expected. Please retry.```

Are you able to run a test using a different workload? I received the same error when trying eventdata, but I did a quick check and was able to use the geonames and percolator workloads.
I'll look into the data corruption issue. We may have missed something when uploading the dataset for eventdata

Thanks. I was able to run other workloads just fine.

Hi Ankit,
You should be able to run eventdata as well now.

authors: achitojha ankitkala travisbenedict

repo: opensearch-project/opensearch-benchmark


text:
[Enhancement] Provide commit id information for each repo before distribution build starts

[Enhancement] Provide commit id information for each repo before distribution build starts.
Return a link to the yml file with all the commit-ids of repos from the input manifest file.

@peterzhuamazon Do you have any updates on this issue? @bshien would need this data soon for his dashboard effort.

@bbarani I will work on it next week.
Thanks.

authors: bbarani peterzhuamazon

repo: opensearch-project/opensearch-build


text:
[BUG] Bundle Snapshot based E2E Cypress tests workflow always failed on push event

What is the bug?
The workflow cypress-workflow-bundle-snapshot-based-windows.yml and cypress-workflow-bundle-snapshot-based.yml always failed on pushing to branch.
Error:
  /usr/bin/git branch --show-current
  fatal: not a git repository (or any of the parent directories): .git

The reason is dorny/paths-filter@v2 doesn't work on push event by default. As it uses git commands to detect changes - repository must be already checked out
How can one reproduce the bug?
Check this example: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7478923388/job/20354961741
What is the expected behavior?
It should not fail the workflow running, we could either skip the workflow on push event, or checkout the code so that dorny/paths-filter can detect the changes.
What is your host/environment?
Operating system, version.
Do you have any screenshots?
If applicable, add screenshots to help explain your problem.
Do you have any additional context?
Add any other context about the problem.

fixed in https://github.com/opensearch-project/opensearch-dashboards-functional-test/pull/1017


text:
security analytics dashboard plugin cypress tests update

Description
Syncs new rule tests of the security analytics dashboard plugin with functional tests
Check List

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

I don't seem to have permissions to apply labels, or run the checks for PRs in this repo. Would someone mind adding the backport 2.x label to this PR? This is targeting the 2.8 release.

@CCongWang @kavilla
It looks like the Security Analytics Release tests, Reports Release tests, and Backport workflows failed with only a This job failed error appearing on the details page. Would you happen to have an idea of what could cause that?
I also noticed that the CodeQL workflow failed with This version of the CodeQL Action was deprecated on January 18th, 2023, and is no longer updated or supported, but I don't think that would necessarily block the PR.

Hi @AWSHurneyt , for failed CodeQL workflow, looks like the deprecation message is just a warning, see this successful example: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/4636082170/jobs/8249870977?pr=608
the true error is:
Uploading results
  Processing sarif files: ["/home/runner/work/opensearch-dashboards-functional-test/results/javascript.sarif"]
  Uploading results
  Error: Server Error

I'm gonna rerun to see if it's something transient

@AWSHurneyt for the other failed workflows with This job failed, I guess it was due to some flakyness, rerunning

Would someone re-run the test? I don't have permission to run the tests.

authors: AWSHurneyt CCongWang jovancacvetkovic


text:
ART-6608 Fix RHCOS links

https://issues.redhat.com/browse/ART-6608
Right now rhcos lookup fails with Encountered error looking up the latest RHCOS build: HTTP Error 403: Forbidden
All rhcos builds 4.9+ have moved to the new pipeline format.
Move rhcos funcs to rhcos.py, fix build urls, remove old pipeline branching code
Test:

./art_bot_dev.py
where in 4.13 is the libslirp rpm used

Merging this, as this is hindering day to day lookups

repo: openshift-eng/art-bot


text:
explore a way to make the variable exposed as CEL

those template variables are nice and simple but we have a lot of feedback that they are too simple compared to triggerbinding.
We need to see if there is some way to make the template variable  exposed as CEL expression which would allow more flexibility on using them...

For me, this would be a very essential feature. I currently try to adopt PAC in a setup, which relies heavily on triggerbindings, the full payload data and even a custom interceptor. But I am not quite sure if this is possible with the limited amount of inputs available as of now.
Same is true for #1014.

It appears this was resolved in #1466.
Should this issue be closed?

yes thanks for the reminder!

authors: chmouel csfreak mpreu

repo: openshift-pipelines/pipelines-as-code


text:
Update message for github-webhook and gitlab-webhook for secret token

Changes
For GitHub:
$ ./tkn-pac setup github-webhook
✓ Setting up GitHub Webhook for Repository https://github.com/openshift-pipelines/pipelines-as-code
? Please enter your controller public route URL:  https://smee.io/kIVoOfQotuoc1Ca
ℹ Go to this URL https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks#secret to know more about secret token
? Please enter the secret to configure the webhook for payload validation
  Hint: secret token is unique random id. eg: "12345":

For GitLab:
$ ./tkn-pac setup gitlab-webhook
? Please enter the project ID for the repository you want to be configured : 17103
? Please enter your controller public route URL:  https://smee.io/kIVoOfQotuoc1Ca
ℹ Go to this URL https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#validate-payloads-by-using-a-secret-token to know more about Secret token
? Please enter the secret token to configure the webhook for payload validation
  Hint: secret token is unique random id. eg: "12345":

Submitter Checklist

[ ] ♽  Run make test lint before submitting a PR (ie: with pre-commit, no need to waste CPU cycle on CI
[ ] 📖 If you are adding a user facing feature or make a change of the behavior, please verify that you have documented it
[ ] 🧪 100% coverage is not a target but most of the time we would rather have a unit test if you make a code change.
[ ] 🎁 If that's something that is possible to do please ensure to check if we can add a e2e test.
[ ] 🔎 If there is a flakiness in the CI tests then don't necessary ignore it, better get the flakyness fixed before merging or if that's not possible there is a good reason to bypass it. (token rate limitation may be a good reason to skip).

probably not a good idea to hint for 1234
what about generating by default a random generated string instead and if user press enter i will use it

probably not a good idea to hint for 1234

what about generating by default a random generated string instead and if user press enter it will use it unless they type >something else,

i think we should not even ask for webhook secret and just generate one for the user and use it on webook/repo webhook >secret, we used to do this before in "tkn pac bootstrap" before the cli redesign cc @sm43

I agree to not to ask for webhook secret from user instead we can handle it internally

/retest

Codecov Report

Merging #778 (93e3ffc) into main (5800120) will decrease coverage by 0.18%.
The diff coverage is 42.85%.

@@            Coverage Diff             @@
##             main     #778      +/-   ##
==========================================
- Coverage   66.21%   66.02%   -0.19%
==========================================
  Files          75       75
  Lines        4537     4571      +34
==========================================
+ Hits         3004     3018      +14
- Misses       1250     1267      +17
- Partials      283      286       +3

Impacted Files
Coverage Δ

pkg/cli/webhook/github.go
53.84% <42.85%> (-0.61%)
:arrow_down:

pkg/cli/webhook/gitlab.go
53.01% <42.85%> (-0.74%)
:arrow_down:

pkg/cmd/tknpac/list/list.go
33.33% <0.00%> (+6.06%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

can you update the description with the final command output?
we can merge then 🙃

can you update the description with the final command output? we can merge then upside_down_face

Updated comment

authors: chmouel codecov-commenter savitaashture sm43


text:
RHODS: take the addon email address from the secrets

ocm_deploy_addon: allow receiving any parameter for the add installation

toolbox: rhods deploy_addon: take email address to register as parameter

testing: ods: take the addon address from the secrets

/cc @kpouget

/test jh-at-scale

/test ods-jh-at-scale

/test ods-jh-at-scale

/test ods-jh-at-scale

/test ods-jh-at-scale

/test ods-jh-at-scale
On Wed, Jun 15, 2022 at 6:42 PM openshift-ci[bot] @.***>
wrote:

@kpouget https://github.com/kpouget: The following test failed, say
/retest to rerun all failed tests or /retest-required to rerun all
mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/ods-jh-at-scale e34be28
https://github.com/openshift-psap/ci-artifacts/commit/e34be28f8b601015e01d358c232105bac270601a
link
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift-psap_ci-artifacts/383/pull-ci-openshift-psap-ci-artifacts-master-ods-jh-at-scale/1537112960046993408
true /test ods-jh-at-scale
Full PR test history
https://prow.ci.openshift.org/pr-history?org=openshift-psap&repo=ci-artifacts&pr=383.
Your PR dashboard
https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:kpouget.
Instructions for interacting with me using PR comments are available here
https://git.k8s.io/community/contributors/guide/pull-requests.md. If
you have questions or suggestions related to my behavior, please file an
issue against the kubernetes/test-infra
https://github.com/kubernetes/test-infra/issues/new?title=Prow issue:
repository. I understand the commands that are listed here
https://go.k8s.io/bot-commands.
—
Reply to this email directly, view it on GitHub
https://github.com/openshift-psap/ci-artifacts/pull/383#issuecomment-1156697693,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABZVQIRE4DEIC6H5SDH55G3VPIBVLANCNFSM5YYWB43Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

CI was down for a few minutes. Should be resolved, so retrying ...
/test ods-jh-at-scale

/test ods-jh-at-scale

/test images

/test ods-jh-at-scale

/test ods-jh-at-scale

not sure what happened, the JSON body request looks sane
{
  "addon": {
    "id": "managed-odh"
  },
  "parameters": {
      "items": [{"id":"notification-email","value":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}]
   }
}

but the request to install the addon failed :/
2022-06-20 08:04:03,641 p=220 u=psap-ci-runner n=ansible | roles/ocm_deploy_addon/tasks/main.yml:39
2022-06-20 08:04:03,641 p=220 u=psap-ci-runner n=ansible | TASK: ocm_deploy_addon : Install addon managed-odh via OCM API
2022-06-20 08:04:03,641 p=220 u=psap-ci-runner n=ansible |
2022-06-20 08:04:03,641 p=220 u=psap-ci-runner n=ansible | <command>
set -o pipefail;
url="/api/clusters_mgmt/v1/clusters/1sut14g3bnj1ijlg98ljivp5lq4o7me2/addons"
body="/logs/artifacts/000__rhods__deploy_addon/addon_managed-odh.json"
output=$(ocm post "$url" --body=$body 2>&1);
echo "$output" >&2; # for observation
echo "$output" | jq -r '.kind'
</command>
2022-06-20 08:04:03,641 p=220 u=psap-ci-runner n=ansible |
2022-06-20 08:04:03,642 p=220 u=psap-ci-runner n=ansible | <stdout> Error
2022-06-20 08:04:03,642 p=220 u=psap-ci-runner n=ansible | <stderr> {
2022-06-20 08:04:03,642 p=220 u=psap-ci-runner n=ansible | <stderr>   "kind": "Error",
2022-06-20 08:04:03,642 p=220 u=psap-ci-runner n=ansible | <stderr>   "id": "500",
2022-06-20 08:04:03,642 p=220 u=psap-ci-runner n=ansible | <stderr>   "href": "/api/clusters_mgmt/v1/errors/500",
2022-06-20 08:04:03,642 p=220 u=psap-ci-runner n=ansible | <stderr>   "code": "CLUSTERS-MGMT-500",
2022-06-20 08:04:03,642 p=220 u=psap-ci-runner n=ansible | <stderr>   "operation_id": "c349f656-d3a8-4cf1-adfd-4ccdc72d9aed"
2022-06-20 08:04:03,642 p=220 u=psap-ci-runner n=ansible | <stderr> }

I'll look at it in a local cluster

/test ods-jh-at-scale

/test ods-jh-full-scale

/test ods-jh-full-scale

/test ods-jh-full-scale

tested locally. Merging.

repo: openshift-psap/ci-artifacts


text:
Add 4.1 RHEL8 puddle conf

Enables RHCOS to build using 4.1 content built in RHEL8 buildroots

/hold
let me add the new job files first

/hold cancel

Thanks for doing this!  Looks sane to me.

authors: cgwalters tbielawa

repo: openshift/aos-cd-jobs


text:
Add new developers to MAIL_LIST_SUCCESS

Add new developers to MAIL_LIST_SUCCESS and MAIL_LIST_FAILURE (Akram and Vibhav)
cc @gabemontero

@adammhaile @sosiouxme @tbielawa does ART have permissions to merge this change ?
We want to add the new owners of openshift/jenkins to be on the mailing list for our RPM update jobs.

authors: akram gabemontero


text:
Add encryption config to APIServer

Signed-off-by: Monis Khan mkhan@redhat.com
@mfojtik @sttts @p0lyn0mial

/lgtm
/approve

authors: enj sttts

repo: openshift/api


text:
Update OWNERS

Update OWNERS file to include @rogbas and remove @aweiteka :tada:

/lgtm

repo: openshift/aws-account-operator


text:
[release-4.1] Bug 1802860: bump(*):

Fix symlinks with k8s.io/kubernetes
Pin github/com/mtrmac/gpgme to v0.1.2

Fixes CVE-2020-8945

/retest
AWS rate limiting

/hold
Image eco is failing due to EOL php image:
Feb 26 19:08:21.815: INFO: At 2020-02-26 19:06:20 +0000 UTC - event for cakephp-example: {buildconfig-controller } BuildConfigInstantiateFailed: error instantiating Build from BuildConfig e2e-test-cakephp-repo-test-4jl8d/cakephp-example (0): Error resolving ImageStreamTag php:7.3 in namespace openshift: unable to find latest tagged image

@gabemontero for the purposes of this PR I'm tempted to override.

/hold
Image eco is failing due to EOL php image:
Feb 26 19:08:21.815: INFO: At 2020-02-26 19:06:20 +0000 UTC - event for cakephp-example: {buildconfig-controller } BuildConfigInstantiateFailed: error instantiating Build from BuildConfig e2e-test-cakephp-repo-test-4jl8d/cakephp-example (0): Error resolving ImageStreamTag php:7.3 in namespace openshift: unable to find latest tagged image

@gabemontero for the purposes of this PR I'm tempted to override.

I would +1 that @adambkaplan

/override e2e-aws-image-ecosystem

/hold cancel

/hold cancel

/retest
/bugzilla refresh

/close
4.1 has reached the end of its lifecycle. This fix does not have sufficient severity to warrant a merge.

authors: adambkaplan gabemontero

repo: openshift/builder


text:
WIP - Mount trusted CA to the registry operand

Building on top of #340 - mounting the cluster trusted CA to the "low-priority" trust source at /usr/share/pki/ca-trust-source/ within the registry.
The registry calls update-ca-trust extract to create the combined CA bundle, which includes the CA for the internal registry and other externally trusted registries.

/assign @coreydaley
/cc @danehans @bparees

Just as a reminder, I need this on to finish https://github.com/openshift/cluster-image-registry-operator/pull/342

/close
Moved to #360

authors: adambkaplan ricardomaraschini


text:
Consume service-account.key instead of service-account.pub

Since openshift/installer@57127ec (coreos/tectonic-installer#3270), the installer sets a kube-apiserver secret in the kube-system with both service-account.key and service-account.pub.  We also do that for the openshift-apiserver secret.  But the public key can be computed from the private key, so it would be nice to not have to set both in the same secret.  It looks like the kube-controller-manager operator uses the private key (although I haven't found code where it pulls it from the installer-generated secret); can we update this operator to use the private key too?  Or should the installer be setting separate secrets for each operator?  Or something (again, the connections are not very clear to me ;)?

Why should we make a private key visible to a component if that is not strictly necessary. Especially the apiserver is exposed to the outer network and hence especially sensitive.

Why should we make a private key visible...

Sounds like you want the "use two secrets" approach.  What does each operator need, and what should the secrets be called?

The secrets are operator-specific already now (in the openshift-kube-* namespaces), rendered by the operator during bootstrapping.
Wondering about the kube-apiserver secret. That should be deleted I guess. Is there any consumer left?

authors: sttts wking


text:
Bug 1991357: bump library.go

Fix the bug on the installer_controller  to only be degraded if lastFailedRevision = TargetRevision

/assign @sttts

/lgtm
/approve

The test TestAPIRemovedInNextReleaseInUse is passing now
/retest
/assign @sanchezl

/retest
/lgtm
/approve

/bugzilla refresh

authors: aojea sttts


text:
HOLD - Revert #196

This PR:

Reverts PR #196 as it conflicts with @vojtechszocs original PR to pull common types from lib-core
Fixes the peer dependency discrepancy within lib-utils and lib-extensions to correctly reflect they now depend on lib-core 2.0.0

/hold

Codecov Report
Base: 38.38% // Head: 38.41% // Increases project coverage by +0.02% :tada:

Coverage data is based on head (1e67a03) compared to base (acf41a3).
Patch coverage: 33.33% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #197      +/-   ##
==========================================
+ Coverage   38.38%   38.41%   +0.02%
==========================================
  Files          64       64
  Lines        1589     1588       -1
  Branches      353      353
==========================================
  Hits          610      610
+ Misses        944      943       -1
  Partials       35       35

Impacted Files
Coverage Δ

packages/lib-utils/src/app/AppInitSDK.tsx
0.00% <0.00%> (ø)

...lib-utils/src/app/api-discovery/discovery-cache.ts
0.00% <0.00%> (ø)

packages/lib-utils/src/app/api-discovery/index.ts
0.00% <0.00%> (ø)

packages/lib-utils/src/app/redux/index.ts
0.00% <0.00%> (ø)

...ckages/lib-utils/src/app/redux/reducers/k8s/k8s.ts
0.00% <0.00%> (ø)

...omponents/details-page-header/utils/ActionMenu.tsx
95.91% <ø> (ø)

...es/lib-utils/src/components/list-view/ListView.tsx
81.81% <ø> (ø)

...ib-utils/src/components/table/VirtualizedTable.tsx
67.07% <ø> (ø)

...tils/src/components/table/VirtualizedTableBody.tsx
39.28% <ø> (ø)

packages/lib-utils/src/index.ts
0.00% <ø> (ø)

... and 5 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

/hold cancel

authors: codecov-commenter florkbr vojtechszocs

repo: openshift/dynamic-plugin-sdk


text:
OCPBUGS-22348: address CVE-2023-44487

Backport of https://github.com/openshift/external-dns-operator/pull/204:

bump controller-runtime v0.13.2
bump the operand image to the latest 1.1 version (fb8ea37)
disable HTTP/2 for kube-rbac-proxy container
disable HTTP/2 for webhook server

/lgtm
@miciah want to approve?

Thanks!
/approve

repo: openshift/external-dns-operator


text:
Pod logs

Fixes

https://issues.redhat.com/browse/HAC-3600
Description

Adds Pod Logs component and related utils
Updates Component list view and adds view pod logs action
Updates view logs to view build logs in ComponentListView action and BuildLogs modal
Updates fetching of GitOpsDeployment using SnapshotEncBinding as existing in-place fetch of GitOpsDeployment fails due to missing labels
Adds related unit tests

Type of change

[x] Feature
[ ] Bugfix
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

Screen shots / Gifs for design review

How to test or reproduce?

Added unit tests

Browser conformance:

[x] Chrome
[x] Firefox
[ ] Safari
[ ] Edge

component list page crashes, when a new component is added.

Deployment is successful and the pod is running bu thte Podstatus says PodScheduled, shouldn't it be in Running state?

@karthikjeeyar updated contional checking for GitOpsDeployment so now add component should work fine and also updated modal to say view build logs

/approve

/retest

repo: openshift/hac-dev


text:
fix: use the correct workspace based on namespace for Github redirect

Fixes

https://issues.redhat.com/browse/KFLUXBUGS-1636
Description

Type of change

[ ] Feature
[ ] Bugfix
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

Screen shots / Gifs for design review

How to test or reproduce?

Browser conformance:

[ ] Chrome
[ ] Firefox
[ ] Safari
[ ] Edge

[test]

/retest

needs tests to suffice codecov requirement

/retest

/retest

/retest

/retest

/retest

/retest

/retest

/retest


text:
HIVE-2491: Raise the memory buildah task

Compiling hiveutil is not for the faint of heart and the default Konflux buildah task was not up to it. The 8gb one should do it.

/lgtm

authors: celebdor lleshchi

repo: openshift/hive


text:
ACM-6435: add pausedUntil create cluster option in CLI

What this PR does / why we need it:
This PR is to add pausedUntil create cluster option in hcp CLI to support MCE's cluster create/update Ansible automation template. https://issues.redhat.com/browse/ACM-1368
MCE's cluster curator controller will clear the spec.pausedUntil from the HostedCluster and NodePools when the conditions are met.
Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #
Checklist

[ ] Subject and description added to both, commit and PR.
[ ] Relevant issues have been referenced.
[ ] This change includes docs.
[ ] This change includes unit tests.

/retitle: ACM-6435: add pausedUntil create cluster option in CLI

/retitle ACM-6435: add pausedUntil create cluster option in CLI

/retest

/retest

Can you elaborate on the value of creating a paused cluster? how are you planning to cleanup the field?
If we move forward with this can we please add the same flag to the dev cli so product one is a subset of the dev one.

@enxebre Please see the updated PR description for the value of creating a paused cluster. I will update the dev CLI one we as well. Thx.

/test e2e-aws

/test e2e-aws

/unhold

authors: bryan-cox enxebre rokej

repo: openshift/hypershift


text:
Replace contributors symlink with actual file

The b5aa26e0c8a0f69431fd727a21a749ebeaa3193d commit introduced a symlink
into the docs content, which breaks the local preview make serve-containerized
target (which throws file not found errors trying to deal with the link).
This commit replaces the symlink with an actual file to fix the regression.
We should consider having CONTRIBUTING.md just link to the docs page instead
of symlinking it.

/lgtm

authors: csrwng ironcladlou


text:
WIP: installerpod: wait for the static pod manifest to be mirrored to API server before terminating

@deads2k I still wonder what is wrong with this approach...
Given the timings, kubelet should notice the static pod manifest change pretty fast, so this should extend the time needed to install new revision by a few seconds. However, if this gets stale for some reason, the event here might be valuable because it will tell us exactly WHEN the copying of manifests started and how long we waited before timing out.

proof: https://github.com/openshift/cluster-kube-controller-manager-operator/pull/604

repo: openshift/library-go


text:
ETCD-627: configure static pod installer rate limits and backoffs

This PR allows the sync context to be configured with rate limits and a flag to throttle informer events. It also allows the static pod installer to pass its sync context from the builder.

/assign @p0lyn0mial

closing in favor of #1744


text:
Bug fix: added missing patches

Description
Bug fix: added missing patches
It allow us to run just one selected unit test, one test module etc.
Type of change

[ ] Refactor
[ ] New feature
[x] Bug fix
[ ] CVE fix
[ ] Optimization
[ ] Documentation Update
[ ] Configuration Update
[ ] Bump-up dependent library

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.80%. Comparing base (4508b86) to head (b839cb3).
Report is 25 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #737      +/-   ##
==========================================
- Coverage   94.83%   94.80%   -0.04%
==========================================
  Files          53       53
  Lines        1896     1904       +8
==========================================
+ Hits         1798     1805       +7
- Misses         98       99       +1

see 1 file with indirect coverage changes

/retest

only touches unit tests and that job passed

authors: bparees codecov-commenter tisnik

repo: openshift/lightspeed-service


text:
OCPBUGS-19563: Move greenboot integration scripts to /usr/lib/greenboot

When greenboot runs scripts, it considers two locations /usr/lib/greenboot and etc/greenboot. It also runs required.d and wanted.d scripts from each path in sequence, that is:

(1) /usr/lib/greenboot/checks/required.d
(2) /usr/lib/greenboot/checks/wanted.d
(3) /etc/greenboot/checks/required.d
(4) /etc/greenboot/checks/wanted.d

At the moment of writing this commit, greenboot shipped in RHEL unconditionally exits with rc=1 when required.d scripts failed.
Because MicroShift's scripts were placed in the (3) directory, they would not run if any script from (1) failed.
In upstream greenboot the behavior has already changed: all scripts will run, regardless of the path or importance (required vs wanted), and then greenboot will exit with appropriate return code.

$ rpm -qlp microshift-greenboot-4.14.0_0.nightly_2023_09_12_195514_20230922084623_fd3310ea3-1.el9.noarch.rpm

/usr/lib/greenboot/check/required.d/40_microshift_running_check.sh
/usr/lib/greenboot/green.d/40_microshift_set_healthy.sh
/usr/lib/greenboot/red.d/40_microshift_pre_rollback.sh
/usr/lib/greenboot/red.d/40_microshift_set_unhealthy.sh
/usr/share/microshift/functions/greenboot.sh

/cherrypick release-4.14

/test images

authors: pacevedom pmtk

repo: openshift/microshift


text:
[Telemetry] Add data about s2i vs devfile component

/kind feature
Which functionality do you think we should add?
Telemetry should show whether components created are s2i vs devfile.
Why is this needed?
This is needed so that we can know how many people are still using s2i and devfile.
Acceptance Criteria

[ ] odo create should send data about this.
[ ] TBD

Probably not worth it as we are already planning to remove s2i related code https://github.com/openshift/odo/issues/4932
/close

authors: kadel valaparthvi

repo: openshift/odo


text:
renamed Url to URL to resolve lint warning and consistency

What kind of PR is this?

/kind cleanup
What does does this PR do / why we need it:
see $SUBJECT
Which issue(s) this PR fixes:
No issue
How to test changes / Special notes to the reviewer:
just internal renaming of variable, tests should pass

/test v4.2-e2e-scenarios

/retest

/retest

Since #2362 got merged after the latest CI failure on this PR, simply re-triggering the 4.3 integration tests.
/test v4.3-integration

/retest

Terraform errors
/retest

/lgtm

/retest

/retest

/retest

authors: cdrage dharmit girishramnani


text:
rebase to 1.15 + switch to go mod

See what switch to go mod looks like.

conflicts, proof was made

repo: openshift/openshift-controller-manager