text:
[WIP] Additions for GBSA support

Checking in my progress on GB support from a while ago. Still need to finish.
The syntax looks like:
 <GBSAForce model="OBC1" radius_units="nanometers" sasa_penalty="5" sasa_penalty_units="cal/mol/angstroms**2">
   <Atom smirks="[#1:1]" radius="0.12" scale="0.85"/>
   <Atom smirks="[#1:1]~[#6]" radius="0.13" scale="0.85"/>
   <Atom smirks="[#1:1]~[#8]" radius="0.08" scale="0.85"/>
   <Atom smirks="[#1:1]~[#16]" radius="0.08" scale="0.85"/>
   <Atom smirks="[#6:1]" radius="0.22" scale="0.72"/>
   <Atom smirks="[#7:1]" radius="0.155" scale="0.79"/>
   <Atom smirks="[#8:1]" radius="0.15" scale="0.85"/>
   <Atom smirks="[#9:1]" radius="0.15" scale="0.88"/>
   <Atom smirks="[#14:1]" radius="0.21" scale="0.8"/>
   <Atom smirks="[#15:1]" radius="0.185" scale="0.86"/>
   <Atom smirks="[#16:1]" radius="0.18" scale="0.96"/>
   <Atom smirks="[#17:1]" radius="0.17" scale="0.8"/>
 </GBSAForce>

The GBSAForceGenerator code hasn't been altered from NonbondedForce yet. Working on that now.

Spec changes look good. LMK when you are ready for code review, @jchodera .

Will do! Still coding for now...

@jchodera :

What is the most important immediate use case? Would it be just OBC2?

Yes! Really we just need to get the basic framework for support of SOME GB model (OBC2 is a good choice) in place. Then, we can later deal under the hood with being able to support more GBSA models.

I've managed to get HCT, OBC1, and OBC2 in, but there is one more major issue to fix with how (radius, screen) are translated into what parameters used to be sr and or in the gbsaforces.py. I hadn't realized the energy expressions were encoded in terms of a shifted radius and a scaled radius rather than the original radius and scale parameters---this will take a few min to figure out, and are required for numerical correctness, but I want to at least make sure the code is doing the right things as far as parameter assignment right now.
In running the tests, I've been getting these errors:
======================================================================
ERROR: Test setup of a molecule which activates partial bond order code.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/choderaj/miniconda3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/Users/choderaj/github/open-forcefield-group/smarty/smarty.open-forcefield-group/smarty/tests/test_forcefield.py", line 448, in test_partial_bondorder
    system = ff.createSystem(topology, [mol], chargeMethod = 'OECharges_AM1BCCSym', verbose = verbose)
  File "/Users/choderaj/github/open-forcefield-group/smarty/smarty.open-forcefield-group/smarty/forcefield.py", line 947, in createSystem
    self._assignPartialCharges(molecule, oechargemethod)
  File "/Users/choderaj/github/open-forcefield-group/smarty/smarty.open-forcefield-group/smarty/forcefield.py", line 856, in _assignPartialCharges
    partial_bondorders.append( bond.GetData("WibergBondOrder"))
  File "/Users/choderaj/miniconda3/lib/python3.5/site-packages/openeye/oechem.py", line 7473, in GetData
    return _oechem.OEBase_GetData(self, *args)
ValueError: GetData: WibergBondOrder not found.

Any idea what is going on here?

@jchodera - are you running OpenEye toolkits 2016.10 or later? The Wiberg bond orders were in the October toolkits versions (and thereafter I believe).
(This is used in tests -- we have a test that looks to see whether partial (Wiberg) bond orders are working correctly by looking at benzene or something similar, using an aromatic C~C bond length which is interpolated between single and double. If the OE AM1-BCC calculation doesn't assign partial bond orders (which it did beginning in the October release) then the test fails.)

If the above is NOT an issue with your OpenEye toolkits, raise a separate issue and I can troubleshoot.

@jchodera - are you running OpenEye toolkits 2016.10 or later?

That was the issue! Thanks!
Now I'm seeing:
======================================================================
ERROR: Test evaluation of energies from parm@frosst ffxml files versus energies of equivalent systems.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/choderaj/miniconda3/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/Users/choderaj/github/open-forcefield-group/smarty/smarty.open-forcefield-group/smarty/tests/test_forcefield.py", line 358, in test_smirff_energies_vs_parmatfrosst
    results = compare_molecule_energies( prmtop, crd, forcefield, mol, verbose = verbose )
  File "/Users/choderaj/github/open-forcefield-group/smarty/smarty.open-forcefield-group/smarty/forcefield_utils.py", line 270, in compare_molecule_energies
    smirfftop, ambersys, smirffsys, amberpos, verbose = verbose, skip_assert = skip_assert, skip_improper = skip_improper )
  File "/Users/choderaj/github/open-forcefield-group/smarty/smarty.open-forcefield-group/smarty/forcefield_utils.py", line 208, in compare_system_energies
    syscheck.check_force_parameters(skipImpropers = skip_improper)
TypeError: check_force_parameters() got an unexpected keyword argument 'skipImpropers'

Also, about a billion warnings like
Warning: data type mismatch found when using generic data

Update your openmoltools.
Maybe I did not remember to update these in the package requirements?

So, warnings like this:

Warning: data type mismatch found when using generic data

Those became normal when I think we started accessing the Wiberg bond orders, I think. They're not functionally important, but I need to figure out how to disable them. I need to create an issue but am on the way out the door; you can create one and assign it to me if you like.

Ah, that was my fault. I neglected to update openmoltools to 0.7.3.

Remaining to be done:

[ ] Fix is how I match up the forcefield parameters (radius and scale) with the parameters in the CustomGBForce energy expressions. I may need to do some retooling of gbsaforces.py.
[ ] Expose the ability to control the SASA penalty
[ ] Generalize the SA models to allow us to extend this to other SA models in the future (which may have different numbers of parameters) like EEF1
[ ] Add GBn and GBn2 models
[ ] Add additional tests of variants of specifying or omitting some GBSA parameters to make sure we trigger informative exceptions when we omit some parameters

At this point, it would be useful to get feedback on the current <GBSAForce/> tag spec to make sure it exposes what is needed.

I won't be able to tackle the rest until the morning, so no rush!

@jchodera - the GBSA spec looks good to me except a couple of minor things:
Relating to alpha, beta, gamma:

Units are for these per-particle parameters (such as radius_units) optionally specified in the  tag.

It's not actually clear to me from this what allowable units are for alpha, beta, and gamma. The link is not that helpful since it just leads to a reference. Can we at least say "has units of length" (or whatever it should have.)?
Also (and maybe this is just my lack of knowledge):

The sa_model attribute specifies the solvent-accessible surface area model ("SA" part of GBSA), which can currently only be ACE or omitted (in which case the SA term is omitted)

What is "ACE"? Spell out on first use?
Otherwise, the spec looks great to me!

It's not actually clear to me from this what allowable units are for alpha, beta, and gamma. The link is not that helpful since it just leads to a reference. Can we at least say "has units of length" (or whatever it should have.)?

I'm not sure what the units of alpha, beta, and gamma for GB(neck) models are either. We'd have to look in the original references to see if these parameters have units. If they do have units, they'd be interpreted in the OpenMM unit system (e.g. nanometers) unless alpha_unit was specified. I'll do some digging tomorrow.

What is "ACE"? Spell out on first use?

I believe the "ACE" model is this model from Schaefer and Karplus, for "analytical continuum electrostatics".

I'm not sure what the units of alpha, beta, and gamma for GB(neck) models are either. We'd have to look in the original references to see if these parameters have units. If they do have units, they'd be interpreted in the OpenMM unit system (e.g. nanometers) unless alpha_unit was specified. I'll do some digging tomorrow.

Haha, OK. Are you sure you don't want to just move this into the category of "we will eventually support..." and punt it for now? The main point at this point, at least as I understood it, was to at least get basic GBSA support for one or more models in so that we could flesh out the spec; you seem to have achieved that.
Thanks for the pointer on ACE.

Quick update: Nearly finished with this, but finishing some additional cleanup/refactoring to make the spec more extensible:

Having separate attributes for the surface area penalty and the units for the surface area penalty didn't make sense---it makes much more sense to just let units be specified in the surface area penalty attribute, like 5*calories/angstrom**2.
I've refactored things to be a bit more general in that you can specify a gb_model and an sa_model for the GB and SA parts, respectively, and each model may have a few global parameters (and per-particle parameter unit specifiers) that can be specified separately.

Will get this wrapped up today.

I'm not sure what the units of alpha, beta, and gamma for GB(neck) models are either. We'd have to look in the original references to see if these parameters have units.

In case this wasn't resolved yet, I just went back to take a peek at the reference. Alpha/beta/gamma are unitless scaling terms (Table 1) just as they are in the OBC models.

I think the draft implementation is now complete. Please take a look at the updated GBSA spec to make sure you're happy with it.

Woo! Tests pass.
Note that I haven't tested GBSA energies for correctness (that's hard)---I've just made sure the format can be parsed and the energies aren't crazy.

@jchodera - this looks excellent to me, and I don't see any problems with the spec. OK to merge?
Then we could proceed to split off smarty.forcefield in a separate PR. You still want to be the one to do this, right?

Actually, I see it's OK to merge since you changed the title. I'll go ahead. :)

Then we could proceed to split off smarty.forcefield in a separate PR. You still want to be the one to do this, right?

Yep! On it.

@jchodera - one other thought on that. There are various things in utilities, examples, etc. that will belong in one repo or the other. I'm happy to handle moving those to where they ought to be, to save you time, after you get the basic skeleton up. That will also provide me with the impetus to do some cleanup/re-check documentation to make sure everything is well organized and clearly documented (I think it is, but it never hurts to re-check). I wouldn't want you to get bogged down in the reorganization because you couldn't tell where something belonged...
Thanks.

authors: ctk3b davidlmobley jchodera

repo: open-forcefield-group/smarty


text:
Update contribution-guide

Update git address ^^

LGTM

source: gharchive/pull-request

authors: MechanicKim ovekyc

repo: open-korean-text/open-korean-text

license_type: permissive

license_source: bigquery


text:
Add legal status in footer

minimal implementation for #227 (: feel free to edit

CI logs said problems with "nokogiri":
10:01:09 PM:   1. Build command from Netlify app                              10:01:09 PM: ──────────────────────────────────────────────────────────────── 10:01:09 PM:  10:01:09 PM: $ jekyll build 10:01:10 PM: /opt/buildhome/.rvm/gems/ruby-2.7.1/gems/bundler-2.2.5/lib/bundler/spec_set.rb:87:in block in materialize': Could not find nokogiri-1.11.1 in any of the sources (Bundler::GemNotFound)`
I see #254 CI failed for similar reasons
building locally I didn't have this problem :/

yeah everything is failing isn't it... https://github.com/open-life-science/open-life-science.github.io/pull/255

Emmy, I think this should be fixed now if you could pull in master? If it passes we can merge

Thanks @emmyft!

authors: bebatut emmyft yochannah

license: MIT

license_source: github-api


text:
Table and Column Descriptions: two types: captured and enriched

Is your feature request related to a problem? Please describe.
When scanning database metadata, we capture table and column descriptions as taken from the application. However, we also want to enrich it will additional commentary, clarification of business process and so on. Currently there is no way to tell if description came from application or if it was enriched by OMD users.
Describe the solution you'd like
I would like to have two description types. One which is ingested and one that can be enriched by community. Meaning:

auto, populated by ingestion service from the source metadata. It will often be blank or short, so no need for much space); if it is blank, it should be collapsed,  ideally.
and one for community, additional updates, which is not touch by re-ingesting the metadata.

Describe alternatives you've considered
Using comments. But comments are difficult to consolidate and good metadata might be lost in the comments thread.
Additional context
Inspiration from another data catalog tool

https://docs.castordoc.com/assets/tables#columns

@harshach @pmbrull Do we have any flag to identity above?

@DovileKr Not sure I follow the goal of this. How I see descriptions is as a way to centralise all known information from an asset, and make sure that we keep it up-to-date.
You should decide if OM becomes the source of truth, or other tools such as dbt. This is why by default, we won't overwrite the description once it reaches OM.
What would happen if the database description gets outdated and Data Stewards add the information in another "description"? How would users know which one to use?
I believe this might carry out more misalignment than actual value. I'll be closing this for now. Let us know your thoughts.
Thanks

@pmbrull thank you for the message. I want to share the idea why this idea came across.
in BQ we have a table for sales.

Tables has description as per our standards no table should be created without description.  Metadata was ingested to OMD
A couple of months later, a new source was added to the table and also the update logic has changed. Table was recreated (same in BQ). So table description has changed. Also description for an existing column as logic changed slightly.

Data engineer has changed table and column description in BQ. And now it is not in sync anymore with OMD. Tables aren't created using something like dbt, but SQL queries...
Here I can

ask DE to update both BQ and OMD
have DE update in OMD and have BQ read descriptions from OMD
or simply ignore the difference

So in summary yes - small inconvenience.  Maybe more prominent for something like SaaS vendors. Cause if they change some meta like descriptions, will be hard to notice.

source: gharchive/issue

authors: DovileKr chirag-madlani pmbrull

repo: open-metadata/OpenMetadata


text:
Add UI Support to Add Custom Profiler Metrics

Feature
Profiler supports the execution of custom profiler metrics using a SQL expression. We should add support to implement/add custom metrics through the UI and display these metrics.
Users will be able to add custom metrics through the add dropdown button on either the table or column level. add test will become add and the drop down will offer 2 choices add [column|table] test and add [column|table] metric. Clicking on either of the button will open a new page to configure either the test or the metric. The right side bar will be the same for test or metric only the config will change.
To display custom metrics we'll add 1 graph per custom metric. three dots on the right upper corner of the custom graph will allow to delete or edit the custom metric. Editing a custom metric should open a pop up and allow users to edit the configuration of the metric
Technical Details
For all the below endpoints id corresponds to the table ID

Create or edit a Custom metric endpoint: PUT /api/v1/tables/{id}/customMetric. If columnName is omitted from the payload then the metric will be attached at the table level.

https://github.com/open-metadata/OpenMetadata/blob/c7e758eccc322e957cc4b72f119d0d27fca9e0c4/openmetadata-spec/src/main/resources/json/schema/api/tests/createCustomMetric.json#L1-L40
Delete a custom metric endpoints

table: DELETE /api/v1/tables//{id}/customMetric/{customMetricName}
column: DELETE /api/v1/tables/{id}/customMetric/{columnName}/{customMetricName}

Get custom metrics
GET requests against tables resources accept a customMetrics field in the fields parameter. If only customMetrics field is passed ONLY table custom metrics will be returned (array of custom metrics definition)
https://github.com/open-metadata/OpenMetadata/blob/c7e758eccc322e957cc4b72f119d0d27fca9e0c4/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json#L1009-L1015
if customMetrics and columns fields are passed then table and column custom metrics definition will be returned
https://github.com/open-metadata/OpenMetadata/blob/c7e758eccc322e957cc4b72f119d0d27fca9e0c4/openmetadata-spec/src/main/resources/json/schema/entity/data/table.json#L318-L324

UI will be pickup once Ingestion side work is done cc @TeddyCr

This has been completed in PR


text:
can't set conditions for quality analysis

In some scenarios, I only want to analyze the data from this year.

closing this as it is tracked https://github.com/open-metadata/OpenMetadata/issues/11926


text:
SAML - renew token support

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
SAML doesn't renew token from existing token
Describe the solution you'd like
A clear and concise description of what you want to happen.
OM should renew token when existing token gets expired
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

this is done


text:
fix(ui): entity selection for es re-indexing

Describe your changes:
Fix User expierence around selecting entities for Elasticsearch Re-Indexing

I worked on ... because ...

Type of change:

[ ] Bug fix
[x] Improvement
[ ] New feature
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My PR title is Fixes <issue-number>: <short explanation>
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@chirag-madlani do we have an option for "All"

@chirag-madlani can we do this similar to policies page

Lets re-use this component

@harshach updated UI as per comments

authors: chirag-madlani harshach


text:
Minor: Fix user details page bugs

I worked on the following:

[x] Fix the user avatar not showing any letter when the displayName is an empty string
[x] Changed the patch request payload for displayName change to remove the displayName field if case of empty input instead of sending an empty string.

Before:
https://github.com/open-metadata/OpenMetadata/assets/51777795/09b96838-30ec-4d97-bc3e-272587f1f411
After:
https://github.com/open-metadata/OpenMetadata/assets/51777795/3fe8a0e2-826c-4ff1-9df1-045b74629dd4

Type of change:

[x] Bug fix
[x] Improvement

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My PR title is Fixes <issue-number>: <short explanation>
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@Sachin-chaurasiya I've pushed the changes. Can you please review it again?


text:
[BUG]Replace the seesaw loss function into a two-stage object detection model results in an error.

Replace the seesaw loss function into a two-stage object detection model results in an error.
Question：1.How do I replace the sesaw loss function for the two-stage object detection model?
2.What is the reason for this `bug？How can I correct this bug？
model = dict(
    type='FasterRCNN',
    backbone=dict(
        type='ResNet',
        depth=50,
        num_stages=4,
        out_indices=(0, 1, 2, 3),
        frozen_stages=1,
        norm_cfg=dict(type='BN', requires_grad=True),
        norm_eval=True,
        style='pytorch',
        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50')),
    neck=dict(
        type='FPN',
        in_channels=[256, 512, 1024, 2048],
        out_channels=256,
        num_outs=5),
    rpn_head=dict(
        type='RPNHead',
        in_channels=256,
        feat_channels=256,
        anchor_generator=dict(
            type='AnchorGenerator',
            scales=[8],
            ratios=[0.5, 1.0, 2.0],
            strides=[4, 8, 16, 32, 64]),
        bbox_coder=dict(
            type='DeltaXYWHBBoxCoder',
            target_means=[0.0, 0.0, 0.0, 0.0],
            target_stds=[1.0, 1.0, 1.0, 1.0]),
        loss_cls=dict(
            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
        loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
    roi_head=dict(
        type='DoubleHeadRoIHead',
        bbox_roi_extractor=dict(
            type='SingleRoIExtractor',
            roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
            out_channels=256,
            featmap_strides=[4, 8, 16, 32]),
        bbox_head=dict(
            type='DoubleConvFCBBoxHead',
            num_convs=4,
            num_fcs=2,
            in_channels=256,
            conv_out_channels=1024,
            fc_out_channels=1024,
            roi_feat_size=7,
            num_classes=8,
            bbox_coder=dict(
                type='DeltaXYWHBBoxCoder',
                target_means=[0.0, 0.0, 0.0, 0.0],
                target_stds=[0.1, 0.1, 0.2, 0.2]),
            reg_class_agnostic=False,
            loss_cls=dict(
                type='SeesawLoss',
                p=0.8,
                q=2.0,
                num_classes=8,
                loss_weight=1.0),
            loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=2.0)),
        reg_roi_scale_factor=1.3),
    train_cfg=dict(
        rpn=dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.7,
                neg_iou_thr=0.3,
                min_pos_iou=0.3,
                match_low_quality=True,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=256,
                pos_fraction=0.5,
                neg_pos_ub=-1,
                add_gt_as_proposals=False),
            allowed_border=-1,
            pos_weight=-1,
            debug=False),
        rpn_proposal=dict(
            nms_pre=2000,
            max_per_img=1000,
            nms=dict(type='nms', iou_threshold=0.7),
            min_bbox_size=0),
        rcnn=dict(
            assigner=dict(
                type='MaxIoUAssigner',
                pos_iou_thr=0.5,
                neg_iou_thr=0.5,
                min_pos_iou=0.5,
                match_low_quality=False,
                ignore_iof_thr=-1),
            sampler=dict(
                type='RandomSampler',
                num=512,
                pos_fraction=0.25,
                neg_pos_ub=-1,
                add_gt_as_proposals=True),
            pos_weight=-1,
            debug=False)),
    test_cfg=dict(
        rpn=dict(
            nms_pre=1000,
            max_per_img=1000,
            nms=dict(type='nms', iou_threshold=0.7),
            min_bbox_size=0),
        rcnn=dict(
            score_thr=0.05,
            nms=dict(type='nms', iou_threshold=0.5),
            max_per_img=100)))
dataset_type = 'CocoDataset'
data_root = '/home/zzheng/zhaizhikun/hicoco/'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations', with_bbox=True),
    dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
    dict(type='RandomFlip', flip_ratio=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(type='Pad', size_divisor=32),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 800),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='Pad', size_divisor=32),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img'])
        ])
]
data = dict(
    samples_per_gpu=8,
    workers_per_gpu=2,
    train=dict(
        type='CocoDataset',
        ann_file=
        '/home/zzheng/zhaizhikun/hicoco/annotations/instances_train20230322094913.json',
        img_prefix='/home/zzheng/zhaizhikun/hicoco/images/train20230322094913/',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(type='LoadAnnotations', with_bbox=True),
            dict(type='Resize', img_scale=(1333, 800), keep_ratio=True),
            dict(type='RandomFlip', flip_ratio=0.5),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='Pad', size_divisor=32),
            dict(type='DefaultFormatBundle'),
            dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
        ]),
    val=dict(
        type='CocoDataset',
        ann_file=
        '/home/zzheng/zhaizhikun/hicoco/annotations/instances_test20230322095835.json',
        img_prefix='/home/zzheng/zhaizhikun/hicoco/images/test20230322094913/',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(1333, 800),
                flip=False,
                transforms=[
                    dict(type='Resize', keep_ratio=True),
                    dict(type='RandomFlip'),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='Pad', size_divisor=32),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ]),
    test=dict(
        type='CocoDataset',
        ann_file=
        '/home/zzheng/zhaizhikun/hicoco/annotations/instances_test20230322095835.json',
        img_prefix='/home/zzheng/zhaizhikun/hicoco/images/test20230322094913/',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(1333, 800),
                flip=False,
                transforms=[
                    dict(type='Resize', keep_ratio=True),
                    dict(type='RandomFlip'),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='Pad', size_divisor=32),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ]))
evaluation = dict(interval=1, metric='bbox', save_best='auto')
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=500,
    warmup_ratio=0.001,
    step=[8, 11])
runner = dict(type='EpochBasedRunner', max_epochs=12)
checkpoint_config = dict(interval=2)
log_config = dict(
    interval=50,
    hooks=[
        dict(type='TextLoggerHook'),
        dict(
            type='WandbLoggerHook',
            init_kwargs=dict(
                project='MMdection-School',
                name='faster_rcnn_hrnetv2p_w32_1x_coco'))
    ])
custom_hooks = [dict(type='NumClassCheckHook')]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
opencv_num_threads = 0
mp_start_method = 'fork'
auto_scale_lr = dict(enable=False, base_batch_size=16)
work_dir = './work_dirs/dh_faster_rcnn_r50_fpn_seesaw_1x_coco'
auto_resume = False
gpu_ids = [1]

error message
Traceback (most recent call last):
  File "tools/train.py", line 247, in <module>
    main()
  File "tools/train.py", line 236, in main
    train_detector(
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/apis/train.py", line 246, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], File "tools/train.py", line 236, in main
    train_detector(
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/apis/train.py", line 246, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 53, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 31, in run_iter
    outputs = self.model.train_step(data_batch, self.optimizer,
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 77, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/detectors/base.py", line 248, in train_step
    losses = self(**data)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 119, in new_func
    return old_func(*args, **kwargs)
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/detectors/base.py", line 172, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/detectors/two_stage.py", line 147, in forward_train
    roi_losses = self.roi_head.forward_train(x, img_metas, proposal_list,
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/roi_heads/standard_roi_head.py", line 104, in forward_train
    bbox_results = self._bbox_forward_train(x, sampling_results,
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/roi_heads/standard_roi_head.py", line 139, in _bbox_forward_train
    loss_bbox = self.bbox_head.loss(bbox_results['cls_score'],
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 208, in new_func
    return old_func(*args, **kwargs)
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/roi_heads/bbox_heads/bbox_head.py", line 270, in loss
    loss_cls_ = self.loss_cls(
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl **kwargs)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 53, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 31, in run_iter
    outputs = self.model.train_step(data_batch, self.optimizer,
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 77, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/detectors/base.py", line 248, in train_step
    losses = self(**data)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 119, in new_func
    return old_func(*args, **kwargs)
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/detectors/base.py", line 172, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/detectors/two_stage.py", line 147, in forward_train
    roi_losses = self.roi_head.forward_train(x, img_metas, proposal_list,
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/roi_heads/standard_roi_head.py", line 104, in forward_train
    bbox_results = self._bbox_forward_train(x, sampling_results,
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/roi_heads/standard_roi_head.py", line 139, in _bbox_forward_train
    loss_bbox = self.bbox_head.loss(bbox_results['cls_score'],
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 208, in new_func
    return old_func(*args, **kwargs)
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/roi_heads/bbox_heads/bbox_head.py", line 270, in loss
    loss_cls_ = self.loss_cls(
  File "/home/zzheng/miniconda3/envs/zzk-mmlab/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/zzheng/zhaizhikun/mmdetection/mmdet/models/losses/seesaw_loss.py", line 224, in forward
    assert cls_score.size(-1) == self.num_classes + 2
AssertionError

I have the same problem, I have 20 classes, run 'print(cls_score.size(-1))' , output: 21. How can I correct this bug？

Have you solved it?

repo: open-mmlab/mmdetection


text:
ValueError: need at least one array to concatenate

When I trained a model with my own dataset, I met this error.
My data was originally in the VOC format labeled by labelimg. After encountering this problem, I converted it to the COCO format, but the problem remained unresolved.
What should I do?
Traceback (most recent call last):
File "tools/train.py", line 108, in
main()
File "tools/train.py", line 104, in main
logger=logger)
File "/home/atr5-1207/mmdetection/mmdet/apis/train.py", line 60, in train_detector
_non_dist_train(model, dataset, cfg, validate=validate)
File "/home/atr5-1207/mmdetection/mmdet/apis/train.py", line 232, in _non_dist_train
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/home/atr5-1207/anaconda3/lib/python3.7/site-packages/mmcv/runner/runner.py", line 358, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/atr5-1207/anaconda3/lib/python3.7/site-packages/mmcv/runner/runner.py", line 260, in train
for i, data_batch in enumerate(data_loader):
File "/home/atr5-1207/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 278, in iter
return _MultiProcessingDataLoaderIter(self)
File "/home/atr5-1207/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 709, in init
self._try_put_index()
File "/home/atr5-1207/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 826, in _try_put_index
index = self._next_index()
File "/home/atr5-1207/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 318, in _next_index
return next(self.sampler_iter)  # may raise StopIteration
File "/home/atr5-1207/anaconda3/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 200, in iter
for idx in self.sampler:
File "/home/atr5-1207/mmdetection/mmdet/datasets/loader/sampler.py", line 63, in iter
indices = np.concatenate(indices)
File "<array_function internals>", line 6, in concatenate
ValueError: need at least one array to concatenate

#210

authors: hellock jzchenriver


text:
How to retrieve bbox predicted by Faster RCNN model?

As mentioned in the header, how can I retrieve the bbox information predicted by my model for each image? I am trying to build an open set recognition pipeline (stage 1 to localise the object to be detected and stage 2 to classify the detected objects).

answer can be found here #5951 , closing this.


text:
[Enhance] support NMS for DETR network in test mode

DETR by default outputs a fixed number of bboxes during inference.
It doesn't run NMS when setting test_cfg as other networks do (say ATSS)
What I have changed:

add nms parameters in test_cfg in mmdet/configs/detr/detr_r50_8x2_150e_coco.py
running multiclass_nms() after get_bboxes() (in detr_head.py:simple_test_bboxes())

Note:

This modification won't affect training.
It applies a default set of parameters for NMS if they are not set explicitly

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
DETR by default outputs a fixed number of bboxes during inference. Developers might want it to run NMS
during test time (by setting test_cfg)
Modification

add nms parameters in test_cfg in mmdet/configs/detr/detr_r50_8x2_150e_coco.py
running multiclass_nms() after get_bboxes() (in detr_head.py:simple_test_bboxes())

BC-breaking (Optional)
Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

We follow the original implementation, NMS is not an essential post-process for DETR, you can refer to the original paper

We follow the original implementation, NMS is not an essential post-process for DETR, you can refer to the original paper

well, I am closing the request

authors: jshilong viibridges


text:
[Feature]: support keep ratio resize

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
Please describe the motivation of this PR and the goal you want to achieve through this PR.
Modification
Please briefly describe what modification is made in this PR.
BC-breaking (Optional)
Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

@RangiLyu LGTM

authors: RunningLeon wanghonglie


text:
[Feature] Add once dataset support

Motivation
Add ONCE Dataset support, to solve https://github.com/open-mmlab/mmdetection3d/projects/2#card-80056273, fix #1353, fix #2259
Modification
Please briefly describe what modification is made in this PR.
Use cases (Optional)
Running once_data_prep function to generate data dict ususally takes around 5-10min, and cost of prep data storage is around ~1G (raw small data ~63G).
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects.
The documentation has been modified accordingly, like docstring or example tutorials.

Why there lots of commits? I developed this originally on master branch. Should I merge to master branch?

Preliminarily test pointpillars model and get results:
|AP@50       |overall     |0-30m       |30-50m      |50m-inf     |
|Car         |63.57       |72.03       |58.73       |46.36       |
|Bus         |62.62       |67.98       |69.09       |47.25       |
|Truck       |33.19       |44.50       |36.16       |18.84       |
|Pedestrian  |14.57       |16.31       |12.56       |9.91        |
|Cyclist     |38.82       |49.32       |32.79       |20.86       |
|mAP         |42.55       |50.03       |41.87       |28.64       |

a little dropped compared with official benchmark (mAP 42.55 to 44.34).
I will not test other models/algorithms, please give a review and ask me to fix codes.

@OrangeSodahub  Thanks for your PR and benchmark experiments. Please fix the lint.

@OrangeSodahub Hi， please fix the lint.

@OrangeSodahub Hi， please fix the lint.

fixed

I'll finish this recently

Results after 80 epochs training PointPillars using 2 RTX3090 GPUs:
|AP@50       |overall     |0-30m       |30-50m      |50m-inf     |
|Car         |64.54       |74.32       |59.80       |47.18       |
|Bus         |59.52       |63.94       |63.79       |46.73       |
|Truck       |34.98       |47.40       |37.29       |19.97       |
|Pedestrian  |15.93       |18.56       |12.36       |9.45        |
|Cyclist     |41.28       |51.67       |35.63       |23.06       |
|mAP         |43.25       |51.18       |41.77       |29.28       |

Training log here. Configs:
_base_ = [
    '../_base_/models/hv_pointpillars_secfpn_once.py',
    '../_base_/datasets/once-3d-5class.py',
    '../_base_/schedules/schedule_2x.py',
    '../_base_/default_runtime.py',
]

optimizer = dict(type='AdamW', lr=0.003, weight_decay=0.01)
optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=1000,
    warmup_ratio=1.0 / 1000,
    step=[48, 60])
momentum_config = None
# runtime settings
runner = dict(type='EpochBasedRunner', max_epochs=80)


text:
[Bug] bugs met in test benchmark

Prerequisite

[X] I have searched Issues and Discussions but cannot get the expected help.
[X] I have read the FAQ documentation but cannot get the expected help.
[X] The bug has not been fixed in the latest version (master) or latest version (1.x).

Task
I'm using the official example scripts/configs for the officially supported tasks/models/datasets.
Branch
1.x branch https://github.com/open-mmlab/mmediting/tree/1.x
Environment
MMEditing 1.0.0rc1
Reproduces the problem - code sample
python .dev_scripts/test_benchmark.py
Reproduces the problem - command or script
python .dev_scripts/test_benchmark.py
Reproduces the problem - error message

basicvsr-pp: mirror sequence bug
sisr-test: psnr crop_border bug

Additional information
No response

Fix in https://github.com/open-mmlab/mmediting/pull/1293 https://github.com/open-mmlab/mmediting/pull/1304 and https://github.com/open-mmlab/mmediting/pull/1319

repo: open-mmlab/mmediting


text:
Expelliarmus article

Write article on Expelliarmus with benchmarks and so on once @stevenabreu7 has run those on his Metavision machine.

first version available here (set to draft=true) https://github.com/open-neuromorphic/open-neuromorphic.github.io/tree/main/content/post/file-reading-benchmarking

MAY I SAY THAT EXPELLIARMUS IS REALLY COOL?

authors: biphasic fabrizio-ottati


text:
onClose is not called for react whispers

LDK: Version 3.9.0
The onClose handler is not being called for react whispers. Here is a very simple test that I set up:
    await ReactWhisper.renderNewWhisper(
      <oh-whisper label="test" onClose={() => console.log('closing whisper')}>
        <oh-markdown body="test" />
      </oh-whisper>,
    );

When I close the whisper, here is the output in logs:
021-12-29 10:40:37.872] [ui] [debug] sending tracking event
[2021-12-29 10:40:37.883] [ui] [debug] sending tracking event
[2021-12-29 10:40:37.000] [desktop-core] [debug] whisper closed interactionType=close whisperType=universal
[2021-12-29 10:40:58.000] [desktop-core] [debug] Sidenote core loop listener called
[2021-12-29 10:40:58.000] [desktop-core] [debug] matched 0 notes for keyword
[2021-12-29 10:40:58.000] [desktop-core] [debug] on-boarding loop clipboard listener called
[2021-12-29 10:40:58.475] [ui] [debug] sending tracking event

Using the same test without react, my log statement is just after
[2021-12-29 10:40:37.000] [desktop-core] [debug] whisper closed interactionType=close whisperType=universal

the code is fairly abstract so I'm unsure without actually pulling it down, but I think I traced the bug down in WhisperRenderInstance
  private async createWhisper(whisperData: NewWhisper) {
    this.whisper = await create({ ...whisperData, onClose: this.handleOnClose });
    this.status = RenderInstanceStatus.Created;
  }

That method sets onClose in the NewWhisper config to this.handleClose which does call this.onCloseHandler?.(); but the only place onCloseHandler is set is in WhisperInstanceWrapper's constructor, and it's set to just do this:
  handleWhisperClose = (): Promise<void> =>
    new Promise((resolve) => {
      this.reconciler.updateContainer(null, this.container, null, resolve);
    });

I'm not sure if there is a good way for you to pull the onClose prop off of the ReactElement or not, but if not maybe an easy way to fix this would be to add a second param to renderWhisper which can be set to onCloseHandler , on the render instance, then the render wrapper can do something like..
renderInstance.setOnClose(async () => {
   await this.handleWhisperClose();
   renderInstance?.onCloseHandler();
});

Or maybe I'm going about this all wrong from my limited exposure to the code

Thanks for reporting this! I've created HELPS-2766 in our Jira, to track a fix for this.

@ssteuteville Sorry, I don't think I meant to close this issue (I think it happened when I merged your PR). Anyways, I just wanted to let you know that we did end up releasing version 3.9.2 earlier, specifically with your changes. Whenever you can, please go ahead and use that in your loop, and let me know if things are fixed for you. 🤞

@ssteuteville Great to hear! And no problem!

test

authors: iangallOlive ssteuteville thelunchbox

repo: open-olive/loop-development-kit


text:
[SIDE-984] Add Initial C# LDK

This PR adds the initial work on the C# LDK:

Stands up the Loop Server (with GRPC Plugin and Stdio services).
Receives Loop Start message.
Implements the Clipboard Service.
Implements the Whisper Markdown and Whisper Confirm messages.
Implements the Logger.
An example program that echoes the Clipboard content in a whisper.

This is not the complete LDK implementation, outstanding, and for implementation in future PRs are:

All services excluding the Clipboard service.
Implementing the Whisper service's WhisperForm message.
Documentation.
Other work necessary for publishing to NuGet.
Unit Tests

@dmorrison-olive I've merged this in but I'd still love to get your review.

You got this working fast. Glad you know what you're doing.

Hey I only kinda know what I'm doing maybe #notAnCSharpExpert

@dmorrison-olive I've merged this in but I'd still love to get your review.

Sounds good! I have it on my list to take some time and review this. Should help me to better understand the LDK and overall Olive Helps architecture also. 👍

authors: dmorrison-olive richseviora


text:
Should the plugin work with Istio 0.8?

We've been unable to get quick start to work with Istio 0.8.  opa-istio is listening, so the issue seems to be with making Istio use that for checks.
The quick start docs include a broken link to https://istio.io/docs/reference/config/istio.mesh.v1alpha1.html#MeshConfig
I can only guess that that pointed to the 0.7 docs: https://archive.istio.io/v0.7/docs/reference/config/istio.mesh.v1alpha1.html#MeshConfig
Any advice on how to get this to work on 0.8 or is there a recommended version?

@bluecamel Thanks for filing the issue. Yes the broken link was to the 0.7 docs which was the last version we tested the plugin with. We are working on getting the plugin to work with Istio 1.0 as described in https://github.com/open-policy-agent/opa-istio-plugin/issues/44.

@ashutosh-narkar Thanks so much for the info.  We'll give it a go with 0.7.

authors: ashutosh-narkar bluecamel

repo: open-policy-agent/opa-istio-plugin


text:
Does hostboot only read Lid files from running DIR？

https://github.com/open-power/hostboot/blob/c9a493fcc2ee016a52ba30b1d252d2dc0efd5688/src/usr/pldm/requests/pldm_fileio_requests.C#L214
What is the relationship between PLDM_FILE_TYPE_LID_RUNNING and the fw of BMC's fw_boot_side_current attr？

The BMC code is responsible for giving Hostboot all of its data.  Hostboot only ever wants the code from the currently running side.  We have deliberately obfuscated if that is 0/1 from the Hostboot layer.  The BMC and PHYP (and theoretically OPAL) do need to keep track of which physical side is in use though, I believe that is what fw_boot_side_current tracks.  More details would need to come from the OpenBMC team.

repo: open-power/hostboot


text:
Slotcar plugin package move

Move the slotcar plugin from rmf_building_sim_* to rmf_robot_sim_*.

I'm getting symbol lookup errors in ignition for compute_ds:
[ign-14] ign gazebo server: symbol lookup error: /home/luca/demos_ws/install/rmf_robot_sim_common/lib/libslotcar_common.so: undefined symbol: _ZN17rmf_plugins_utils10compute_dsEdddddd

Investigating a fix now

#6 fixes the startup crash, as well as cleaning up utils a bit

authors: gbiggs luca-della-vedova


text:
Fix: syntax highlighting and formatting update

Adds proper JSON syntax highlighting and updates formatting form 4 width tabs to 2 spaces.
fixes #22

:tada: This PR is included in version 1.0.0 :tada:
The release is available on:

GitHub release
1.0.0

Your semantic-release bot :package::rocket:

authors: mattbford openrpc-bastion

repo: open-rpc/logs-react


text:
Can't contact OTEL agent daemon set from other PODs without using hostNetwork

when configuring the environment variables as
- name: HOST_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  name: http://${HOST_IP}:4318

opentelemetry-collector chart does not allow to set hostNetwork for the DaemonSet which is both useful and required when using some CNIs like Cilium
PR created here

Feels like a great feature add to me.

@dmitryax this issue can be closed.

authors: TylerHelmuth carlosjgp


text:
ci: [Bug] DNS Lookup and Promise Lookup test for error NOT_FOUND failing #1136

Which problem is this PR solving?
[Bug] DNS Lookup and Promise Lookup test for error NOT_FOUND failing #1136
Short description of the changes
DNS resolution in local environment is exceeding 2 seconds (due to lots of fallback DNS servers) for invalid URL this is causing these 2 tests to timeout with the default 2 second delay.
Resolution:
Extend the timeout for these 2 tests so for environments where DNS resolution can take longer than expected the tests still pass.

Seems fine as a short term solution but wouldn't it be better to mock the DNS and not rely on a slow external service?

I suspect most people won't hit this issue so I think mocking could be overkill.

repo: open-telemetry/opentelemetry-js-contrib


text:
Is React specific SDK available?

Hi,
Can I use this JS SDK in React application or any alternate SDKs available ?

There is a tracer for the web (see here) that you can use. I'm not sure what you are refering to for a "react specific" sdk

@vmarchaud  Thanks I will check that. Seems the npm modules are bit specific for node environment, So I asked about the web front-end support.

I will close the issue, feel free to re-open if you have other questions

authors: Ajmal-M-A vmarchaud

repo: open-telemetry/opentelemetry-js


text:
Would a PR for plugin support for Logging Libraries (winston, bunyan, pino, etc) be accepted?

Hello,
Just wondering whether, if I ported over the plugin/instrumentation for some of the logging libraries in dd-trace-js, such as winston or bunyan, would a PR be accepted against opentelemetry-js-contrib?
Winston, for example: https://github.com/DataDog/dd-trace-js/blob/master/packages/datadog-plugin-winston/src/index.js
more info: https://datadoghq.dev/dd-trace-js/interfaces/plugins.winston.html
Essentially all it's doing is injecting the trace_id and span_id into the json logs emitted by these logging libraries. This helps users correlate logs emitted by their application with traces that were active at the time the log was emitted. The dd-trace-js implementations have some vendor specific details that would need to be to abstracted away/modified (so instead of injecting keys like dd.trace_id it could be otel.trace_id or <arbitrary_user_supplied_option_with_reasonable_default>.trace_id But otherwise the use case has value for any user and is relatively vendor agnostic imo. So i'd love to have it be something anyone can find in js-contrib, but i'm not sure if the behavior it achieves would too significantly off-spec to have it land there. In which case, no worries, can always host elsewhere and try to just have it referenced in the otel registry.
Is there currently anything in the Otel Spec for this?

I have not seen anything in the spec. Previously, @obecny had created a similar plugin PR, but it was not merged at that time.
The issue I have with such a plugin is that it modifies user program behavior by modifying log output. At the very least, I would not want this installed/enabled by default without some user intervention saying that yes they do want a behavior change. Some logging libraries (including winston I think) already have support for plugins/formatters that can inject fields. Maybe it would be better to release a winston Format which uses the public API methods to inject fields?

Maybe we can get some guidance from the @open-telemetry/technical-committee here?

FYI, I still have it in case there is desire to have it so it will be a matter of read that and fixing conflicts ;)

According to the readme, http and grpc plugins are included directly in this repo, while all others are in contrib. I can understand http if it's based on stdlib, but why was grpc singled out?
I would recommend placing all plugins (unless they are for stdlib) into contrib.

@yurishkuro http(s) is language built-in so it is in main, and gRPC from what I understand was identified early as an important plugin that MUST be supported by all SDKs, so it was also kept here. It can easily be moved to contrib though if it doesn't have the special status that I thought it did.
The question here though is about logging plugins. Is it acceptable to have an autoistrumentation which modifies logging output to include trace ids? If it is acceptable, should it be included in our "all plugins" package?

@yurishkuro fwiw i only opened the PR here for visibility, i didn't really intend to broach the topic of where all the plugins should live.
I tend to agree with @dyladan here.

it modifies user program behavior by modifying log output. At the very least, I would not want this installed/enabled by default without some user intervention saying that yes they do want a behavior change. `

If logging plugins were to land in js-contrib, they should be something that's explicitly enabled. This is the case in dd-trace-js currently fwiw (you pass in config flag or env var to enable).
That being said, if there's a reasonable way to do this already that doesn't end up bumping up against Spec, or fits within an existing otel convention, i'm fine with that approach rather than having something that's problematic, even if it requires slightly more user configuration (one of the benefits of a plugin is that user config is as simple as a config flag/env var). I think what you mentioned re: winston format is a very reasonable (i'll admit i need to learn a bit more about it and also investigate if bunyan has something similar). If that's the approach that's preferred i'd be happy to document the appropriate config/formatting required, but it would be nice to have this land somewhere such as /docs where users can discover it

but it would be nice to have this land somewhere such as /docs where users can discover it

Or the registry?

In bunyan it is very easy. We would have something like this:
import { trace } from "@opentelemetry/api";
import { format } from "logform";

const tracer = trace.getTracer("@opentelemetry/logform", "0.3.2");

/*
 * Returns a new instance of the OpenTelemetry Trace Context Format which adds an OpenTelemetry Trace Context
 */
export const TraceContextFormatter = format((info) => {
    const { traceId, spanId, traceFlags } = tracer.getCurrentSpan().context();
    info.traceContext = {
        traceId,
        spanId,
        sampled: !!(traceFlags & 0x1)
    };
    return info;
});

And usage would look like this:
import { TraceContextFormatter } from "@opentelemetry/logform"
import { createLogger, transports, format } from "winston";

const logger = createLogger({
    format: format.combine(
        traceContextFormatter(),
        format.json()
    ),
    transports: [new transports.Console()],
})

logger.info("some log message")
// OUTPUT: {"message":"hey","level":"info","traceContext":"[traceId=0 spanId=0 sampled=0]"}

I haven't tested this code, but it should require only very little if any modification to get working.

        traceContextFormatter(),

This is README worthy imho, and examples for pino and winston would be nice as well.
What's !!(traceFlags & 0x1) do btw?

I know that this was recently merged in the specs, don't know if it should apply to 3rd party logger or only to the future logs-sdk

That bunyan example is docs worthy imho, and examples for pino and winston would be nice as well.
What's !!(traceFlags & 0x1)is a tad obscure imho, what's it do exactly?
is !! essentially Boolean()?

!! casts to Boolean yes. It is a slightly more esoteric way to do it I guess. ! is not, so !! is not-not.
traceFlags & 0x1 checks to see if the first bit is set by doing a bitwise AND operation on traceflags and 00000001. The idea is that the trace flag is only the first bit, so any traceFlags value with that bit set should still return true such as 1, 3, 5, etc (any odd number has the first bit set).

I know that this was recently merged in the specs, don't know if it should apply to 3rd party logger or only to the future logs-sdk

The idea behind the data model is that you should be able to convert any existing known logging data and convert it to our own data model so it can be ingested by backends. It should be losslessly convertible in both directions (conversion from source logs like syslog to our model and back to the source format should be "perfect").

BTW I am today doing something similar with https://github.com/pinojs/pino
I leave the traceFlags key as provided from context though, @dyladan any particular reason you modify it to sampled ?
import tracer from './tracer'
import * as pino from 'pino';

export default pino({
  formatters: {
    log(object) {
      return {...object, ...tracer.getCurrentSpan()?.context()};
    },
  },
});

Also worth noting, for the http request logger https://github.com/pinojs/pino-http I also replace reqId with spanId as such:
import tracer from './lib/tracer';
import logger from './lib/logger';
import * as pinoHttp from 'pino-http';

app.use(
  pinoHttp({
    logger,
    genReqId: () => {
      const spanId = tracer.getCurrentSpan()?.context().spanId;
      return spanId ?? 'no span id!';
    },
  })
);

@dyladan any particular reason you modify it to sampled ?

I think sampled=1 is more readable to most people than TraceFlags=1. Sampled doesn't require previous knowledge of Trace Context spec and the associated TraceFlags.

As we have winston, bunyan, pino instrumentations now I'm going to guess the answer was yes and this can be closed? @dyladan

authors: dyladan ericmustin mhennoch naseemkullah obecny vmarchaud yurishkuro


text:
fix(otlp-grpc-exporter-base): avoid TypeError on exporter shutdown

Which problem is this PR solving?
Calling shutdown() on any gRPC exporter throws a TypeError - preventing apps from gracefully shutting down.
The transport should call close() instead of shutdown() on the _client. The test I wrote for this case was entirely ineffective as I also managed to mock the wrong thing.
Fixes #4611
Short description of the changes

call this._client?.close() instead of this._client?.shutdown()
adapt tests to actually test that behavior

Type of change

[x] Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

[x] Updated unit tests

Looks good but I'm curious how the types allowed shutdown to be called if it doesn't exist in the first place. Probably not important for this PR, but our GRPC types might be wrong

Yes, I've been using any instead of Client from the gRPC library as there are no types for the service client functions itself.
I've pushed a commit to use the Client type.
We still have to use it as any when we call export(), but now at least the common Client functions are now typed.

hello @pichlermarc,
Thanks for your fix (I need it to force flushing BatchSpanProcessor when my script ends, on beforeExit event)
Can you please publish this new version ?
so that @Vunovati can update his PR https://github.com/Vunovati/otlp-logger/pull/61 to release new versions.

hello @pichlermarc,
Thanks for your fix (I need it to force flushing BatchSpanProcessor when my script ends, on beforeExit event)
Can you please publish this new version ? so that @Vunovati can update his PR Vunovati/otlp-logger#61 to release new versions.

On it, hoping to get the release out by monday :slightly_smiling_face:

authors: jdespatis pichlermarc


text:
Implement initial XRay propagator implementation

This PR adds an injector and extractor for AWS XRay context propagation.
/cc @alolita

@fbogsany could you take another look 🙏🏾 ?

It is not obvious and we often forget, but you'll need to add the new propagator to the releases.yml file here: https://github.com/open-telemetry/opentelemetry-ruby/blob/2d2208ee0d44c191076e3d048d1f91f0b006571c/.toys/.data/releases.yml#L145-L155

@fbogsany I've resolved the remaining issues. Things look good now

👋🏼 @SamirHafez thank you very much for this contribution.
I have posted some questions for you in that I was hoping you could help clarify since I am not familiar with XRay.
One thing I am curious about is if XRay has anything that maps to Baggage. I do not see anything about it in the other implementations but I see something in the OpenTracing libraries. Perhaps it is something to consider in the future:
https://github.com/opentracing-contrib/java-xray-tracer/blob/c8cc506679be76604b91fa369bc6f87ce5942f87/src/main/java/io/opentracing/contrib/aws/xray/AWSXRayTracer.java#L59

I did consider that when I started this PR but ultimately decided to keep this first crack simpler. I was hoping that support could be expanded in the future.

authors: SamirHafez fbogsany

repo: open-telemetry/opentelemetry-ruby


text:
Explain why do we need a tools package

https://github.com/open-telemetry/opentelemetry-service/tree/master/tools
Can this be removed?

@owais can you shine on this? I don't remember what for was this exactly.

Also if this is internal consider to move it under the internal directory.

It was added so we could pin tool dependencies to go.mod and have better guarantees about everyone using the same version of the tools. It doesn't have to be a separate package and can be moved to root.
More on it here: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module

Probably can be in the internal directory. Trying to avoid having too many packages/files in the root if not necessary.

Makes sense. Will move to internal.

Also please add a comment in the file why is it needed. That ensures others will not get confused like me.

Will add the comment. Do you think keeping it in internal (as opposed to root) might incorrectly suggest that the tools are only required by the internal package?

I don't, I think we have no intention to export public anything from this file so it should not be in the root.

authors: bogdandrutu owais tigrannajaryan

repo: open-telemetry/opentelemetry-service


text:
[wip] - Update ruby docs to latest

Pulls in some improvements and a new article on configuring instrumentation libs

PR that fixes the build: https://github.com/open-telemetry/opentelemetry-ruby/pull/1081

Weird, so I can actually run npm run serve on this branch and the site builds and is browsable with the new content. I guess it's to do with the rest here: https://github.com/open-telemetry/opentelemetry-ruby/issues/1104

To check links run npm run check-links.

We'll want to wait for https://github.com/open-telemetry/opentelemetry-ruby/pull/1120 before updating and merging this PR.

Superseded by #1099

authors: cartermp chalin


text:
Community: add otel project roadmap to website

Fixes #2167: this is a copy of the roadmap document from the community repo, similar to what we have with the "Mission, Vision & Values" page we make it easier to find by that.
This PR also includes some reordering of the navigation tree in /community
cc @open-telemetry/governance-committee

Note:
This PR is only copying the roadmap.md from the community repo. Any changes to the text itself need to be made there first.

Would the lambda work fall under continue to invest in otel artifacts @svrnm ?

Would the lambda work fall under continue to invest in otel artifacts @svrnm ?

@cartersocha this PR here is only concerned with copying the roadmap from the community repo to the website, so ideally you check with the @open-telemetry/governance-committee at https://github.com/open-telemetry/community

I'd like to reiterate my question from https://github.com/open-telemetry/opentelemetry.io/pull/2176#pullrequestreview-1262814901:

Is the plan to remove open-telemetry/community/roadmap.md once this lands?

If the answer is no, then we should bring in the roadmap page along with the mission page through the work of #1809. I'd be glad to set that up if you'd like. Although I guess that since we already have a duplicate of the Mission page, we're not setting a precedent of making a duplicate of the roadmap.
I'd be ok with merging this as is. I (or you) can then work on #1809 ASAP. Thoughts?

FYI,

#2239

has landed as a "quick fix" in support of

#2237

until we get the actual roadmap into the website (through this or another PR).

Superseded by:

#2296

authors: cartersocha chalin svrnm


text:
perf(README): emotional intelligence framework

Personality- and Memory-based Framework for Emotionally Intelligent Agents
N.B.: draft PR for the time being; arXiv link will be added after RO-MAN is over, where it is being presented.

Thanks for the suggestion. Please consider creating a new PR when the paper is publicly available. :-)

@andreaskoepf sure, will do then!


text:
Raw UDP over IPv4 traffic with fixed source and destination ports

Config

Two back-to-back connected test ports

One for tx
Another for rx

One Flow with,

Fixed frame size of 128B
Rate of 50 PPS
Packet count 100
Packet headers ethernet, ipv4 and udp

UDP header with,

Fixed source port
Fixed destination port

Validation

Push configuration
Start capture on rx port
Start transmit on tx port
Wait until following is as expected,

Frames tx/rx, bytes tx/rx and transmit state in flow metrics

Fetch capture and ensure following is as expected,

ethernet.ether_type
ipv4.protocol, ipv4.total_length
udp.src_port, udp.dst_port, udp.length
size of each frame
(best effort) checksum of all headers

https://github.com/open-traffic-generator/conformance/blob/main/features/flows/headers/udp/test_udp_header.py
https://github.com/open-traffic-generator/conformance/blob/main/features/flows/headers/udp/udp_header_test.go

repo: open-traffic-generator/conformance


text:
feat/group validation

feat(form-control): add support for group validations
feat(form-control): add requiredCheckedValidator
chore: release

The static prop really just says that if the component is in a group and it enters into a valid state, to mark all instances within that group (same name/localName within a root) as similarly valid. You should ever only set that once. If a control (like a checkbox proxy component) is a standalone it can potentially be marked as part of a group and used as a solo component, no extra work will be done.

repo: open-wc/form-participation


text:
docs: change library name

As you can see here, the name of the project is UI5 Web Components. :blush:

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

hey,
thanks for that clarification 🤗
if you were to sign the CLA we can definitely merge it 👍

Signed :blush:

thank you for your contribution ❤

authors: CLAassistant daKmoR fifoosid

repo: open-wc/open-wc


text:
putTcpClientRemoteAddress example requires a change

As per the schema definition for the OAM service putTcpClientRemoteAddress , maximum allowed entry is 1. But, in the example , both ip-address/ipv-4-address and domain-name are present.
{
  "tcp-client-interface-1-0:remote-address": {
    "ip-address": {
      "ipv-4-address": "1.1.4.1"
    },
    "domain-name": "application-pattern.mw-domain.transport.ger.telefonica.com"
  }
}

Proposal :
Either we can keep only "ip-address/ipv-4-address" or  "domain-name".
Impact :
No impact in server side.
In the swagger GUI , an example responseBody with both "ip-address/ipv-4-address" and  "domain-name" will be present.

Change to an example on schema level, like:
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
                - tcp-client-interface-1-0:remote-address
              properties:
                tcp-client-interface-1-0:remote-address:
                  type: object
                  minProperties: 1
                  maxProperties: 1
                  additionalProperties: false
                  properties:
                    ip-address:
                      type: object
                      minProperties: 1
                      additionalProperties: false
                      properties:
                        ipv-4-address:
                          type: string
                          pattern: '^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$'
                    domain-name:
                      type: string
                      pattern: '^([a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,}$'
            example:
              tcp-client-interface-1-0:remote-address:
                ip-address:
                  ipv-4-address: '1.1.4.1'

Corrected in https://github.com/openBackhaul/ApplicationPattern/pull/522. Hence closing this issue

authors: IswaryaaS PrathibaJee openBackhaul

repo: openBackhaul/ApplicationPattern


text:
frame and iframe related rules have accessible names

descriptions for frame and iframes need to be supported based on screen readers using the descriptions to support page navigation, without descriptions screen readers often use confusing information for the accessible name.
Pattern similar for images:

Frame needs a name
If name is empty verify that no description is needed (test with AT)
If name is not empty manual check to verify description

Added intial rules in version 0.9.9.2


text:
us-pa-greene

http://www.co.greene.pa.us/secured/gc2/depts/ed/gis.htm Greene County GIS

Not available to general public. Property layer available but VERY expensive.

License says no re-distribution.

no data

authors: geobrando jalbertbowden justinelliotmeyers nvkelso

repo: openaddresses/openaddresses


text:
Nation-wide data for Canada available

Statistics Canada has released a nation-wide address database. Data available here: https://www150.statcan.gc.ca/n1/pub/46-26-0002/462600022022001-eng.htm. Licence is open (https://www.statcan.gc.ca/en/reference/licence). Release December 2023.
The data is provided in 2 seperate files - one for addresses and one for locations. These two share a common LOC ID field. Files are provided by province and territory (i.e. 13 address files, 13 location files). I've joined these files into one single address database, with 16 million records. Not familair with json so much but I can provide you what I have - or you can get it directly form the source.

Thanks for the suggestion!
In situations like this, we usually build a script to do the work, document it thoroughly so we can run it in the future, and then put it in https://github.com/openaddresses/openaddresses/tree/master/scripts. We don't have automation to run these scripts at this point, so someone runs the script manually, we post the output data somewhere (we can store it in our S3 bucket) and add a source JSON to this repository to incorporate the data into our overall output.
If you're up for it, can you write a pull request to add the script+documentation? You can upload the output data you have to https://www.dropbox.com/request/rUcmasTnxZkiDR6rJc4l and I will move it over to S3, too.

i could have done this wrong, but does this look right @pheersink

Pretty much what I get so yes it is ok. Coverage is inconsistent but there should be around 16.5 million addresses on the map.

Nice, looks like Canada!

@iandees you mean it looks like Canada 'eh!
@pheersink in the past we have collaborated with statcan to fill in gaps. I'd love to explore that option for them as well as ESRI and openaddresses. Little tricky with different municipal legal hoops, but we do this stuff a lot, so not hard, but takes effort. Thoughts about trying to fill in gaps up there 'eh?

I work with @pheersink on some of the address work.
I built a script to download and process the National Address Register data and opened PR https://github.com/openaddresses/openaddresses/pull/7224 with a script for it. It joins the address and locations files together on the LOC_GUID field and adds the REPPOINT_LATITUDE and REPPOINT_LONGITUDE fields to the addresses, and zips that up. I uploaded the result file to the Dropbox link that @iandees posted above with file canada-nar-combined.csv.gz

Thanks @jwass

Awesome @jacob
Sent from my iPhone
On May 11, 2024, at 12:43 PM, Justin Meyers @.***> wrote:

EXTERNAL
Thanks @jwasshttps://urldefense.com/v3/__https://github.com/jwass__;!!G_POQyfmwzJ5ZCxH!EUt_ltgIUcRi_CeX9ePNSFpTLwmvnHoX6-G7xLDb8GzMXXKBJpeQNs6betIWHpDaq3CpQQJQ-AZh9-FDafvKbrix$
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/openaddresses/openaddresses/issues/7192*issuecomment-2105950832__;Iw!!G_POQyfmwzJ5ZCxH!EUt_ltgIUcRi_CeX9ePNSFpTLwmvnHoX6-G7xLDb8GzMXXKBJpeQNs6betIWHpDaq3CpQQJQ-AZh9-FDaWcaOkUJ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AB5DDHLEMBGFV7327YBAKOLZBZDCZAVCNFSM6AAAAABHCH5C3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVHE2TAOBTGI__;!!G_POQyfmwzJ5ZCxH!EUt_ltgIUcRi_CeX9ePNSFpTLwmvnHoX6-G7xLDb8GzMXXKBJpeQNs6betIWHpDaq3CpQQJQ-AZh9-FDabUcLM95$.
You are receiving this because you were mentioned.Message ID: @.***>
Confidentiality Notice: This message may be confidential and is intended only for the named recipients. If you received this in error, please notify the sender and delete the message.
Avis de confidentialité: Ce message peut être confidentiel ; il est destiné uniquement pour les destinataires qui y sont désignés. Si vous avez reçu un message par erreur, nous vous prions d'en aviser l'expéditeur et d’effacer le message.
If you no longer wish to receive emails from Esri Canada, unsubscribe herehttps://www.esri.ca/en-ca/communications/unsubscribe. Privacy Policyhttps://www.esri.ca/en-ca/legal/privacy
Si vous ne désirez plus recevoir de courriels d’Esri Canada, veuillez vous désinscrire icihttps://www.esri.ca/fr-ca/communications/unsubscribe. Politique de confidentialitéhttps://www.esri.ca/fr-ca/legal/privacy
Esri Canada | 12 Concorde Place | Suite 900 | Toronto, ON | M3C 3R8

The data is available at https://data.openaddresses.io/cache/uploads/iandees/760039/canada-nar-combined.csv.gz

authors: iandees justinelliotmeyers jwass pheersink


text:
how to distill from CLIP to get a tiny model?

Dear all:
I want to retain a smaller model from CLIP network. However, when i try to run knowledge distillation from Vit32model, I found model.encode_image(image) get a vector of lenght 512, while model(image_feature, text_feature) network need the text input. I donot know how to get the soft label from CLIP. Anyone get some idea about how to distill knowledge from CLIP? thanks .

I guess there are different strategies depending on your goals. If you want to make a smaller vision model that encodes image into the feature space similar to CLIP's, so that you can use CLIP's robust representation with the faster encoder, I'd train the student encoder to minimize L2 or cosine distances to CLIP's features.
If your goal is to make a specific zero-shot image classifier faster with a predetermined set of labels, you can train your image encoder to minimize the KL divergence from the probability distribution resulting from the logits returned by the CLIP model.

@jongwook Thank you for the detailed tips. I finally get what you mean. Have you test training a mobile backbone using CLIP setting. Is it possible to get a reasonble zero-shot performance?

We don't have results on smaller model sizes than the released ones, and we haven't experimented CLIP on mobile devices.  ResNet50 or ViT-B/32 model requires only a few gigaflops for the forward pass, and you might not need distillation at all if it's fine for your use case.

@dragen1860 have you tried distilling CLIP's image-encoder? If yes, which one (ViT or RN based)?

Hello, Can you share me the weight after distill or the code about distill the clip model?  My email is 13816267895@163.com
Thank you very much!

@dragen1860 did you manage to make a succesful distillation?

repo: openai/CLIP

license: mit


text:
IDM Code

Thanks so much for making the code available, it's super readable! I don't think I see the inverse dynamics model, are y'all planning on releasing that as well?

Thanks for the kind words! It is not all there is but hope it is a good starter :)
@brandonhoughton Do you think we could do this at some point (share IDM model)? I expect the process to be much smoother after this one.

Appreciate the fast response! Thanks @brandonhoughton and @Miffyli!

Looking more likely - I'll close this when IDM code is public but no timeline yet

@kevinzakka The IDM code has now been added :)

authors: Miffyli brandonhoughton kevinzakka

repo: openai/Video-Pre-Training


text:
baselines/baselines/pposgd/pposgd_simple.py- clipping causing no gradient?

    surr1 = ratio * atarg # surrogate from conservative policy iteration
    surr2 = U.clip(ratio, 1.0 - clip_param, 1.0 + clip_param) * atarg #
    pol_surr = - U.mean(tf.minimum(surr1, surr2)) # PPO's pessimistic surrogate (L^CLIP)

atarg is just a tf.placeholder. So when the ratio term deviates from 1.0 by too much, isn't it causing the gradient to be zero? Because I think both 1.0 - clip_param and 1.0 + clip_param are not tf.Variables.

I think gradient being zero is the purpose. for ratio(ratio > 1-clip then atarg > 0, ratio < 1 - clip then atarg < 0) being enough, no need to optimize.

@CosmosShadow Thanks!
But I still do not understand. You are explainig the meaning of the - U.mean(tf.minimum(surr1, surr2)).
I know that, of course, say if atarg > 0, and ratio > 1.0 + clip_param, then ratio will be clipped, and of course, the objective cannot be optimized anymore, because the objective would be a constant.
My question does not lie on whether the objective can be optimized or not. Rather I just want to ask why is this form of the objective? Is it normal? Doesn't it cause the gradient to be zero for some cases, and hence this form is okay? Because it would bring the fact of gradient being zero, so I question its properness.
But anyway, thanks a lot for your analysis. ^_^

@lezhang-thu  I think it's exactly what the trust region means. When the change belows the upper bound 1+epsilon (when A > 0), we think this change beyond the trust region so we don't adpot this data (therefore the gradient is zero and parameters is not updated).

authors: CosmosShadow lezhang-thu yueyang130

repo: openai/baselines


text:
New functionalities

Currently, in Procgen, (1) one can not get the level id that is being played before the first action is taken. (2) Also, one can not set the level id unless the environment has only one level, and if you are dealing with many levels you may have to create the environment from scratch for each level. (3) Last, one can not specify an initial state other than the original initial state (e.g. a random valid state).
It would be cool to have these features in Procgen. Any comment on how to start adding these features would also be appreciated.
We need the third feature more urgently as we have workarounds for others. Would it be correct if we assign an arbitrary valid observation as the initial observation instead of using env.reset()?

I'm afraid I don't understand why you would want to do #3.  What constitutes a random valid state?  Is that different from a random initial state?

Actually, what I meant by a random valid state is a particular valid state from the state space. In my thinking, let's say in maze game, valid states would correspond to different positions of the agent in the map, changing walls or the cheese would be invalid.
It is different from any random initial state in the sense that I should be able to initiate the agent from a particular state of my interest. In other words, I want to observe the agents behavior when it starts playing from a particular state.

Oh, so is it sufficient to be able to save and restore environment state?  You need an agent to produce the states that you want to save.

Yes, this should do I guess. But when I restored a previously saved state reward should start from 0.

Some of your requests may be addressed by the next version of procgen though it won't be out for a few weeks.

OK thank you very much for the responses. I will either implement it by myself or find a workaround. I have another questoin related to initial states and Procgen.
From what I read, diversifying the initial states can help agent to learn better (e.g. human start in ALE etc.). Do you see any reason that random initial states can not help for

better training performance
better generalization performance

in Procgen environments?
My first experiments on this resulted with a much poorer training performance in Starpilot game. Currently, I am reviewing my code if there is a bug or something.

It probably helps more in ALE because there is basically only a single initial state.
In Procgen there is a fairly diverse set of initial states (1 per level seed) and the set of available states are subjectively more diverse.

But still this diversity may not be enough for generalization (e.g. 500 level generalization). I hypothesize that, if adding more levels helping generalization (as shown in the paper), then one can diversify the initial states of the existing levels to get a better generalization performance.
Could you also let us know if restoring states is likely to be included in the next version?

It's likely

For now, we have implemented a workaround in our own fork.

authors: christopherhesse hasanferit

repo: openai/procgen


text:
error applying manifests

I get this errors applying
kustomize build .  | kubectl apply -f -
unable to recognize "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "STDIN": no matches for kind "ShinyProxy" in version "openanalytics.eu/v1alpha1"

Hi
I seem to have missed this issue. The reason you get this, is that the CRD resource is not yet applied when you try to apply the other resources. The simple solution is to run the command a second time. By design there is not really a proper way to fix this using kustomize.
I hope this helps, let me know if you encounter any other issue with the operator!

authors: LEDfan behrica

repo: openanalytics/shinyproxy-operator


text:
Trouble with shiny bookmark feature

Hi,
I am having trouble running my shiny application which contains shiny bookmark function (usage of bookmark button) on shinyproxy.
My application uses bookmark button in shiny to save a specific state of the shiny application as an url so that the users are able to retrieve any scenarios they saved for future research.
When we use bookmark function in a shiny application it creates a directory called shiny_bookmarks (Which will be used by shiny to retrieve the state of the application once they click on the saved urls) within the directory where the code for the application is present but when I deploy it on Shiny proxy as it runs within the docker containers this folder gets lost once the user closes the application unable to retrieve it. Moreover it looks like a separate docker container for the application is being created per user.
So, I am assuming that if I am able to gather these local (to the container) shiny_bookmarks folder  from all the instances of the container whenever a user bookmarks something and store them at a common place (like server or s3 bucket) and update the local folder within the container by copying it back from the common place to the container being used whenever some user tries to retrieve a saved url then it solve my issue.
I am not well experienced with docker containers. What will be the ideal way to go about it or is there any function within shinyproxy that can help with the situation. I am attaching an example code for bookmarking below incase if it help understand my problem better.
Any help is much appreciated.
Thank you ..!!!
`
ui <- function(request) {
fluidPage(
sidebarPanel(
sliderInput("n", "Value to add", min = 0, max = 100, value = 50),
actionButton("add", "Add"), br(), br(),
bookmarkButton()
),
mainPanel(
h4("Sum of all previous slider values:", textOutput("sum"))
)
)
}
server <- function(input, output, session) {
vals <- reactiveValues(sum = 0)
Save extra values in state$values when we bookmark
onBookmark(function(state) {
state$values$currentSum <- vals$sum
})
Read values from state$values when we restore
onRestore(function(state) {
vals$sum <- state$values$currentSum
})
Exclude the add button from bookmarking
setBookmarkExclude("add")
observeEvent(input$add, {
vals$sum <- vals$sum + input$n
})
output$sum <- renderText({
vals$sum
})
}
shinyApp(ui, server, enableBookmarking = "url")
`

Closing the issue as I was able to resolve it using container-volumes option in the config file.

repo: openanalytics/shinyproxy


text:
Preferences edits

Mostly improved description of how safety multipliers work, with a table of examples.

Ugh. So embarrassing. I hadn't seen those preferences at the bottom of the document (tl;dr). The note at the top will be super helpful.

No worries @parenthetic! Thanks so much for this. Please keep them coming! :)

De nada. Thank you!!!
On Fri, Sep 29, 2017 at 2:31 PM, Dana Lewis notifications@github.com
wrote:

No worries @parenthetic https://github.com/parenthetic! Thanks so much
for this. Please keep them coming! :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openaps/docs/pull/1039#issuecomment-333203706, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AUVFUTtF6j9lclhYgASj3IX5_Futt06fks5snTeOgaJpZM4PaUf3
.

authors: danamlewis parenthetic

repo: openaps/docs


text:
IPMI Test use cases for Basic IPL

IPMI Power ON,
Power OFF,
Power Cycle
Power Soft,
Power Reset,
BMC Reset
BMC Reboot
IPMI Warm/Cold MC Reset  <-- This is in review

@manashsarma Once you are done with other. Here is next stuff you can pick up.

@manashsarma : We have below IPMI commands related to power functions. Out of which "status, on, off" is already done. So you can automate "cycle, reset, diag and soft".
chassis power Commands: status, on, off, cycle, reset, diag, soft
You can start automation in below order.

Soft - Host OS should shutdown softly.
Reset - BMC should reset with this command.
Cycle
Diag

Good input @rahulmah
@manashsarma try to use minimum REST code as much as possible. 80% of the code is REST based. We want this to be logistically IPMI only codes related

'Diag' test case not added as development team replied that 'Diag' is not supported on my query to them.

So it's not supported or going to be supported? If it's going to be supported, go ahead and open one task for it.

authors: gkeishin manashsarma rahulmah

repo: openbmc/openbmc-test-automation


text:
Recursive capture

can u make capture like sentry
like this
type1=xxxx expired1=xxxx, type2 =xxxx expired2=xxxxxx
OB capture
type1=xxxx type2=xxx expired1=xxxx expired2=xxxx
hope u understand. tq

When you make a recursive parse, it will make a LIST variable. You can zip or map LIST variables together like this to have a clean output:
UTILITY List "First" Zip "Second" -> VAR "NewList"
UTILITY List "First" Map "Second" -> VAR "NewDict"

@openbullet how to add more list?  3 4  var?

UTILITY List "Producto" Map "ActivationCode" -> CAP "1"
UTILITY List "ActivationCode" Map "DaysReminder" -> CAP "2"
want to combine cap1 and cap2

repo: openbullet/openbullet


text:
[BUG] Proxy checker not working

Look pickture it explain all

This is the OB1 repository, you're on the wrong repo

authors: ChristianErdtmann openbullet


text:
Track example for students

Sometimes instructors create tracks with example annotations to show students when to use which annotations. Instructors should be able to specify tracks which are always visible upon loading the annotation tool.

In a second step it would be desirable to add an export / import function for tracks.

Our users did ask for the same feature. Here some specification we had in mind:

Option «Show track all users» available under "Update/Edit track" > "Access" AND as own icon under "Change visibility
Sub-options of "Show all users" (only appears when "Show all users" is selected):
a) Writable: Users can make additions to the track, but not changes.
b) Non-writable: Users cannot make any changes to the track.
showing a track to all users can be undone.
When the option «Show track all users» is checked, a copy of the selected track is created for all users.
Changes made by Admins to the track displayed for all users are automatically synchronized to the users. This may require a reload of the browser.
Offer Admin the option: how should the editable track be set in terms of public/privat/shared with Admin

@dagraf
First mockup of the UI parts:

In German it says "Allen Benutzern anzeigen."
By the way, shouldn't the English text be "Show to all users?"

@dagraf do you have any preference about the order the tracks are shown in by default? The "example tracks" should probably go on top, right?

@dagraf do you have any preference about the order the tracks are shown in by default? The "example tracks" should probably go on top, right?
You're right: "Show to all users" is what it does. Please change english wording.

Some specs/questions:

Please change order: Shown to all users / Public / Shared with administrator (please rename) / Private
Would it be possible to add some explanation below the new option? For example: "The track and all containing annotations made by administrators will automatically be displayed to all users."

authors: JulianKniephoff dagraf ebbertd

repo: opencast/annotation-tool


text:
docker secrets?

Is it posible tu use docker secrets instead of environment params for sensitive data?

Maybe, something like this (https://medium.com/@basi/docker-environment-variables-expanded-from-secrets-8fa70617b3bc) can be added to the images.

The available environment options are more meant for quit tests rather than production deployments. In the latter case, you would rather mount Opencast's configuration files into the container or use secrets and link to them in the configuration folder. Regardless, I agree that we could add support passing secrets in this manner.
The official images from Docker use the patter <var>_FILE to point to a file that contains the value for the environment variable <var> (see mysql, mongo, postgres). I would suggest to stick to that pattern.
I will see, when I can implement this feature. It should be easy to implement.

#102 implements this feature. It is available in latest, 5.2, and next images.

authors: miesgre mtneug

repo: opencast/opencast-docker


text:
Unable to insert new OpenCFP\Domain\Entity\TalkTag - Errors: array ( )

Steps required to reproduce the problem

Try to submit a new talk to https://opencfp.phpkonf.org

Expected Result

See a thank you page / returned to list (can't remember what the success case is)

Actual Result

See a white page with the text Unable to insert new OpenCFP\Domain\Entity\TalkTag - Errors: array ( ) at the top.
Note that the talk appears to have submitted correctly as manually navigating to https://opencfp.phpkonf.org/dashboard shows the talk on the list.

Getting the same error! Any ideas on how to fix it?

I'm getting this too! haven't looked into the cause yet though.

Closing because I have removed tags from being part of talks, will eventually add in a migration to drop the impacted tables as well.

authors: akrabat andrewwoods anshul35 chartjes

repo: opencfp/opencfp


text:
Re-adding Sorting and Rating UI

Per issue: https://github.com/opencfp/opencfp/issues/285
Sorting UI was removed and a different one added instead. I don't believe this was done on purpose, but merely product of an older version of OpenCFP being worked on. This PR re-adds the sort UI that has sort direction arrows.
Also, the up and down vote rating icons are being re-added in this PR.

Sorry about that @jakefolio. Hope all is well. Thanks!

authors: jakefolio mdwheele


text:
Add 4 hour horizon

Add 4 hour forecast onto both national and GSPS graphs. This will show the forecast made 4 hours before target_time. This is useful for the control to see how the 4 hour forecast is doing.
Done: This needs work in the API to expose some new data. @peterdudfield needs to do this
Also need to define the colour of the line on the map. We are thinking some sort of blue.

Colour: could be 50% opacity yellow, or light blue
it should behind the yellow and black line

API is now ready

protoytype from @braddf is here

repo: openclimatefix/nowcasting


text:
useAuth: The Login Gatekeeper

Task Details:
Your mission is to create a custom useAuth hook, a powerful tool for managing user authentication. Here are the specific details:
Details
Craft a useAuth hook that utilizes useContext to determine the login status based on the presence of a special token in localStorage.
When the cherished token exists, let useAuth boldly declare, "Login granted!" with a true value.
In the absence of the token, let it softly murmur, "Not logged in" with a false value.
Ensure graceful handling of scenarios where the token is missing—no crashes allowed.
Customize the hook to be able to seamlessly integrate with your project's existing authentication setup.
Keep the implementation concise and purpose-driven.
File Location:
Write this dynamic component in src/hooks/useAuth.jsx.
Feel free to ask regarding any confusion in the given task.
Are you ready to embark on this challenge and unlock the authentication prowess of useAuth for GrepIt? Comment below to claim the task!

claim

Heyy @rnavaneeth992, You are already working on a task in this repo. Kindly complete the issue before claiming another one.

c
laim

@kushal7201  Assigned!

Claim

claim

c

repo: opencodeiiita/GrepIt-Frontend