text:
Some weights of the model checkpoint

Hi, I encountered a warning, is there a problem with this?
Some weights of the model checkpoint at THUDM/chatglm-6b were not used when initializing ChatGLMForConditionalGeneration: ['transformer.layers.11.post_attention_layernorm.weight', 'transformer.layers.12.input_layernorm.weight', 'transformer.layers.7.post_attention_layernorm.weight', 'transformer.layers.14.attention.dense.weight', 'transformer.layers.1.mlp.dense_4h_to_h.weight', 'transformer.layers.27.mlp.dense_h_to_4h.bias', 'transformer.layers.15.post_attention_layernorm.bias', 'transformer.layers.27.attention.dense.bias', 'transformer.layers.17.mlp.dense_4h_to_h.weight', 'transformer.layers.8.input_layernorm.weight', 'transformer.layers.9.attention.query_key_value.bias', 'transformer.layers.16.attention.rotary_emb.inv_freq', 'transformer.layers.27.input_layernorm.bias', 'transformer.layers.24.mlp.dense_h_to_4h.weight', 'transformer.layers.5.mlp.dense_4h_to_h.bias', 'transformer.layers.24.attention.rotary_emb.inv_freq', 'transformer.layers.20.input_layernorm.bias', 'transformer.layers.6.input_layernorm.bias', 'transformer.layers.25.mlp.dense_h_to_4h.weight', 'transformer.layers.16.post_attention_layernorm.bias', 'transformer.layers.16.mlp.dense_4h_to_h.bias', 'transformer.layers.23.attention.query_key_value.weight', 'transformer.layers.6.attention.rotary_emb.inv_freq', 'transformer.layers.10.attention.query_key_value.bias', 'transformer.layers.18.post_attention_layernorm.weight', 'transformer.layers.20.attention.query_key_value.weight', 'transformer.layers.11.mlp.dense_h_to_4h.weight', 'transformer.layers.5.mlp.dense_h_to_4h.bias', 'transformer.layers.13.attention.dense.bias', 'transformer.layers.11.attention.dense.weight', 'transformer.layers.5.input_layernorm.bias', 'transformer.layers.7.input_layernorm.bias', 'transformer.layers.13.mlp.dense_4h_to_h.bias', 'transformer.layers.18.attention.query_key_value.bias', 'transformer.layers.2.mlp.dense_h_to_4h.weight', 'transformer.layers.22.mlp.dense_h_to_4h.weight', 'transformer.layers.21.mlp.dense_4h_to_h.bias', 'transformer.layers.20.mlp.dense_h_to_4h.weight', 'transformer.layers.13.attention.query_key_value.weight', 'transformer.layers.20.attention.dense.bias', 'transformer.layers.23.attention.dense.weight', 'transformer.layers.22.attention.rotary_emb.inv_freq', 'transformer.layers.25.attention.query_key_value.weight', 'transformer.layers.16.attention.query_key_value.weight', 'transformer.layers.22.attention.dense.weight', 'transformer.layers.15.post_attention_layernorm.weight', 'transformer.layers.6.attention.dense.weight', 'transformer.layers.20.attention.query_key_value.bias', 'transformer.layers.27.input_layernorm.weight', 'transformer.layers.24.input_layernorm.bias', 'transformer.layers.25.attention.rotary_emb.inv_freq', 'transformer.layers.3.attention.dense.weight', 'transformer.layers.14.post_attention_layernorm.bias', 'transformer.layers.25.attention.dense.bias', 'transformer.layers.9.input_layernorm.weight', 'transformer.layers.2.input_layernorm.bias', 'transformer.layers.1.input_layernorm.bias', 'transformer.layers.13.attention.rotary_emb.inv_freq', 'transformer.layers.23.attention.query_key_value.bias', 'transformer.layers.1.mlp.dense_4h_to_h.bias', 'transformer.layers.5.attention.rotary_emb.inv_freq', 'transformer.layers.11.attention.query_key_value.bias', 'transformer.layers.4.attention.dense.bias', 'transformer.layers.14.attention.query_key_value.bias', 'transformer.layers.2.attention.rotary_emb.inv_freq', 'transformer.layers.17.post_attention_layernorm.bias', 'transformer.layers.10.attention.dense.weight', 'transformer.layers.11.attention.rotary_emb.inv_freq', 'transformer.layers.16.post_attention_layernorm.weight', 'transformer.layers.12.mlp.dense_4h_to_h.bias', 'transformer.layers.24.post_attention_layernorm.bias', 'transformer.layers.1.attention.dense.bias', 'transformer.layers.9.post_attention_layernorm.weight', 'transformer.layers.8.mlp.dense_h_to_4h.weight', 'transformer.layers.25.post_attention_layernorm.weight', 'transformer.layers.3.attention.dense.bias', 'transformer.layers.9.attention.dense.bias', 'transformer.layers.16.mlp.dense_h_to_4h.weight', 'transformer.layers.15.attention.query_key_value.weight', 'transformer.layers.14.attention.query_key_value.weight', 'transformer.layers.7.post_attention_layernorm.bias', 'transformer.layers.12.post_attention_layernorm.bias', 'transformer.layers.19.attention.dense.bias', 'transformer.layers.10.input_layernorm.bias', 'transformer.layers.3.mlp.dense_4h_to_h.bias', 'transformer.layers.12.mlp.dense_h_to_4h.weight', 'transformer.layers.13.input_layernorm.weight', 'transformer.layers.9.mlp.dense_h_to_4h.bias', 'transformer.layers.24.attention.query_key_value.weight', 'transformer.layers.10.input_layernorm.weight', 'transformer.layers.15.attention.dense.bias', 'transformer.layers.20.input_layernorm.weight', 'transformer.layers.1.post_attention_layernorm.bias', 'transformer.layers.2.mlp.dense_4h_to_h.bias', 'transformer.layers.24.attention.dense.weight', 'transformer.layers.13.post_attention_layernorm.bias', 'transformer.layers.10.mlp.dense_h_to_4h.weight', 'transformer.layers.15.mlp.dense_h_to_4h.weight', 'transformer.layers.6.attention.query_key_value.weight', 'transformer.layers.27.mlp.dense_4h_to_h.weight', 'transformer.layers.20.post_attention_layernorm.bias', 'transformer.layers.5.mlp.dense_4h_to_h.weight', 'transformer.layers.22.mlp.dense_4h_to_h.bias', 'transformer.layers.17.attention.query_key_value.weight', 'transformer.layers.27.mlp.dense_4h_to_h.bias', 'transformer.layers.25.attention.dense.weight', 'transformer.layers.19.post_attention_layernorm.bias', 'transformer.layers.7.attention.query_key_value.weight', 'transformer.layers.26.input_layernorm.bias', 'transformer.layers.10.mlp.dense_h_to_4h.bias', 'transformer.layers.17.post_attention_layernorm.weight', 'transformer.layers.16.input_layernorm.weight', 'transformer.layers.2.input_layernorm.weight', 'transformer.layers.20.attention.rotary_emb.inv_freq', 'transformer.layers.9.attention.dense.weight', 'transformer.layers.27.mlp.dense_h_to_4h.weight', 'transformer.layers.27.post_attention_layernorm.weight', 'transformer.layers.23.mlp.dense_4h_to_h.weight', 'transformer.layers.2.attention.query_key_value.bias', 'transformer.layers.22.mlp.dense_4h_to_h.weight', 'transformer.layers.12.mlp.dense_4h_to_h.weight', 'transformer.layers.18.attention.rotary_emb.inv_freq', 'transformer.layers.16.attention.query_key_value.bias', 'transformer.layers.11.mlp.dense_h_to_4h.bias', 'transformer.layers.7.attention.rotary_emb.inv_freq', 'transformer.layers.11.post_attention_layernorm.bias', 'transformer.layers.5.attention.dense.weight', 'transformer.layers.25.input_layernorm.weight', 'transformer.layers.19.attention.query_key_value.weight', 'transformer.layers.4.post_attention_layernorm.bias', 'transformer.layers.9.mlp.dense_4h_to_h.weight', 'transformer.layers.10.mlp.dense_4h_to_h.bias', 'transformer.layers.16.input_layernorm.bias', 'transformer.layers.2.attention.dense.weight', 'transformer.layers.22.input_layernorm.bias', 'transformer.layers.18.mlp.dense_4h_to_h.weight', 'transformer.layers.2.attention.dense.bias', 'transformer.layers.4.attention.query_key_value.weight', 'transformer.layers.21.mlp.dense_4h_to_h.weight', 'transformer.layers.26.mlp.dense_h_to_4h.weight', 'transformer.layers.23.post_attention_layernorm.bias', 'transformer.layers.3.attention.rotary_emb.inv_freq', 'transformer.layers.19.mlp.dense_h_to_4h.bias', 'transformer.layers.15.input_layernorm.weight', 'transformer.layers.7.attention.dense.bias', 'transformer.layers.12.attention.dense.bias', 'transformer.layers.1.attention.query_key_value.bias', 'transformer.layers.10.mlp.dense_4h_to_h.weight', 'transformer.layers.25.mlp.dense_4h_to_h.weight', 'transformer.layers.9.mlp.dense_h_to_4h.weight', 'transformer.layers.20.post_attention_layernorm.weight', 'transformer.layers.9.input_layernorm.bias', 'transformer.layers.17.attention.dense.bias', 'transformer.layers.9.attention.rotary_emb.inv_freq', 'transformer.layers.25.mlp.dense_h_to_4h.bias', 'transformer.layers.11.attention.query_key_value.weight', 'transformer.layers.18.attention.dense.weight', 'transformer.layers.12.attention.dense.weight', 'transformer.layers.4.attention.query_key_value.bias', 'transformer.layers.2.mlp.dense_4h_to_h.weight', 'transformer.layers.16.attention.dense.weight', 'transformer.layers.5.mlp.dense_h_to_4h.weight', 'transformer.layers.12.post_attention_layernorm.weight', 'transformer.layers.14.input_layernorm.weight', 'transformer.layers.1.mlp.dense_h_to_4h.weight', 'transformer.layers.3.attention.query_key_value.bias', 'transformer.layers.27.attention.rotary_emb.inv_freq', 'transformer.layers.18.input_layernorm.bias', 'transformer.layers.19.mlp.dense_4h_to_h.weight', 'transformer.layers.24.attention.query_key_value.bias', 'transformer.layers.3.input_layernorm.weight', 'transformer.layers.15.mlp.dense_h_to_4h.bias', 'transformer.layers.7.mlp.dense_h_to_4h.bias', 'transformer.layers.12.attention.query_key_value.weight', 'transformer.layers.12.mlp.dense_h_to_4h.bias', 'transformer.layers.15.attention.query_key_value.bias', 'transformer.layers.26.post_attention_layernorm.weight', 'transformer.layers.4.mlp.dense_4h_to_h.bias', 'transformer.layers.8.post_attention_layernorm.weight', 'transformer.layers.6.mlp.dense_4h_to_h.weight', 'transformer.layers.21.input_layernorm.weight', 'transformer.layers.22.attention.query_key_value.bias', 'transformer.layers.23.input_layernorm.weight', 'transformer.layers.26.attention.query_key_value.bias', 'transformer.layers.24.mlp.dense_4h_to_h.bias', 'transformer.layers.21.attention.dense.weight', 'transformer.layers.8.attention.dense.bias', 'transformer.layers.5.attention.query_key_value.bias', 'transformer.layers.15.mlp.dense_4h_to_h.bias', 'transformer.layers.21.attention.dense.bias', 'transformer.layers.22.post_attention_layernorm.weight', 'transformer.layers.17.mlp.dense_h_to_4h.bias', 'transformer.layers.13.attention.query_key_value.bias', 'transformer.layers.4.mlp.dense_h_to_4h.bias', 'transformer.layers.13.mlp.dense_h_to_4h.bias', 'transformer.layers.5.attention.query_key_value.weight', 'transformer.layers.23.mlp.dense_4h_to_h.bias', 'transformer.layers.10.attention.query_key_value.weight', 'transformer.layers.13.attention.dense.weight', 'transformer.layers.27.post_attention_layernorm.bias', 'transformer.layers.5.post_attention_layernorm.weight', 'transformer.layers.22.attention.dense.bias', 'transformer.layers.7.attention.dense.weight', 'transformer.layers.8.attention.query_key_value.weight', 'transformer.layers.5.post_attention_layernorm.bias', 'transformer.layers.13.mlp.dense_h_to_4h.weight', 'transformer.layers.3.mlp.dense_h_to_4h.bias', 'transformer.layers.23.attention.rotary_emb.inv_freq', 'transformer.layers.6.mlp.dense_4h_to_h.bias', 'transformer.layers.11.mlp.dense_4h_to_h.weight', 'transformer.layers.14.mlp.dense_h_to_4h.bias', 'transformer.layers.21.input_layernorm.bias', 'transformer.layers.3.mlp.dense_4h_to_h.weight', 'transformer.layers.4.mlp.dense_4h_to_h.weight', 'transformer.layers.18.attention.dense.bias', 'transformer.layers.8.mlp.dense_h_to_4h.bias', 'transformer.layers.24.input_layernorm.weight', 'transformer.layers.13.input_layernorm.bias', 'transformer.layers.21.post_attention_layernorm.bias', 'transformer.layers.4.input_layernorm.bias', 'transformer.layers.15.mlp.dense_4h_to_h.weight', 'transformer.layers.3.post_attention_layernorm.bias', 'transformer.layers.1.post_attention_layernorm.weight', 'transformer.layers.5.input_layernorm.weight', 'transformer.layers.6.mlp.dense_h_to_4h.bias', 'transformer.layers.14.mlp.dense_h_to_4h.weight', 'transformer.layers.10.post_attention_layernorm.weight', 'transformer.layers.18.input_layernorm.weight', 'transformer.layers.23.mlp.dense_h_to_4h.weight', 'transformer.layers.26.post_attention_layernorm.bias', 'transformer.layers.26.attention.dense.bias', 'transformer.layers.26.attention.query_key_value.weight', 'transformer.layers.24.attention.dense.bias', 'transformer.layers.13.mlp.dense_4h_to_h.weight', 'transformer.layers.2.post_attention_layernorm.weight', 'transformer.layers.22.post_attention_layernorm.bias', 'transformer.layers.22.input_layernorm.weight', 'transformer.layers.23.attention.dense.bias', 'transformer.layers.9.attention.query_key_value.weight', 'transformer.layers.6.attention.dense.bias', 'transformer.layers.15.attention.dense.weight', 'transformer.layers.14.attention.dense.bias', 'transformer.layers.24.post_attention_layernorm.weight', 'transformer.layers.7.attention.query_key_value.bias', 'transformer.layers.1.mlp.dense_h_to_4h.bias', 'transformer.layers.14.mlp.dense_4h_to_h.bias', 'transformer.layers.7.mlp.dense_4h_to_h.bias', 'transformer.layers.27.attention.dense.weight', 'transformer.layers.2.post_attention_layernorm.bias', 'transformer.layers.24.mlp.dense_4h_to_h.weight', 'transformer.layers.17.attention.rotary_emb.inv_freq', 'transformer.layers.15.attention.rotary_emb.inv_freq', 'transformer.layers.5.attention.dense.bias', 'transformer.layers.7.mlp.dense_4h_to_h.weight', 'transformer.layers.27.attention.query_key_value.weight', 'transformer.layers.9.mlp.dense_4h_to_h.bias', 'transformer.layers.8.attention.rotary_emb.inv_freq', 'transformer.layers.22.attention.query_key_value.weight', 'transformer.layers.19.attention.rotary_emb.inv_freq', 'transformer.layers.11.input_layernorm.weight', 'transformer.layers.18.mlp.dense_4h_to_h.bias', 'transformer.layers.11.input_layernorm.bias', 'transformer.layers.19.mlp.dense_4h_to_h.bias', 'transformer.layers.2.mlp.dense_h_to_4h.bias', 'transformer.layers.8.attention.query_key_value.bias', 'transformer.layers.12.attention.query_key_value.bias', 'transformer.layers.26.mlp.dense_4h_to_h.bias', 'transformer.layers.1.attention.dense.weight', 'transformer.layers.14.post_attention_layernorm.weight', 'transformer.layers.6.attention.query_key_value.bias', 'transformer.layers.6.post_attention_layernorm.weight', 'transformer.layers.22.mlp.dense_h_to_4h.bias', 'transformer.layers.20.attention.dense.weight', 'transformer.layers.9.post_attention_layernorm.bias', 'transformer.layers.17.attention.dense.weight', 'transformer.layers.24.mlp.dense_h_to_4h.bias', 'transformer.layers.4.attention.dense.weight', 'transformer.layers.18.post_attention_layernorm.bias', 'transformer.layers.13.post_attention_layernorm.weight', 'transformer.layers.18.mlp.dense_h_to_4h.weight', 'transformer.layers.26.mlp.dense_h_to_4h.bias', 'transformer.layers.20.mlp.dense_4h_to_h.bias', 'transformer.layers.12.input_layernorm.bias', 'transformer.layers.1.attention.rotary_emb.inv_freq', 'transformer.layers.19.post_attention_layernorm.weight', 'transformer.layers.26.attention.dense.weight', 'transformer.layers.14.attention.rotary_emb.inv_freq', 'transformer.layers.20.mlp.dense_h_to_4h.bias', 'transformer.layers.1.input_layernorm.weight', 'transformer.layers.23.input_layernorm.bias', 'transformer.layers.21.attention.query_key_value.bias', 'transformer.layers.7.mlp.dense_h_to_4h.weight', 'transformer.layers.21.attention.rotary_emb.inv_freq', 'transformer.layers.20.mlp.dense_4h_to_h.weight', 'transformer.layers.6.input_layernorm.weight', 'transformer.layers.4.mlp.dense_h_to_4h.weight', 'transformer.layers.18.mlp.dense_h_to_4h.bias', 'transformer.layers.25.mlp.dense_4h_to_h.bias', 'transformer.layers.16.mlp.dense_4h_to_h.weight', 'transformer.layers.16.mlp.dense_h_to_4h.bias', 'transformer.layers.17.attention.query_key_value.bias', 'transformer.layers.10.attention.rotary_emb.inv_freq', 'transformer.layers.4.post_attention_layernorm.weight', 'transformer.layers.8.post_attention_layernorm.bias', 'transformer.layers.21.attention.query_key_value.weight', 'transformer.layers.2.attention.query_key_value.weight', 'transformer.layers.4.attention.rotary_emb.inv_freq', 'transformer.layers.19.input_layernorm.bias', 'transformer.layers.10.attention.dense.bias', 'transformer.layers.15.input_layernorm.bias', 'transformer.layers.14.input_layernorm.bias', 'transformer.layers.4.input_layernorm.weight', 'transformer.layers.3.input_layernorm.bias', 'transformer.layers.23.mlp.dense_h_to_4h.bias', 'transformer.layers.6.post_attention_layernorm.bias', 'transformer.layers.19.input_layernorm.weight', 'transformer.layers.26.mlp.dense_4h_to_h.weight', 'transformer.layers.23.post_attention_layernorm.weight', 'transformer.layers.17.mlp.dense_h_to_4h.weight', 'transformer.layers.26.attention.rotary_emb.inv_freq', 'transformer.layers.11.mlp.dense_4h_to_h.bias', 'transformer.layers.8.mlp.dense_4h_to_h.bias', 'transformer.layers.3.attention.query_key_value.weight', 'transformer.layers.3.post_attention_layernorm.weight', 'transformer.layers.25.post_attention_layernorm.bias', 'transformer.layers.11.attention.dense.bias', 'transformer.layers.21.post_attention_layernorm.weight', 'transformer.layers.8.attention.dense.weight', 'transformer.layers.7.input_layernorm.weight', 'transformer.layers.21.mlp.dense_h_to_4h.weight', 'transformer.layers.17.input_layernorm.weight', 'transformer.layers.27.attention.query_key_value.bias', 'transformer.layers.12.attention.rotary_emb.inv_freq', 'transformer.layers.25.input_layernorm.bias', 'transformer.layers.8.input_layernorm.bias', 'transformer.layers.26.input_layernorm.weight', 'transformer.layers.6.mlp.dense_h_to_4h.weight', 'transformer.layers.14.mlp.dense_4h_to_h.weight', 'transformer.layers.17.input_layernorm.bias', 'transformer.layers.19.mlp.dense_h_to_4h.weight', 'transformer.layers.10.post_attention_layernorm.bias', 'transformer.layers.16.attention.dense.bias', 'transformer.layers.21.mlp.dense_h_to_4h.bias', 'transformer.layers.1.attention.query_key_value.weight', 'transformer.layers.8.mlp.dense_4h_to_h.weight', 'transformer.layers.18.attention.query_key_value.weight', 'transformer.layers.19.attention.query_key_value.bias', 'transformer.layers.19.attention.dense.weight', 'transformer.layers.17.mlp.dense_4h_to_h.bias', 'transformer.layers.3.mlp.dense_h_to_4h.weight', 'transformer.layers.25.attention.query_key_value.bias']
- This IS expected if you are initializing ChatGLMForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ChatGLMForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

你不是全参数， 有的层没利用起来 ，所以有这个警告也是正常的.

谢谢，我试一下全量参数的效果。感谢回复。

你不是全参数， 有的层没利用起来 ，所以有这个警告也是正常的.

怎么进行全量参数

我的也出现了类似问题，但是跟你的还不太一样：Some weights of ChatGLMForConditionalGeneration were not initialized from the model checkpoint at /home/duser/hzl/Project/rlhf/applications/DeepSpeed-Chat/output/actor-models/chatglm and are newly initialized: ['lm_head.weight', 'transformer.layers.21.attention.rotary_emb.inv_freq', 'transformer.layers.22.attention.rotary_emb.inv_freq', 'transformer.layers.5.attention.rotary_emb.inv_freq', 'transformer.layers.13.attention.rotary_emb.inv_freq', 'transformer.layers.23.attention.rotary_emb.inv_freq', 'transformer.layers.11.attention.rotary_emb.inv_freq', 'transformer.layers.20.attention.rotary_emb.inv_freq', 'transformer.layers.26.attention.rotary_emb.inv_freq', 'transformer.layers.0.attention.rotary_emb.inv_freq', 'transformer.layers.15.attention.rotary_emb.inv_freq', 'transformer.layers.14.attention.rotary_emb.inv_freq', 'transformer.layers.10.attention.rotary_emb.inv_freq', 'transformer.layers.3.attention.rotary_emb.inv_freq', 'transformer.layers.6.attention.rotary_emb.inv_freq', 'transformer.layers.12.attention.rotary_emb.inv_freq', 'transformer.layers.1.attention.rotary_emb.inv_freq', 'transformer.layers.7.attention.rotary_emb.inv_freq', 'transformer.layers.27.attention.rotary_emb.inv_freq', 'transformer.layers.16.attention.rotary_emb.inv_freq', 'transformer.layers.19.attention.rotary_emb.inv_freq', 'transformer.layers.17.attention.rotary_emb.inv_freq', 'transformer.layers.2.attention.rotary_emb.inv_freq', 'transformer.layers.25.attention.rotary_emb.inv_freq', 'transformer.layers.4.attention.rotary_emb.inv_freq', 'transformer.layers.24.attention.rotary_emb.inv_freq', 'transformer.layers.18.attention.rotary_emb.inv_freq', 'transformer.layers.9.attention.rotary_emb.inv_freq', 'transformer.layers.8.attention.rotary_emb.inv_freq']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

https://github.com/ssbuild/chatglm_finetuning/blob/4e431c9207d7e1875613076f64bdb69c90a7fc4e/config/chatglm_sft_config.py#L14
@LanShanPi
check num_layers = -1

authors: 980202006 ForgetThatNight LanShanPi ssbuild


text:
Proposal for a new description and link for ArcGIS

As I mentioned on this comment, I wouldn't describe ArcGIS like "a desktop GIS software" anymore, and the link is pointing to "www.arcgis.com" which is the landing page for "ArcGIS Online", the cloud service, the official page is esri.com/arcgis.
I have spent some time working a diagram to describe the ecosystem of technologies that form the Esri platform, and based on my experience, I think it would make more sense to change the description to "Cross-platform with free, open and paid data and mapping tools, APIs, SDKs, etc."?

Note: I would include "free" because there are more than 7000 curated datasets than can be used for free, but also a developer account that allow to use ArcGIS Online with 10 SDKs, 5 app builders, etc, for free with a pay-as-you-go model that includes monthly credits to use APIs for free (for example to host +200MB of data in the free account).

Cheers!

I agree the old link should be replaced. But the description including too many things. I supposed Geographic Information System Software should focus on Desktop Software, but not APIs, SDKs...
One more thing, is ArcGIS cross-platform(MacOS/Linux) now?

Aha, ok then the name of the section and other elements in the list (like GISInternals) confused me, I thought Geographic Information System in the broadest sense of the term, [like in the wikipedia]:(https://en.wikipedia.org/wiki/Geographic_information_system):

...tools that allow users to create interactive queries (user-created searches), analyze spatial information, edit data in maps, and present the results of all these operations

Why not call it then GIS Desktop Software or just Desktop Software?
Having that in mind:

I would replace ArcGIS by ArcGIS Desktop.
You are right, ArcGIS Desktop is just for Windows. The only out of the box product (besides web and mobile apps) that runs on other OS is ArcGIS Enterprise, formerly called ArcGIS Server, which also runs on Linux. The technologies to build cross-platform tools are some of the SDKs like Qt, Java, and Xamarin.

So for ArcGIS Desktop, I would say:

ArcGIS Desktop: Extendable desktop suite to manage, visualize and analyze GIS data in 2D and 3D, including image processing. Includes ArcGIS Pro, ArcMap, ArcCatalog, and ArcGIS Online.

That's sounds great, could you modify your PR?

Sure, done!

What do you think about replacing...

## Geographic Information System Software
for: ## Desktop Software

? 😄

How about  GIS Desktop Software?

source: gharchive/pull-request

authors: hhkaos sshuair

repo: sshuair/awesome-gis

license: MIT

license_type: permissive

license_source: github-api


text:
Modifying project card style [GSSoC '23]

Problem
The project cards have category and project title on them, and some cards which have light images , on them the texts are little difficult to read.
Solution
I would like to add little overlay on the cards and change the position of the category and project title , so that it would look good and also be readable.
Please assign me this, so that I could work on this and make the cards more user friendly.

assigned to u and In PR include screenshot after changes to assess your work. Dont forget to follow contributing.md guidelines after reviewing PR by mentor

thanks @Dharmesh177 , will surely abide by the contributing.md guidelines

Hey @Dharmesh177 ,
This is what I came up with. If you like I can create a pr

Before

After

@Dharmesh177 Please review my pr#105

source: gharchive/issue

repo: ssitvit/Code-Canvas


text:
fix: auth sst.cloudflare with email/key pair

fixes #760
the any on the getAuth() function's return type is because undici's types are being weird

@thdxr sorry for pinging you on like everything but can you please review this whenever you get a chance

repo: sst/ion


text:
Bats swallows output in tests

Because of https://github.com/sstephenson/bats/issues/190, I tried to do the obvious and print $output to standard out, but echo won't actually print anything:
@test "where's my output" {
  echo "echo"
}

this prints nothing to standard out.
Am I missing something? How do you inspect $output etc. when a test fails?

This is intentional. From the wiki:

Anything written to stdout or stderr in setup, teardown, or a test function is captured by Bats. If the test case fails, the error trap prints the output to the TAP stream as a comment. Outside a test case, though, stdout is the TAP stream, so you'll want to specifically redirect any output to stderr.

You can use this to show relevant information on failure.
@test 'test-a' {
  run bash -c 'echo ERROR; false'
  echo "status = ${status}"
  echo "output = ${output}"
  [ "$status" -eq 0 ]
}

Bats will show all output made before the failing assertion. In this case, $output and $status.
 ✗ test-a
   (in test file /tmp/test.bats, line 5)
     `[ "$status" -eq 0 ]' failed
   status = 1
   output = ERROR

Aslo see my answer to your previous question #190.

Thanks for the quick reply! This is really helpful 👍 Why not make this a bit more visible and/or add your example to the wiki? I'm sure I'm not the first one to run head first into this.
Closing this.

Agreed!
This issue is awesome for:

covering something common to newcomers that is super important
quick response
easy solution

The only thing that’s missing is a high-visibility place for other newcomers to see the solution before they have the same problem.
STDOUT and STDERR are important for just about any program that’s likely to use BATS. I think it’s worth doing the equivalent of a “pull quote” from the normal docs. :)

Yes, this should be more visible. Along with a dozen other gotchas and best practices. But the maintainer does not have enough time. See #150. A number of us volunteered to help out, but no one serious has stepped forward to take over maintenance. I'm not blaming anyone. After all, I'm still in debt with my long list of proposed improvements that I promised. Bats is an important piece of many projects' infrastructure. Too bad that it does not receive enough attention.

authors: Zearin mttkay ztombol

repo: sstephenson/bats

license: mit

license_source: bigquery


text:
Do not include whitespace when applying inline markdown styles

If you test out markdown on a gist, markdown like this:
**Test **

(with trailing or leading whitespace) is not properly parsed. (examples here: https://gist.github.com/Rosey/3f848d77b46f7ab337a53211117acd87)
This change only surrounds the text with markdown inline styles and preserves
the whitespace, but doesn't attempt to style it, fixing the issue of
incompatibility with markdown parsers.

I'm not super confident about the efficiency of my regular expression here. It works, but I am worried that it is not optimal. Perhaps someone with better regex-fu will know if it can be improved performance-wise.

@Rosey are you planning to fix the conflicts? Your PR most probably also solved https://github.com/sstur/react-rte/issues/127

authors: Rosey binarybaba

repo: sstur/draft-js-export-markdown

license: isc


text:
add support for CodeFormer correction model

https://github.com/sczhou/CodeFormer

I have this partially written, but there does not seem to be a pip package for CodeFormer, which will make that slightly more difficult.

I have this working on CPU using https://pypi.org/project/codeformer-perceptor/, but adapting that to run on ONNX will take a little bit of work, since the library currently downloads its own checkpoints.

repo: ssube/onnx-web


text:
[FEAT] Advanced support for multi-line

Is your feature request related to a problem? Please describe.
The multi-line support doesn't allow for reach formatting of the answer.
Here is the example of the card I would like to have:
Question?
??
Answer 1

Empty line above

Code Block

More empty lines.

Links to other cards

The work around I can have right now is to use blockquote
Question?
??
>Answer 1
>
>Empty line above
>
>Code Block
>
>More empty lines.
>
>Links to other cards

This doesn't make it look nice. I can play with formatting more and don't wrap code block but it would be nice to have better support for multi-line
Describe the solution you'd like
Introduce configurable delimiters for the answer boundaries. By default it could be comments.
Question?
<!--SR:START-->
Answer 1

Empty line above

Code Block

More empty lines.

Links to other cards

<!--SR:END-->

Thank you.

Hi @jvdevlab
Thanks for the suggestion.
There are some related features already requested, please see:

[FEAT] Support blank lines in multi-line flashcards
More flexible formatting (especially blank lines) for flashcards

The current functionality supports changing the string that separates the question from the answer. I haven't tried it, but I would think that you would be able to change it to <!--SR:START-->.
Note that #601 has support for blank lines in the question as well as the answer:
^^^
This is the first line of my question.

This is the second line my question.
+++
This is the first line of my answer.

This is the second line of my answer.
vvv

Is there anything that you are after that would not be supported if #601 is implemented?
Cheers
Ronny

Hi @ronzulu,
Thank you for the reply. Any solution that would support blank lines will do.
I feel like comments approach will make it be less prone to conflict with other delimiters used in other notes and easier to parse.
I did try the comment as a start delimiter.
`## Test questions?

Test answer
`
It cuts off the beginning of the question

But as mentioned above any solution will do.
Thanks

Hi @jvdevlab glad we have managed to consolidate functionality at #601.
I'll close this off now, but if there is anything else related you think of, feel free to update that issue.
Cheers
Ronny

authors: jvdevlab ronzulu


text:
Conversion operator between SE3 and hpp::fcl::Transform3f

What would you think of:

a (explicit ?) conversion operator in class SE3Tpl (or SE3Base) to hpp::fcl::Transform3f and
a (explicit ?) constructor of class SE3Tpl that takes a hpp::fcl::Transform3f as input ?

At the moment, there are two functions toFclTransform3f and toPinocchioSE3. They are convenient but make the code a bit less readable.
Another possible way of implementing this is to implement a SE3Base derived class that implements the SE3 interface from a hpp::fcl::Transform3f.
This method requires more work and more code.

I'm not in favour of having hpp-fcl dependencies in Spatial classes. Because at some point, hpp-fcl is optional and other frameworks for collision detection will be included in the near future.
I think the current solution is the best we can do, having implemented an easy interface for Python.
What we can add with this respect is the common interface to Eigen homogenous transform with a dedicated constructor in SE3.

I understand your point. I will stick to the existing function then. Thanks for your quick answer.

We can meet your wish with my second proposition if you want ;) It will be common to all frameworks then.

If SE3 can be converted to Eigen::Transform and Transform3f has a constructor from a Eigen::Transform, will the compiler understand:
void foo(const Transform3f& t);

int main()
{
SE3 m;
foo(m);
}

Because if we need to add something, then, toFclTransform3f is good enough.

Not directly, you will need to call the cast operator to Eigen::Transform.

So I don't think there are benefits today.

authors: jcarpent jmirabel

repo: stack-of-tasks/pinocchio


text:
Selectively "fixing" (actuated) joints at URDF-parsing time

Dear all,
I would like to bring up the following issue, regarding parsing URDF's including highly redundant URDFs in pinocchio.
The issue is best explained using an example: let's consider a setup where we have a manipulator, e.g. a robot arm combined with a very dexterous gripper, e.g. a hand with a three finger gripper. Usually, for controlling the arm, we can neglect the dynamic behaviour of the fingers, and simply aggregate all the finger inertia into the last link of the arm.
What I see many people doing nowadays, is that the simply keep track of multiple URDFs some containing the full dexterous model, some containing the simplified model. Here is a potential issue where pinocchio could differentiate itself: by implementing a URDF-parser which allows to selectively override the joint-types in URDFs at parsing time, for instance to selectively override revolute joints in gripper fingers as "fixed" joints.
Currently fixed joints are handled in the way that the following link parameters (inertia, mass..) are aggregated to its parent. Therefore, a simple solution would be to provide an optional argument to buildModel(), for example a list of actuated or blocked joints, which selectively override the original definition from the URDF.
Personally I feel that there might be a need for such a parser variant, but I'm curious to hear what you guys think!

Dear @markusgft,
This is a very good initiative and suggestion.
I would suggest to not put this feature at the URDF level but at the model level.
In Pinocchio, we already have some functions acting on two models to append them together.
We can imagine having a new function called:
Model lockJointsInModel(const Model & model, const std::vector<std::string> & list_of_joints_to_lock) which will return a new model where the joints present in the list will be locked.
@markusgft @traversaro What do you think?

I would suggest to not put this feature at the URDF level but at the model level.
Indeed that makes sense, I'd also prefer this solution!

I'd propose to even go so far to provide 2 different joint-locking-API's. One function could be called

Model lockJointsInModel(const Model & model, const std::vector<std::string> & list_of_joints_to_lock)

as you suggested above. But it could even be that the "inverse" of above solution with a list of joints to keep unlocked is more useful in practice (usually we know exactly which joints we want to control, and don't care about the rest!) So the function call would look like:
Model lockJointsInModel(const Model & model, const std::vector<std::string> & list_of_**unlocked**_joints)

Sounds great for me! @markusgft Do you validate the two new signatures:
Model lockSomeJoints(const Model & original_model, const std::vector<std::string> & list_of_joints_to_lock)
and for the second signature:
Model keepSomeJointsUnlocked(const Model & original_model, const std::vector<std::string> & list_of_**unlocked**_joints)
?

Concerning the suggested naming, feel free to adjust them ;)
I may be not good at choosing correct namings.

@markusgft @traversaro What do you think?

But it could even be that the "inverse" of above solution with a list of joints to keep unlocked is more useful in practice (usually we know exactly which joints we want to control, and don't care about the rest!) So the function call would look like:
Model lockJointsInModel(const Model & model, const std::vector<std::string> & list_of_**unlocked**_joints)

I totally agree with the proposal to have an API to just list the "not-locked" joints, we have that in iDynTree (see iDynTree::createReducedModel and is fundamental for the users of whole-body control, as typically for the iCub robot we may have a lot of joints that are not relevant for whole-body control (e.g. hands, eyes, eyelids) that may or not be present depending on the specific URDF model used.
A variant that we never implemented, but as been requested by users for a long time is the possibility to specify the value of the joints that are being fixed, see https://github.com/robotology/idyntree/issues/495 .

Thanks @traversaro for this nice reply.
I think indeed important to specify the value of the locked joint. I will add this feature to both signatures.

A possible name for the signatures could also be:
pinocchio::createReducedModel(const Model & model, const std::vector<std::string> & list_of_joints, Enum::LockType lock_or_unlock)
But I'm happy to leave the final decision to you @jcarpent ! Thank you again for reacting so quickly!!!

I will adopt your signature. It is pretty concise and clear.

@markusgft @traversaro Thanks a lot for your feedback.

I like the idea.
I would suggest to have a method name with lock somewhere inside, as I found the wording "reduce" somehow inaccurate. Maybe simply lockJoints(...) or createModelByLockingJoints(...).
I am surprised by the API with vector of string, instead of vector of joint indexes. It does not seem to follow the (implicit) standards, is it?
I thought during 2 minutes of proposing to create the reserve method, while it is obviously more suitable to keep the original (unlock)  model and always restart from it when changing the joint status.

Very useful feature. Maybe another possible API would be to first build the model and the call a function: aggregateTree(joint_id). I also think that would solve the problem with joint indexes vs. joints name because you can look up the ids before. However i cannot judge the complexity of this way, but it would make this feature independant from the urdf loader.

Solved by #927.

authors: alexswerner jcarpent markusgft nmansard traversaro


text:
ci: Add pre-commit action

This adds a re-usable pre-commit action. It optionally installs the Rust toolchain and Hadolint in the requested version.

Also, the readme will need to have a note about needing to adjust the fetch-depth on checkout.

authors: NickLarsenNZ Techassi

repo: stackabletech/actions


text:
API Request Fails with LM Studio's LLM (CORS Enabled) and Unexpected Ollama Errors

Describe the bug
I encountered the following issues while using LM Studio and would like some assistance. I am running bolt.diy directly on WSL without using Docker.
CORS設定を有効化
As shown in the attached image, I enabled the "CORS" option in LM Studio's server settings. After doing so, I was able to select the LLM (Large Language Model) from the dropdown menu.
APIリクエストエラー
When I selected LM Studio's LLM and attempted to generate code, I received the following error message:
`
There was an error processing your request: An error occurred.
`
Ollama関連の警告
Despite not using the Ollama server, I see these warning messages in the terminal:
‘
WARN Constants Failed to get Ollama models: fetch failed
WARN Constants Failed to get Ollama models: fetch failed
‘
Environment Details:
WSL (Windows Subsystem for Linux): Running bolt.diy directly (without Docker).
I would appreciate any guidance on:
Why the API request to LM Studio's server fails.
Why Ollama warnings appear even though I am not using it.
Thank you for your support!
Link to the Bolt URL that caused the error
http://localhost:5173/chat/10
Steps to reproduce
I entered the prompt, but I get an error
Expected behavior
I noticed in the screenshot of the debug info that the URL for the chat API in LMStudio seemed different from what LMStudio shows.
Screen Recording / Screenshot

Platform

OS: [ Linux]
Browser: [Chrome]
Version: [eb6d435(v0.0.3) - stable]

Provider Used
No response
Model Used
No response
Additional context
I am getting the following error on the LMStudio server.
2024-12-18 21:50:37 [ERROR] Unexpected endpoint or method. (GET /api/health). Returning 200 anyway
I think the URL that bolt.diy requests to LMStudio's API is different? I thought so.
2024-12-18 20:52:33  [INFO] [LM STUDIO SERVER] Success! HTTP server listening on port 1234
2024-12-18 20:52:33  [INFO]
2024-12-18 20:52:33  [INFO] [LM STUDIO SERVER] Supported endpoints:
2024-12-18 20:52:33  [INFO] [LM STUDIO SERVER] ->	GET  http://localhost:1234/v1/models
2024-12-18 20:52:33  [INFO] [LM STUDIO SERVER] ->	POST http://localhost:1234/v1/chat/completions
2024-12-18 20:52:33  [INFO] [LM STUDIO SERVER] ->	POST http://localhost:1234/v1/completions
2024-12-18 20:52:33  [INFO] [LM STUDIO SERVER] ->	POST http://localhost:1234/v1/embeddings

in lm studio try to enable: Serve on Local Network
if that dose not work than also try the local ip address of the computer that LM Stuido is running on
In this video I used docker for bolt.diy and ollama but it might help: https://youtu.be/TMvA10zwTbI

Thank you very much.
I have turned on serving on the local network of the LMStudio server.
But the error did not improve.
With the Docker method, I was able to use the Model of LMStudio.
Thank you very much for teaching me how to do this.

Sorry.
I re-cloned the repository this morning.
And I also turned on serving the LM Studio URL on the local network.
I set this URL in env.local and started it up, and it worked with WSL.
I was able to use it with WSL without using Docker.


text:
Deploy is not working properly

Describe the bug
I gave bolt.new make a clone google classroom but with better version and add extra it works but when said it to deploy in netlify, it just shows error
Link to the Bolt URL that caused the error
https://bolt.new/~/sb1-ycekeu
Steps to reproduce
1 create anything 2 say it to deploy app, application or website in netlify 4 error
Expected behavior
I expected that it will directly open netlify and the all code will be deploy in it.
Screen Recording / Screenshot
No response
Platform
Browser name = Safari
Full version = 604.1
Major version = 604
navigator.appName = Netscape
navigator.userAgent = Mozilla/5.0 (iPhone; CPU iPhone OS 17_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/130.0.6723.37 Mobile/15E148 Safari/604.1
performance.memory = undefined
Username = AashishThakuri
Chat ID = 13dcfd4f07a7

Additional context
Error:
An error occured while streaming this message.

@AashishThakuri this issue is being tracked in #1022


text:
Fix up the updated at time

Currently everything seems to use created at.

there is a new updated at time in the crd, updated when the phase is updated

repo: stackhpc/azimuth-caas-operator


text:
Helm tweaks

This started out as non-functional changes - see commit messages for these.
There are a few functional changes though:

It's undocumented, but using subpath in volumeMounts is what is preventing config maps from being updated. So this changes it so /etc/slurm/ is the volume (so the empty directory in the image gets overwritten. A helm upgrade will now update the configmap, and (it may take 10s of seconds) the slurm.conf file will be updated. An scontrol update will then load the changed config. Unfortunately there is no way of doing this with the slurmdbd due to the need to have slurmdbd.conf a) in the same directory as slurm.conf and b) owned by slurm, but that doesn't use too much of the configuration in slurm.conf anyway.

Options for slurm daemons can now be set in the kubernetes resource template using the container args.

Its possible to use something like this: docker run --rm -it <container tag> /bin/bash now to look at the container outside of Kubernetes. This is useful for e.g. seeing which files exist.

Its possible to pass debug as the container arg (instead of e.g. slurmd) in a Kubernetes resource to just start munge, which allows you to shell into the running container and e.g. debug changes to the entrypoint script.

Because nodes are defined in slurm.conf, they are defaulting https://slurm.schedmd.com/slurm.conf.html#OPT_CPUs 1x CPU. Which means launching jobs requiring > 1 task per node fails. I've modified slurm.conf to use the current cluster configuration of 4x CPUs, mostly as a reminder this needs fixing via Azimuth. The RealMemory parameter should also be set by Azimuth.

Tested and all works on my end, just a few tweaks and doc changes
Also when trying to run scontrol reconfig after mounting in a new slurm.conf, if I try to change properties like MaxNodeCount or NodeName, it fails with error slurm_reconfigure error: Zero Bytes were transmitted or received and loses the worker nodes until they are manually restarted. It still works okay with stuff like changing the debug level. Is this expected behaviour?

I think so. See https://slurm.schedmd.com/slurm.conf.html#OPT_MaxNodeCount:

.... The slurmctld daemon must be restarted for changes to this parameter to take effect.

The slurm.conf docs similarly note any other parameters where this is required, and the scontrol reconfigure docs also say:

Parameters requiring a restart will be noted in the slurm.conf(5) man page. The slurmctld daemon and all slurmd daemons must also be restarted if nodes are added to or removed from the cluster.

repo: stackhpc/slurm-docker-cluster


text:
ROX-13627: Extend GetNodeVulnerabilities API by supporting Node Inventory

This PR extends the GetNodeVulnerabilities API to accept a Node message that contains NodeInventory (see https://github.com/stackrox/stackrox/pull/3755 and https://github.com/stackrox/stackrox/pull/3757). The respective Central part of this request is still in progress and there is no PR for it yet.
This is required to support RHCOS Node scanning over the GetNodeVulnerabilities API.
Additionally, this PR includes:

A new feature flag RHCOSNodeScanning
Improvements in the Makefile that would make it usable for:

Colleagues who have spaces   in their PATH
~Colleagues who do not have wget but have curl~

How tested

[x] Added an e2e and run it locally with:

make image
make deploy-local
Open port-forward to the scanner pod 8443:8443
Run tests go test -tags e2e -timeout=10s -v -run ^TestGRPCGetRHCOSNodeVulnerabilities$  github.com/stackrox/scanner/e2etests

[x] Tested the changes in the Makefile by doing make clean-proto-deps and doing the steps described in  https://github.com/stackrox/scanner#steps
[x] Deploying with Central locally and seeing the vulnerabilities in the UI
[x] CI

Current dependencies on/for this PR:

master

PR #1004   👈

This comment was auto-generated by Graphite.

/test all

/test all

/test all

/test all

/test all

/test all

/test all

I will rebase as well as suggested.

/retest

After enabling the feature flag in code (setting default to true), the test executes properly:
(from https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/stackrox_scanner/1004/pull-ci-stackrox-scanner-master-e2e-tests/1595484736086609920/artifacts/e2e-tests/e2e-tests/build-log.txt)
=== RUN   TestGRPCGetRHCOSNodeVulnerabilities
=== RUN   TestGRPCGetRHCOSNodeVulnerabilities/case-0
Handling connection for 8443
    node_scan_rhcos_test.go:141: gotFeat.InventoryFeatures: [name:"libksba" version:"1.3.5-7.el8.x86_64" feature_type:"rpm" vulnerabilities:<name:"RHSA-2022:7089" description:"KSBA (pronounced Kasbah) is a library to make X.509 certificates as well as the CMS easily accessible by other applications.  Both specifications are building blocks of S/MIME and TLS.\n\nSecurity Fix(es):\n\n* libksba: integer overflow may lead to remote code execution (CVE-2022-3515)\n\nFor more details about the security issue(s), including the impact, a CVSS score, acknowledgments, and other related information, refer to the CVE page(s) listed in the References section." link:"https://access.redhat.com/errata/RHSA-2022:7089" metadata_v2:<published_date_time:"2022-10-24T00:00Z" last_modified_date_time:"2022-10-24T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H" score:8.1 exploitability_score:2.2 impact_score:5.9 > > fixed_by:"0:1.3.5-8.el8_6" severity:"Important" > fixed_by:"1.3.5-8.el8_6"  name:"tar" version:"1.27.1.el8.x86_64" feature_type:"rpm" vulnerabilities:<name:"CVE-2005-2541" description:"DOCUMENTATION: The MITRE CVE dictionary describes this issue as: Tar 1.15.1 does not properly warn the user when extracting setuid or setgid files, which may allow local users or remote attackers to gain privileges. \n            STATEMENT: This CVE was assigned to what is the documented and expected behaviour of tar.  There are currently no plans to change tar behaviour to strip setuid and setgid bits when extracting archives." link:"https://access.redhat.com/security/cve/CVE-2005-2541" metadata_v2:<last_modified_date_time:"2021-06-28T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:L/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H" score:7 exploitability_score:1 impact_score:5.9 > > severity:"Moderate" > vulnerabilities:<name:"CVE-2019-9923" description:"DOCUMENTATION: The MITRE CVE dictionary describes this issue as: pax_decode_header in sparse.c in GNU Tar before 1.32 had a NULL pointer dereference when parsing certain archives that have malformed extended headers." link:"https://access.redhat.com/security/cve/CVE-2019-9923" metadata_v2:<last_modified_date_time:"2021-10-27T00:00Z" cvss_v3:<vector:"CVSS:3.0/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L" score:3.3 exploitability_score:1.8 impact_score:1.4 > > severity:"Low" > vulnerabilities:<name:"CVE-2021-20193" description:"DOCUMENTATION: A flaw was found in the src/list.c of tar. This flaw allows an attacker who can submit a crafted input file to tar to cause uncontrolled consumption of memory. The highest threat from this vulnerability is to system availability." link:"https://access.redhat.com/security/cve/CVE-2021-20193" metadata_v2:<last_modified_date_time:"2022-04-17T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L" score:3.3 exploitability_score:1.8 impact_score:1.4 > > severity:"Low" > ]
    node_scan_rhcos_test.go:141: gotFeat.InventoryFeatures: [name:"libksba" version:"1.3.5-7.el8.x86_64" feature_type:"rpm" vulnerabilities:<name:"RHSA-2022:7089" description:"KSBA (pronounced Kasbah) is a library to make X.509 certificates as well as the CMS easily accessible by other applications.  Both specifications are building blocks of S/MIME and TLS.\n\nSecurity Fix(es):\n\n* libksba: integer overflow may lead to remote code execution (CVE-2022-3515)\n\nFor more details about the security issue(s), including the impact, a CVSS score, acknowledgments, and other related information, refer to the CVE page(s) listed in the References section." link:"https://access.redhat.com/errata/RHSA-2022:7089" metadata_v2:<published_date_time:"2022-10-24T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H" score:8.1 exploitability_score:2.2 impact_score:5.9 > > fixed_by:"0:1.3.5-8.el8_6" severity:"Important" > fixed_by:"1.3.5-8.el8_6"  name:"tar" version:"1.27.1.el8.x86_64" feature_type:"rpm" vulnerabilities:<name:"CVE-2005-2541" description:"DOCUMENTATION: The MITRE CVE dictionary describes this issue as: Tar 1.15.1 does not properly warn the user when extracting setuid or setgid files, which may allow local users or remote attackers to gain privileges. \n            STATEMENT: This CVE was assigned to what is the documented and expected behaviour of tar.  There are currently no plans to change tar behaviour to strip setuid and setgid bits when extracting archives." link:"https://access.redhat.com/security/cve/CVE-2005-2541" metadata_v2:<last_modified_date_time:"2021-06-28T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:L/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H" score:7 exploitability_score:1 impact_score:5.9 > > severity:"Moderate" > vulnerabilities:<name:"CVE-2019-9923" description:"DOCUMENTATION: The MITRE CVE dictionary describes this issue as: pax_decode_header in sparse.c in GNU Tar before 1.32 had a NULL pointer dereference when parsing certain archives that have malformed extended headers." link:"https://access.redhat.com/security/cve/CVE-2019-9923" metadata_v2:<last_modified_date_time:"2021-10-27T00:00Z" cvss_v3:<vector:"CVSS:3.0/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L" score:3.3 exploitability_score:1.8 impact_score:1.4 > > severity:"Low" > vulnerabilities:<name:"CVE-2021-20193" description:"DOCUMENTATION: A flaw was found in the src/list.c of tar. This flaw allows an attacker who can submit a crafted input file to tar to cause uncontrolled consumption of memory. The highest threat from this vulnerability is to system availability." link:"https://access.redhat.com/security/cve/CVE-2021-20193" metadata_v2:<last_modified_date_time:"2022-04-17T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L" score:3.3 exploitability_score:1.8 impact_score:1.4 > > severity:"Low" > ]
--- PASS: TestGRPCGetRHCOSNodeVulnerabilities (4.51s)
    --- PASS: TestGRPCGetRHCOSNodeVulnerabilities/case-0 (4.51s)

Now I would be interested how can we enable the feature flags for e2e tests in the PRs and master, but keep it disabled (if the default is false) for releases.

After enabling the feature flag in code (setting default to true), the test executes properly:
(from https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/stackrox_scanner/1004/pull-ci-stackrox-scanner-master-e2e-tests/1595484736086609920/artifacts/e2e-tests/e2e-tests/build-log.txt)
=== RUN   TestGRPCGetRHCOSNodeVulnerabilities
=== RUN   TestGRPCGetRHCOSNodeVulnerabilities/case-0
Handling connection for 8443
    node_scan_rhcos_test.go:141: gotFeat.InventoryFeatures: [name:"libksba" version:"1.3.5-7.el8.x86_64" feature_type:"rpm" vulnerabilities:<name:"RHSA-2022:7089" description:"KSBA (pronounced Kasbah) is a library to make X.509 certificates as well as the CMS easily accessible by other applications.  Both specifications are building blocks of S/MIME and TLS.\n\nSecurity Fix(es):\n\n* libksba: integer overflow may lead to remote code execution (CVE-2022-3515)\n\nFor more details about the security issue(s), including the impact, a CVSS score, acknowledgments, and other related information, refer to the CVE page(s) listed in the References section." link:"https://access.redhat.com/errata/RHSA-2022:7089" metadata_v2:<published_date_time:"2022-10-24T00:00Z" last_modified_date_time:"2022-10-24T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H" score:8.1 exploitability_score:2.2 impact_score:5.9 > > fixed_by:"0:1.3.5-8.el8_6" severity:"Important" > fixed_by:"1.3.5-8.el8_6"  name:"tar" version:"1.27.1.el8.x86_64" feature_type:"rpm" vulnerabilities:<name:"CVE-2005-2541" description:"DOCUMENTATION: The MITRE CVE dictionary describes this issue as: Tar 1.15.1 does not properly warn the user when extracting setuid or setgid files, which may allow local users or remote attackers to gain privileges. \n            STATEMENT: This CVE was assigned to what is the documented and expected behaviour of tar.  There are currently no plans to change tar behaviour to strip setuid and setgid bits when extracting archives." link:"https://access.redhat.com/security/cve/CVE-2005-2541" metadata_v2:<last_modified_date_time:"2021-06-28T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:L/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H" score:7 exploitability_score:1 impact_score:5.9 > > severity:"Moderate" > vulnerabilities:<name:"CVE-2019-9923" description:"DOCUMENTATION: The MITRE CVE dictionary describes this issue as: pax_decode_header in sparse.c in GNU Tar before 1.32 had a NULL pointer dereference when parsing certain archives that have malformed extended headers." link:"https://access.redhat.com/security/cve/CVE-2019-9923" metadata_v2:<last_modified_date_time:"2021-10-27T00:00Z" cvss_v3:<vector:"CVSS:3.0/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L" score:3.3 exploitability_score:1.8 impact_score:1.4 > > severity:"Low" > vulnerabilities:<name:"CVE-2021-20193" description:"DOCUMENTATION: A flaw was found in the src/list.c of tar. This flaw allows an attacker who can submit a crafted input file to tar to cause uncontrolled consumption of memory. The highest threat from this vulnerability is to system availability." link:"https://access.redhat.com/security/cve/CVE-2021-20193" metadata_v2:<last_modified_date_time:"2022-04-17T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L" score:3.3 exploitability_score:1.8 impact_score:1.4 > > severity:"Low" > ]
    node_scan_rhcos_test.go:141: gotFeat.InventoryFeatures: [name:"libksba" version:"1.3.5-7.el8.x86_64" feature_type:"rpm" vulnerabilities:<name:"RHSA-2022:7089" description:"KSBA (pronounced Kasbah) is a library to make X.509 certificates as well as the CMS easily accessible by other applications.  Both specifications are building blocks of S/MIME and TLS.\n\nSecurity Fix(es):\n\n* libksba: integer overflow may lead to remote code execution (CVE-2022-3515)\n\nFor more details about the security issue(s), including the impact, a CVSS score, acknowledgments, and other related information, refer to the CVE page(s) listed in the References section." link:"https://access.redhat.com/errata/RHSA-2022:7089" metadata_v2:<published_date_time:"2022-10-24T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H" score:8.1 exploitability_score:2.2 impact_score:5.9 > > fixed_by:"0:1.3.5-8.el8_6" severity:"Important" > fixed_by:"1.3.5-8.el8_6"  name:"tar" version:"1.27.1.el8.x86_64" feature_type:"rpm" vulnerabilities:<name:"CVE-2005-2541" description:"DOCUMENTATION: The MITRE CVE dictionary describes this issue as: Tar 1.15.1 does not properly warn the user when extracting setuid or setgid files, which may allow local users or remote attackers to gain privileges. \n            STATEMENT: This CVE was assigned to what is the documented and expected behaviour of tar.  There are currently no plans to change tar behaviour to strip setuid and setgid bits when extracting archives." link:"https://access.redhat.com/security/cve/CVE-2005-2541" metadata_v2:<last_modified_date_time:"2021-06-28T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:L/AC:H/PR:N/UI:R/S:U/C:H/I:H/A:H" score:7 exploitability_score:1 impact_score:5.9 > > severity:"Moderate" > vulnerabilities:<name:"CVE-2019-9923" description:"DOCUMENTATION: The MITRE CVE dictionary describes this issue as: pax_decode_header in sparse.c in GNU Tar before 1.32 had a NULL pointer dereference when parsing certain archives that have malformed extended headers." link:"https://access.redhat.com/security/cve/CVE-2019-9923" metadata_v2:<last_modified_date_time:"2021-10-27T00:00Z" cvss_v3:<vector:"CVSS:3.0/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L" score:3.3 exploitability_score:1.8 impact_score:1.4 > > severity:"Low" > vulnerabilities:<name:"CVE-2021-20193" description:"DOCUMENTATION: A flaw was found in the src/list.c of tar. This flaw allows an attacker who can submit a crafted input file to tar to cause uncontrolled consumption of memory. The highest threat from this vulnerability is to system availability." link:"https://access.redhat.com/security/cve/CVE-2021-20193" metadata_v2:<last_modified_date_time:"2022-04-17T00:00Z" cvss_v3:<vector:"CVSS:3.1/AV:L/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L" score:3.3 exploitability_score:1.8 impact_score:1.4 > > severity:"Low" > ]
--- PASS: TestGRPCGetRHCOSNodeVulnerabilities (4.51s)
    --- PASS: TestGRPCGetRHCOSNodeVulnerabilities/case-0 (4.51s)

Now I would be interested how can we enable the feature flags for e2e tests in the PRs and master, but keep it disabled (if the default is false) for releases.

Scanner has never really differentiated much between release and dev before for this kind of thing. Something like this, for example, cannot even be used unless Central requests for it anyway, as the network policy dictates only Central can make requests to Scanner. Scanner does not even enforce use of the default, only, when it's a release vs dev (like the stackrox repo does) because it never really had such a need before

Now I would be interested how can we enable the feature flags for e2e tests in the PRs and master, but keep it disabled (if the default is false) for releases.

Scanner has never really differentiated much between release and dev before for this kind of thing. Something like this, for example, cannot even be used unless Central requests for it anyway, as the network policy dictates only Central can make requests to Scanner. Scanner does not even enforce use of the default, only, when it's a release vs dev (like the stackrox repo does) because it never really had such a need before

Would this suggest to use the feature flag in central and remove it from scanner?

Now I would be interested how can we enable the feature flags for e2e tests in the PRs and master, but keep it disabled (if the default is false) for releases.

Scanner has never really differentiated much between release and dev before for this kind of thing. Something like this, for example, cannot even be used unless Central requests for it anyway, as the network policy dictates only Central can make requests to Scanner. Scanner does not even enforce use of the default, only, when it's a release vs dev (like the stackrox repo does) because it never really had such a need before

Would this suggest to use the feature flag in central and remove it from scanner?

I'd say for now just enable the flag, and we can disable it before next release, if needed

Asked about the e2e failure in https://srox.slack.com/archives/C033Z8KMZAM/p1670597763818699

/test e2e-tests

I wanted to answer to this comment, but cannot find a way to do so :/

Well, the problem is that grep which currently serves as an example package without vulnerabilities may get vulnerabilities detected in the future. The comparison with 0 will start failing then.
We can either:

do nothing now and address the issue by changing the package when tests start breaking,
find a different package that does so little that it will not have vulnerabilities (e.g. in Debian there are meta-packages that have empty contents but bring many transitive dependencies once installed),
see how to use a snapshot of vulnerability data in test so that you can be confident that detected vulnerabilities don't change.
I'm fine with any of these options.

I acknowledge these options. While I tend to move on with the first option (keep as it is), I will add a comment stating that these tests may start failing in the future when new vulnerabilities are discovered for grep and the test would need to be adapted.
Regarding the different package than grep - I am open for suggestions. Running rpm -qa did not return any virtual packages as known from Debian, but I see that people ask in the Internet how to build a virtual package on their own using rpm.

The e2e test failure is consistently appearing but unrelated to this PR:
TestGRPCGetImageVulnerabilities/quay.io/rhacs-eng/qa:ose-jenkins/jenkins-2-plugins/4.10.1650890594-1.el8.noarch
 Messages:   	Expected to find CVE-2022-30945 in scan results

Regarding the different package than grep - I am open for suggestions. Running rpm -qa did not return any virtual packages as known from Debian, but I see that people ask in the Internet how to build a virtual package on their own using rpm.

The answer was couple more attempts away (10 minutes).
$ docker run --rm -it --entrypoint=/bin/bash registry.access.redhat.com/ubi8/ubi:8.7
$ dnf install less
$ dnf search '*' | grep noarch | sort | less

The ones which seemed most innocent:
git-core-doc.noarch
golang-docs.noarch
iso-codes.noarch
tzdata.noarch

authors: RTann msugakov vikin91

repo: stackrox/scanner


text:
fix(ci): qa:apache-server-scannerci

example failure
The vulns listed in the test cases associated with cron and ubuntu:14.04 in image quay.io/rhacs-eng/qa:apache-server-scannerci are no longer found. The test case has been updated to check for vulns from a different package.
The new vulns were chosen at random from the scan results (reviewers please share if there is a reason this particular test case exists and if different vulns should be chosen, of note the new vulns have fixedBy's but the old ones did not)
By inspecting the last 'genesis dump' from CI for one of the vulns in question confirmed it is no longer listed for ubuntu:14.04:
cat os_vulns.json | jq -r '.[] | select(.Name == "CVE-2017-9525") | "\(.Name) \(.Namespace.Name) \(.Namespace.VersionFormat) \(.Link)"' | grep -i ubuntu | sort | column -s " " -t

CVE-2017-9525  ubuntu:16.04     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:18.04     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:19.10     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:20.04     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:20.10     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:21.04     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:21.10     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:22.04     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:22.10     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:23.04     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:23.10     dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:24.04     dpkg  https://ubuntu.com/security/CVE-2017-9525

Compared to the genesis dump for StackRox 4.6.1, the vuln is indeed listed for ubuntu:14.04:
gsutil cp gs://stackrox-scanner-ci-vuln-dump/genesis-20241024031240.zip .
...
cat os_vulns.json | jq -r '.[] | select(.Name == "CVE-2017-9525") | "\(.Name) \(.Namespace.Name) \(.Namespace.VersionFormat) \(.Link)"' | grep -i ubuntu | sort | column -s " " -t

CVE-2017-9525  ubuntu:14.04  dpkg  https://ubuntu.com/security/CVE-2017-9525 <-- missing above
CVE-2017-9525  ubuntu:16.04  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:18.04  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:19.10  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:20.04  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:20.10  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:21.04  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:21.10  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:22.04  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:22.10  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:23.04  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:23.10  dpkg  https://ubuntu.com/security/CVE-2017-9525
CVE-2017-9525  ubuntu:24.04  dpkg  https://ubuntu.com/security/CVE-2017-9525

It appears that future new genesis dumps (CI creates a new one every run) will not contain this vuln for ubuntu:14.04.
The Ubuntu page for CVE-2017-9525 shows this (not sure if related or what the status was prior):

TBD: find out what changed in the source that led to these vulns no longer appearing

@dcaravel here's the change which triggered this: https://git.launchpad.net/ubuntu-cve-tracker/commit/active/CVE-2017-9525?id=43a92d54e18289468b0c0616c3d847c5fbe35284
This was done yesterday. More end-of-year fun for us

@dcaravel here's the change which triggered this: https://git.launchpad.net/ubuntu-cve-tracker/commit/active/CVE-2017-9525?id=43a92d54e18289468b0c0616c3d847c5fbe35284
This was done yesterday. More end-of-year fun for us

The vulns being omitted from the genesis dump due to the status change (to ignored) appears to be handled as expected.
Is there something additional that we should do outside of what's in this PR? (for example, if we should still include these vulns, we could adjust the logic to not omit vulns the 'end of ESM support' note or similar?)

authors: RTann dcaravel


text:
fix(Scanner): validate updates bundle version in v4 mode

Description
Validate offline bundle version in v4 mode
Add tests for mapping json download
Checklist

[x] Investigated and inspected CI test results
[x] Unit test and regression tests added

Testing Performed

Deploy

export OFFLINE_MODE=true
export ROX_SCANNER_V4=true
KUBECONFIG=/path/to/artifacts/kubeconfig deploy/k8s/deploy-local.sh

Check vuln data

unzip -l scanner-vuln-updates.zip
Archive:  scanner-vuln-updates.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
140154404  02-14-2024 11:41   scanner-defs.zip
   248088  02-14-2024 11:41   k8s-istio.zip
144211510  02-14-2024 11:41   scanner-v4-defs-4.3.zip
---------                     -------
284614002                     3 files

Upload
curl --insecure -u admin:[PWD] -F "file=@scanner-vuln-updates.zip" https://localhost:8000/api/extensions/scannerdefinitions -v

Successful results

* * We are completely uploaded and fine
< HTTP/2 200
< vary: Accept-Encoding
< content-type: text/plain; charset=utf-8
< content-length: 57
< date: Tue, 27 Feb 2024 23:35:09 GMT
<
* Connection #0 to host localhost left intact
Successfully stored the offline vulnerability definitions

curl --insecure -u admin:[pwd] https://localhost:8000/api/extensions/scannerdefinitions?file=repo2cpe > repo2cpe.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2041k  100 2041k    0     0  1040k      0  0:00:01  0:00:01 --:--:-- 1040k
ls -lh repo2cpe.json
-rw-r--r-- 1 yili staff 2.0M Feb 27 17:54 repo2cpe.json

Failure:
unzip -l scanner-vulns-4.2.zip
Archive:  scanner-vulns-4.2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
140524129  02-21-2024 19:41   scanner-defs.zip
   248088  02-21-2024 19:41   k8s-istio.zip
   112200  02-21-2024 19:41   scanner-v4-defs-4.2.zip
---------                     -------
140884417                     3 files

curl --insecure -u admin:[pwd] -F "file=@scanner-vulns-4.2.zip" https://localhost:8000/api/extensions/scannerdefinitions -v
> Content-Length: 140885152
> Content-Type: multipart/form-data; boundary=------------------------hJtnza1jCwuGMNr6NqswqY
>
* We are completely uploaded and fine
< HTTP/2 400
< vary: Accept-Encoding
< content-type: text/plain; charset=utf-8
< content-length: 233
< date: Tue, 27 Feb 2024 23:49:40 GMT
<
* Connection #0 to host localhost left intact
{"code":3,"message":"failed to upload offline file bundle, uploaded file is version: 4.2 and system version is: 4.3.x-1348-gd88b87b02b; please upload an offline bundle version: 4.3, consider using command roxctl scanner download-db"}

Reminder for reviewers
In addition to reviewing code here, reviewers must also review testing and request further testing in case the
performed one does not seem sufficient. As a reviewer, you must not approve the change until you understand the
performed testing and you are satisfied with it.

FYI, tested this out with roxctl scanner upload-db  in a few scenarios:
Bundle with manifest version 4.3, works as shown in PR.
$ roxctl scanner upload-db --scanner-db-file scanner-vulns-4.3.zip
2024/02/28 11:51:08 [DEBUG] POST https://34.75.104.156:443/api/extensions/scannerdefinitions

Successfully stored the offline vulnerability definitions

Bundle with manifest version 4.5, errors as expected:
$ roxctl scanner upload-db --scanner-db-file scanner-vulns-4.5.zip
2024/02/28 11:58:03 [DEBUG] POST https://34.75.104.156:443/api/extensions/scannerdefinitions

ERROR:	could not connect with scanner definitions API: expected status code 200, but received 400. Response Body: {"code":3,"message":"failed to upload offline file bundle, uploaded file is version: 4.5 and system version is: 4.3.x-1358-g9db19c9878; please upload an offline bundle version: 4.3, consider using command roxctl scanner download-db"}

Bundle with version that matches exactly the running version of Central (would expect this to succeed as mentioned in my earlier review, but for completeness sake felt should be tested):
$ roxctl scanner upload-db --scanner-db-file scanner-vulns-4.3.x-1358-g9db19c9878.zip

ERROR:	could not connect with scanner definitions API: expected status code 200, but received 400. Response Body: {"code":3,"message":"failed to upload offline file bundle, uploaded file is version: 4.3.x-1358-g9db19c9878 and system version is: 4.3.x-1358-g9db19c9878; please upload an offline bundle version: 4.3, consider using command roxctl scanner download-db"}

authors: daynewlee dcaravel

repo: stackrox/stackrox


text:
test(compliance): Add update scan configuration e2e test

Description

This PR adds an e2e test for the update configuration path.
User-facing documentation

[x] CHANGELOG update is not needed
[x] Documentation is not needed

Testing

[ ] inspected CI results

Automated testing

[x] added e2e tests

How I validated my change

[x] Running manually the e2e tests
[ ] CI

/test ocp-4-12-compliance-e2e-tests


text:
feat(notifications): notification service

Description
A detailed explanation of the changes in your PR.
Feel free to remove this section if it is overkill for your PR, and the title of your PR is sufficiently descriptive.
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
Testing Performed
TODO(replace-me)
Use this space to explain how you tested your PR, or, if you didn't test it, why
you did not do so. Valid reasons include, for example, "CI is sufficient",
"No testable changes". Feel free to attach JSON snippets, curl commands,
screenshots.
In addition to reviewing your code, reviewers must also review your testing
instructions and make sure they are sufficient.

Current dependencies on/for this PR:

master

PR #7509

PR #7510   👈

This comment was auto-generated by Graphite.


text:
ENH: Use sympy.Interval?

Having spent the better part of today trying to use this with timestamp Intervals, I gave up due to type fragility -- not just with this library but also pandas which provides datetime class timestamps with read_csv which it refuses to accept in calls to create Intervals. But then I ran into problems with this library being unable to handle certain arguments to closed for certain calls.
What may work better as an implementation than the approach you've taken would be to wrap the sympy.Interval class, converting to 64bit integer internally but keeping around the originating class.  Although this is slower (probably) the ostensible purpose of this library is a stop-gap anyway.

Hi James, thanks for the feedback and idea.  You're right in that this library is a "stop-gap" until pandas implements the functionality, and I assume they'll take a completely different approach using Interval Trees.  This project was started as there was some "low hanging fruit" that could be achieved provided certain assumptions were satisfied, which they seem to be for most use cases.
Are you able to expand on the problems you ran into with pandas?

authors: jabowery venaturum

repo: staircase-dev/piso


text:
Explicitly expose port 8080

Port 8080 must be explicitly exposed
EXPOSE 8080

won't make sense to expose this on base docker image; so, won't do / won't fix

repo: stakater/dockerfile-base-centos


text:
🛑 Jenkins Server is down

In 9ec9a78, Jenkins Server (https://jenkins01.its-telekom.eu/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Jenkins Server is back up in c8dcae7.

repo: stamateas/upptime


text:
⚠ GitLab Server has degraded performance

In 6478916, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 7654 ms

Resolved: GitLab Server performance has improved in 13fffa9.


text:
https does not work on maps.stamen.com

Not strictly related to terrain-classic, but since you've used https in the email and you probably want to launch with the https version, you should fix the certificate. It'll also need a Leaflet hosted from a https CDN, like npmcdn.

The link in the email was a mistake. We intend to support HTTPS as some point, but probably not until after we launch.
Thanks for the heads up!

authors: hyperknot mojodna

repo: stamen/terrain-classic


text:
Remove stanc2

Submisison Checklist
Remove stanc2 makefile logic and update readme.
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Rok Češnovar
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

The code was removed from the stan repository so this is ready as well.

The code was removed from the stan repository so this is ready as well.

repo: stan-dev/cmdstan


text:
Inconsistent error handling

Description
Error handling is not consistent between different mathematical functions. Most primitive functions only call std implementation and return NaN in case of invalid input. Some check for invalid input and throw exceptions.
Example
stan::math::log() directely calls std::log(). So any negative argument results in NaN return value.
stan::math::log1p() checks if its argument is less than -1 and throws std::domain_error if it is.
Expected Output
Consistent error handling.
Current Version:
v3.0.0

Checking for invalid values and throwing exceptions is slightly worse from performance standpoint. However it could detect problems earlier. Which one should we use?

This has been a longstanding discussion.
One thing I think we all agree on is that the primitive versions of the functions should match the exception behavior of the autodiff versions.  That's what the test framework tests for right now for test cases that throw exceptions for one or more inputs.
We originally implemented autodiff functions that overloaded the standard math library to not throw.  Over time, that's drifted as multiple developers and reviewers bring different perspectives.
I'd be OK making everything throw as early as possible.  That'd mean fixing a whole lot of the stan::math overloads of std:: functions and making sure the stan::math versions get called rather than the std:: ones in code.

On Nov 28, 2019, at 6:09 AM, Tadej Ciglarič notifications@github.com wrote:
Checking for invalid values and throwing exceptions is slightly worse from performance standpoint. However it could detect problems earlier. Which one should we use?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Having checks everywhere seems a bit too much to me. We can get an infinity by dividing a large value by a small one or even by adding together two huge numbers. In these cases checks would likely more expensive than computations. I think checks make more sense where the computation is orders of magnitude more expensive than check, such as matrix decompositions or matrix multiplication.
Actually log1p() seems to be the only primitive function that contains such a check. I would prefere just removing this one check.

I'm OK removing the check on log1p.  I like having the standard library functions match the std:: behavior.
The motivation for having the checks is that it's easier to debug if a program fails earlier because the failure will be closer to the source of the problem.  It's hard to evaluate how much that's worth compared to efficiency.
There's also a distinction between errors like returning infinity or NaN and errors like indexes out of bounds that will segfault and bring down interfaces like R or Python.
Checks on basic arithmetic would be much more expensive than the computation.  It's much closer with autodiff, because that always involves an interpreted call on the reverse pass with more arithmetic.

On Nov 28, 2019, at 10:16 AM, Tadej Ciglarič notifications@github.com wrote:
Having checks everywhere seems a bit too much to me. We can get an infinity by dividing a large value by a small one or even by adding together two huge numbers. In these cases checks would likely more expensive than computations. I think checks make more sense where the computation is orders of magnitude more expensive than check, such as matrix decompositions or matrix multiplication.
Actually log1p() seems to be the only primitive function that contains such a check. I would prefere just removing this one check.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

+1 to the whole conversation.
One thing I’d request is evaluating performance. If the performance is the
same, I think it’s clear that it’s better to fail early. If there is a
clear performance hit, then it’s worth considering removing it, changing
the contract on the function, and having the errors propagated for the sake
of speed.
In lots of the checks, we haven’t demonstrated any change to performance by
removing checks when there are no exceptional conditions. That’s one of the
reasons these are still in is because the simple ones are really optimized
very well by the C++ compilers. That’s not to say they don’t affect
performance; we just haven’t been able to spot it under the optimization
levels we compile at with the compilers we use.
On Thu, Nov 28, 2019 at 10:42 AM Bob Carpenter notifications@github.com
wrote:

I'm OK removing the check on log1p. I like having the standard library
functions match the std:: behavior.
The motivation for having the checks is that it's easier to debug if a
program fails earlier because the failure will be closer to the source of
the problem. It's hard to evaluate how much that's worth compared to
efficiency.
There's also a distinction between errors like returning infinity or NaN
and errors like indexes out of bounds that will segfault and bring down
interfaces like R or Python.
Checks on basic arithmetic would be much more expensive than the
computation. It's much closer with autodiff, because that always involves
an interpreted call on the reverse pass with more arithmetic.

On Nov 28, 2019, at 10:16 AM, Tadej Ciglarič notifications@github.com
wrote:
Having checks everywhere seems a bit too much to me. We can get an
infinity by dividing a large value by a small one or even by adding
together two huge numbers. In these cases checks would likely more
expensive than computations. I think checks make more sense where the
computation is orders of magnitude more expensive than check, such as
matrix decompositions or matrix multiplication.
Actually log1p() seems to be the only primitive function that contains
such a check. I would prefere just removing this one check.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/stan-dev/math/issues/1468?email_source=notifications&email_token=AADH6F6HYC6MJ4XOH3T6GJLQV7RHJA5CNFSM4JSS4SZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFM6HAQ#issuecomment-559539074,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADH6F3WCYDIDEV7UD7NXBDQV7RHJANCNFSM4JSS4SZA
.

I understand the benefits to debugging. I found this inconsistency while trying to use implementations from Eigen, some of which are vectorized and faster than std:: variants. However it would be quite hard using those vectorized implemantations and checks at the same time. So the reason for opening this issue is essentially asking whether these can be removed.
I will do a benchmark tomorrow, but I am not sure if I will be able to show performance difference, as log1p() is quite expensive to compute compared to a check.

Ohand I also missed it earlier, but log1m also does a similar check.

The presence of checks in some functions is also coming from this pull, where boost functions were replaced with std functions:
https://github.com/stan-dev/math/issues/1010#issuecomment-445527864
The discussion about whether to maintain throwing behaviour (for consistency with boost) or not, was in this thread: https://github.com/stan-dev/math/pull/1067#issuecomment-443400498

From: Tadej Ciglarič notifications@github.com
Sent: Friday, November 29, 2019 7:23:28 AM
To: stan-dev/math math@noreply.github.com
Cc: Subscribed subscribed@noreply.github.com
Subject: Re: [stan-dev/math] Inconsistent error handling (#1468)
Ohand I also missed it earlier, but log1m also does a similar check.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/stan-dev/math/issues/1468?email_source=notifications&email_token=AGTPCCGOOTSPT4AWG5BHTTTQWASEBA5CNFSM4JSS4SZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFNMNAQ#issuecomment-559597186, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGTPCCB25TBGIMZZ2OUVO6LQWASEBANCNFSM4JSS4SZA.

Thanks for the links. So to summatize: there was a lot of interest in removing of all throws, but the end decision was not to make changes to when functions throw. So I guess I can just close this issue?
Also I think if issue #1449 gets resolved in a way that check can be turned off into no-ops, than their performance is not an issue anymore. So we can put them into every single function.

That was the decision then, but the conversation so far sounds like the decision could be more performance-driven

Here is the benchmark and results:
https://gist.github.com/t4c1/ba3231f488ae217dbb22c1e064a84b75
I compare std and eigen implementations of functions. I also figured a simple way to add checks to Eigen implementations, so both versions are also tested with check_infinite() (i just picked one check function - they should be similar performance-wise). Std versions have checks added in two different ways. First is to check the container beforehand - this is often used in more complex functions that work on matrices in stan. Second is checking each value as it is used in calculations - that is how log1p() and log1m() work.
With default compiler options I see little difference for all functions, except for exp() and sqrt(). Eigen seems to have much faster implementation for these two. If we use Eigen implementation for these slowdowns from checks become significant.
Than I figured that my compiler by default does not enable any vectorization. I reran the benchmarks with -march=native, which on my CPU enables AVX instuction set. In this case also exp(), sqrt(), floor(), round() and ceil() show significant speedup from using Eigen and slowdown from checks.

On Nov 28, 2019, at 11:25 AM, Daniel Lee notifications@github.com wrote:
+1 to the whole conversation.
One thing I’d request is evaluating performance. If the performance is the
same, I think it’s clear that it’s better to fail early.

I think consistency is important, too.  As is, log1p is the only function
overload of a std:: function that throws.  This change would simply make
the whole interface more consistent.
I think it'd be possible to add checks to other functions.  We've
been going around on this one for ages.
This is not about speed.  The speed of log1p is already slow.  It
already has branching in it to deal with precision issues.

If there is a
clear performance hit, then it’s worth considering removing it, changing
the contract on the function, and having the errors propagated for the sake
of speed.

The isnan check was introduced Dec 9 2018 with this commit:
https://github.com/stan-dev/math/commit/01bb6645815cd5546701c0575fcb4f38d4a41cca
that added the test and throw behavior.  It was in line with the other functions before that, relying on our boost policy type to return error codes rather than raise exceptions.  See:
https://github.com/stan-dev/math/blob/develop/stan/math/prim/scal/fun/boost_policy.hpp

Wow.  That's interesting news on the performance of Eigen implementations of exp() and sqrt().  Do they satisfy the IEEE standard or are they taking shortcuts?  If they're accurate enough, we should be able to just plug and play.
There are also ways to turn down compiler implementations of checks and lower precision to get speedups (which is partly what the Intel library does).
Do we not he -march=native turned on in our compilation?  If not, should we?

On Nov 29, 2019, at 5:22 AM, Tadej Ciglarič notifications@github.com wrote:
Here is the benchmark and results:
https://gist.github.com/t4c1/ba3231f488ae217dbb22c1e064a84b75
I compare std and eigen implementations of functions. I also figured a simple way to add checks to Eigen implementations, so both versions are also tested with check_infinite() (i just picked one check function - they should be similar performance-wise). Std versions have checks added in two different ways. First is to check the container beforehand - this is often used in more complex functions that work on matrices in stan. Second is checking each value as it is used in calculations - that is how log1p() and log1m() work.
With default compiler options I see little difference for all functions, except for exp() and sqrt(). Eigen seems to have much faster implementation for these two. If we use Eigen implementation for these slowdowns from checks become significant.
Than I figured that my compiler by default does not enable any vectorization. I reran the benchmarks with -march=native, which on my CPU enables AVX instuction set. In this case also exp(), sqrt(), floor(), round() and ceil() show significant speedup from using Eigen and slowdown from checks.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

Do they satisfy the IEEE standard or are they taking shortcuts?

No idea. This page seems to vaguely claim they are based on standard library.

Do we not he -march=native turned on in our compilation?  If not, should we?

I think we do not. In most cases Stan programs are run on the same computer they are compiled on, so we could use -march=native. But there might be exceptions? I don't know.
Even if there are exceptions we could document that in most cases user can add that to make/local and expect improved performance.

We used to recommend -march=native in the RStan install instructions.  I should just have it set in my make/local.   I think -march=native is safe.
I'm not sure where to put all these instructions.

On Nov 30, 2019, at 4:15 AM, Tadej Ciglarič notifications@github.com wrote:
Do they satisfy the IEEE standard or are they taking shortcuts?
No idea. This page seems to vaguely claim they are based on standard library.
Do we not he -march=native turned on in our compilation? If not, should we?
I think we do not. In most cases Stan programs are run on the same computer they are compiled on, so we could use -march=native. But there might be exceptions? I don't know.
Even if there are exceptions we could document that in most cases user can add that to make/local and expect improved performance.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

authors: andrjohns bob-carpenter syclik t4c1

repo: stan-dev/math


text:
add inv_Phi_log which is inv_Phi(log(x))

@andrjohns has done some tests on inv_Phi and it's more stable to compute everything in terms of logs. He has a Stan implementation at https://github.com/andrjohns/brms/blob/feature/gaussian-copula/inst/chunks/fun_inv_Phi_log.stan.
It is exactly the same algorithm as in stan-math but allows the user to input the value on the log scale. Doing all the interval calculations on this scale but still returning the same result as inv_Phi.
> inv_Phi_log(log(0.3))
[1] -0.5244005
> qnorm(0.3)
[1] -0.5244005

It certainly gives you more range.  It's also the same result as normal_lcdf(x | 0, 1), so it'd be good to improve normal_lcdf if there are algorithmic improvements here.
Can we call this log_inv_Phi, though, to match our other functions (e.g., log_inv_logit, log_1m_inv_logit, etc.)?

The naming is a bit tricky, since it's not quite the same as the other log_ functions. log_inv_Phi implies that the function is log(inv_Phi(x)), but it's actually inv_Phi(exp(x)).
Also, I might be mistaken, but isn't this distinct from the normal_lcdf since inv_Phi is the quantile function of the standard normal, not the CDF?

Sorry, @andrjohns---if that's the behavior, then your name is correct and it's not synonymous with the normal log cdf.

authors: andrjohns bob-carpenter spinkney


text:
👎 on Rails/SaveBang

From the docs:

This cop identifies possible cases where Active Record save! or related should be used instead of save because the model might have failed to save and an exception is better than unhandled failure.

This will ignore calls that return a boolean for success if the result is assigned to a variable or used as the condition in an if/unless statement. It will also ignore calls that return a model assigned to a variable that has a call to ‘persisted?`. Finally, it will ignore any call with more than 2 arguments as that is likely not an Active Record call or a Model.update(id, attributes) call.

https://www.rubydoc.info/gems/rubocop/0.44.1/RuboCop/Cop/Rails/SaveBang
I'd like to advocate for removing this from Standard Ruby's plugin for Rails. It's too aggressive and doesn't seem able to focus on only ActiveRecord code.
👍 Violations that are AR models
When the violation is on an AR model, then I'm totally fine with a violation being triggered!
class Artwork < ApplicationRecord
end

# please save me from myself!
artwork = Artwork.create(params)

👍 / 👎 Violations in code we own
If we write a class that does not inherit from AR then I should be able to define a method called create and that should not be a violation:
class ArtworkBuilder
  def create
  end
end

# It's possible that this violation exposes a code smell but I still think I should be able to do this!
artwork = ArtworkBuilder.new.create(params)

It's true that since this is code we own we could consider this violation an indication that we have a code smell but imho that's too aggressive. I could live with this but...
👎 Violations in code we don't own
The worst-case scenario is when the violation is triggered by code we don't own!
stripe_customer_id = Stripe::Customer.create(param[:user_id])
order_attrs = {
  amount_cents: params[:amount_cents],
  stripe_customer_id: stripe_customer_id
}
order = Order.create!(order_attrs)

Here I have a gem that contains a class that violates the rule - what should I do to comply with this rule??

:+1:

Related issue: rubocop/rubocop-rails#1154.

This rule is similar to the recently disabled - https://github.com/standardrb/standard-rails/pull/36 - Rails/PluckId ... both are disabled in rubocop-rails, and for same reason (can't safely correct, false positives likely for similar reasons).
That said, I think this one is both a) possibly more useful in finding real issues, b) slightly less bad in it's finding false positives - because of the AllowedReceivers configuration option.
However, in terms of including in stock standard-rails ... that configuration option is going to be project specific, and there's no way to come up with reasonable defaults there.
Assuming we want a default "don't enable something in standard-rails that rubocop-rails doesnt distribute as enabled", I think this one would be good to disable as well.

Yeah let's disable it. I'd merge a PR

Disabled in https://github.com/standardrb/standard-rails/pull/37, can close this.

authors: jonallured krystof-k mjankowski searls

repo: standardrb/standard-rails