text:
左键框选范围的时候 线  不会被选中， getSelectedCells拿不到应该选中的线

Describe the bug
左键框选范围的时候 线  不会被选中， getSelectedCells拿不到应该选中的线，只能选中节点。目前我的节点是自定义的vue节点，节点可以选中，线不行
Your Example Website or App
暂无
Steps to Reproduce the Bug or Issue
左键框选范围的时候 线  不会被选中， getSelectedCells拿不到应该选中的线，只能选中节点。目前我的节点是自定义的vue节点，节点可以选中，线不行
Expected behavior
左键框选范围的时候 线  不会被选中， getSelectedCells拿不到应该选中的线，只能选中节点。目前我的节点是自定义的vue节点，节点可以选中，线不行
Screenshots or Videos

Platform

OS: [e.g. macOS, Windows, Linux]
Browser: [e.g. Chrome, Safari, Firefox]
Version: [e.g. 91.1]
Windows   Chrome 应该都有

Additional context
No response

需要开启 rubberEdge 配置，才能选中连线。

authors: NewByVector angular-nz


text:
自定义节点如何缩放节点

问题描述
当我在自定义一个节点后，创建这个节点，但是缩放框异常了，拖动缩放框，图形并没有跟着缩放框改变

我的代码是这样的

麻烦帮我看看是不是有什么问题
重现链接
https://stackblitz.com/edit/vitejs-vite-qaxtja?file=package.json,src%2Fcomponents%2FHelloWorld.vue,src%2Fcomponents%2Fgraph.js,src%2FApp.vue,src%2Fcomponents%2Findex.js
重现步骤
1.我使用registerNode注册了一个节点类型，2.我在画布中创建了这个节点
问题在于：我不知道为什么缩放框失效了，它和图形直接没有联系了，图形没有跟着缩放框进行缩放
预期行为
希望能和https://x6.antv.vision/zh/examples/showcase/practices#flowchart中的基本流程图的图形一样可以正常缩放就行
平台

操作系统: [Windows]
网页浏览器: [Google Chrome]
X6 版本: [1.28]

屏幕截图或视频（可选）

补充说明（可选）
求大佬帮忙瞅瞅  309520291@qq.com

可能要自己resize一下

source: gharchive/issue

license: MIT

license_type: permissive

license_source: github-api


text:
支持js object实例化chart

用json比语法编程会不会更简洁、更友好？每一项都要调api配置太累了，也不易复用
var echarts = require('echarts');
var myChart = echarts.init(document.getElementById('main'));
// 绘制图表
myChart.setOption({
title: { text: 'ECharts 入门示例' },
tooltip: {},
xAxis: {
data: ["衬衫","羊毛衫","雪纺衫","裤子","高跟鞋","袜子"]
},
yAxis: {},
series: [{
name: '销量',
type: 'bar',
data: [5, 20, 36, 10, 10, 20]
}]
});

这个我们目前正在整，四月中旬发布 2.3.0 会支持

跳票了 不开心

啥？ 乖宝宝，偶们支持了呢~~~~ https://antv.alipay.com/g2/api/chart-options.html


text:
g2后续是否可以添加支持双X轴的功能呢

G2 Version: 3.2.4
Platform: windows10
Mini Showcase(like screenshots):
CodePen Link:

大概是像这种多X轴的对比图。
目前我通过官方的实例只做到了这一步

不知道是目前不支持这个功能，还是我打开的姿势不对呢？
如果支持，可否告知一下具体如何配置，如果不支持后续是否可以添加此功能呢？
感谢！

https://antv.alipay.com/zh-cn/gallery/g2/area-rainfall.html

去掉就不是双X轴了， 我这边想达到的效果是双X轴。然后共享一个Y轴，

大概是上图这样...

@leungwensen  谢谢您的回答， 我说的是公用一个y轴，好像创建一个view就会自动生成一个y轴吧？
不能实现就算了。感谢！

@pasBone 这个 case 的数据差异比较大，共用 y 轴的话也可以做，但效果不会很好。共用 y 轴的方法有两个：

把字段变成一个，然后 chart 设置 scale sync
给两个 y 轴字段 scale 设置相同的 min 和 max，然后隐藏掉其中一个 y 轴

参考 scale 和 axis 的文档：

https://antv.alipay.com/zh-cn/g2/3.x/api/chart.html#_chart.scale-field-,-scaleConfig-
https://antv.alipay.com/zh-cn/g2/3.x/api/chart.html#_chart.axis-false-

authors: leungwensen pasBone


text:
Very useful, thank you very much.

It's exactly what I need.

Thanks @EkkoGithub2021, instead giving reviews here it might be helpful to give reviews on jetbrains official plugins page 😅

Thanks @EkkoGithub2021, instead giving reviews here it might be helpful to give reviews on jetbrains official plugins page 😅


text:
BUG: Updating Twitter icon to X icon

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Assigned!
Make sure to attach before & after screenshots in your PR. Also, star the repo
before working on it. While creating your PR, ensure you pull all existing changes
beforehand to avoid conflicts.
Happy open-source contributing!

authors: RitiChandak SuhainaFathimaM


text:
BUG: CSS text visible at top

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?

There is some css text visible in the deployed site. This shouldnt be there.
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Assigned @vansh-sardana


text:
Removed duplicate hover effect from navbar

Related Issue
resolves #1484
Fixes:  #(issue no.)
Description
[Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.]

Type of PR

[x ] Bug fix
[ ] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
[Attach any relevant screenshots or videos demonstrating the changes. Make sure to attach before & after screenshots in your PR.]

Checklist:

[x ] I have made this change from my own.
[ ] I have taken help from some online resources.
[x ] My code follows the style guidelines of this project.
[x ] I have performed a self-review of my own code.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] I have made corresponding changes to the documentation.
[x ] My changes generate no new warnings.
[x ] I have tested the changes thoroughly before submitting this pull request.
[x ] I have provided relevant issue numbers and screenshots after making the changes.

@Amruta7203  please fix these

@Amruta7203 Fix it in every page. Cross check in your local host. Fix & open a fresh PR for review

source: gharchive/pull-request


text:
#made the navbar similar

Related Issue
[Cite any related issue(s) this pull request addresses. If none, simply state “None”]
Fixes:  #1834
Description
I have made the navbar of "about us" page similar to this rest of the pages.

Type of PR

[ ] Bug fix
[ ] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
before

home page navbar

about us navbar

Checklist:

[ ] I have made this change from my own.
[ ] I have taken help from some online resources.
[ ] My code follows the style guidelines of this project.
[ ] I have performed a self-review of my own code.
[ ] I have commented my code, particularly in hard-to-understand areas.
[ ] I have made corresponding changes to the documentation.
[ ] My changes generate no new warnings.
[ ] I have tested the changes thoroughly before submitting this pull request.
[ ] I have provided relevant issue numbers and screenshots after making the changes.

@huamanraj please assign me this pull request so that i can get my scores.

@nandani-goyal Done and sorry for the inconvenience caused

Thank u so much @RitiChandak for your efforts but I have still not got my scores shown in the leaderboard for this pull request.

Please can u tell me what could possible problem?

authors: RitiChandak nandani-goyal


text:
spectaql 2.0.0 can't be installed with npm

I am unable to install this tool after 2.0.0 was released. Older versions install just fine.
cosban@quasar:/Users/cosban/dev $
> npm --version
8.19.2
cosban@quasar:/Users/cosban/dev $
> node --version
v18.12.1
cosban@quasar:/Users/cosban/dev $
> npm i -g spectaql@2.0.0
npm WARN deprecated opn@6.0.0: The package has been renamed to `open`
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm ERR! code 127
npm ERR! path /Users/cosban/.nvm/versions/node/v18.12.1/lib/node_modules/spectaql
npm ERR! command failed
npm ERR! command sh -c -- yarn build:vendor
npm ERR! sh: yarn: command not found

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/cosban/.npm/_logs/2022-12-07T19_28_26_900Z-debug-0.log

For now, can you please install yarn on your machine and try again?

Yes, doing so works and allows the install to occur.

cosban@quasar:/Users/cosban/dev $
> npm install -g yarn

added 1 package, and audited 2 packages in 464ms

found 0 vulnerabilities
cosban@quasar:/Users/cosban/dev $
> npm install -g spectaql@2.0.0
npm WARN deprecated opn@6.0.0: The package has been renamed to `open`
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.

added 13 packages, removed 62 packages, changed 332 packages, and audited 346 packages in 7s

37 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Ok I'll get a fix out asap.

@cosban this should be fixed in main and spectaql@2.0.1. Can you remove yarn and try again to confirm?

Thanks for the quick turn around. I have confirmed that 2.0.1 resolves this issue.

authors: cosban newhouse

repo: anvilco/spectaql


text:
New account sync failed

Have you read a contributing guide?

[X] I have read CONTRIBUTING.md
[X] I have searched the existing issues and didn't find any that were similar
[X] I have considered creating a pull request with fixes instead of a bug report and want to proceed

Current Behavior
My first account works good
And the second one not sync.
Expected Behavior
Second account work properly
Steps To Reproduce

Create first account using client.yml on self hosted server
Create second account - not synced

Environment
- OS: Ubuntu, Iphone
- Version: 22.04, 15

Anything else?
No response

What I can do? Could this repo support multiple accounts in my self hosted docker environment?

I can't reproduce: I created two separate accounts, and both synced fine.
Account one:

Account two:

repo: anyproto/any-sync-dockercompose


text:
add get_stored_account_meta_callback to storage

Problem
moving to not mmapping append vec files. Soon, lifetimes of borrowed account data will be limited to a callback.
Summary of Changes
Add get_stored_account_meta_callback(offset) and implement a single caller. get_stored_account_meta_callback() can return an account with a lifetime scoped to the callback.
Soon, storages will not be mmapped. This will mean lifetimes are different (and shorter).
Fixes #

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.8%. Comparing base (2e91155) to head (245c391).
Report is 5 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##           master     #836     +/-   ##
=========================================
- Coverage    81.8%    81.8%   -0.1%
=========================================
  Files         851      851
  Lines      231724   231729      +5
=========================================
+ Hits       189647   189651      +4
- Misses      42077    42078      +1

authors: codecov-commenter jeffwashington

repo: anza-xyz/agave


text:
User with "Hide Files Page" should not be able to get any information…

If a user has the permission option "Hide Files Page", he should not be able to get any information from the files, including the filesize.
Relevant issue number(s) if applicable h1 #722386
License
I confirm that this contribution is made under an Apache 2.0 license and that I have the authority necessary to make this contribution on behalf of its copyright owner.

Implemented using middleware https://github.com/aol/moloch/pull/1273

repo: aol/moloch


text:
Instance test 2

This is a follow-up on #20:

Fix a bug with incorrect argument name
Allow task_instance[KEY_PREDICTION] and task_instance['test_patch'] to be empty.

I mentioned 2 in the previous PR: In an agent workflow, we want to run tests without a prediction to verify which are valid. We also want to run without a test patch so the agent can evaluate its own edits without 'leaking' any data from the patch.
I'm not sure if modifying swebench_docker/evaluate_instance.py requires any other changes, like rebuilding the docker images?

To get the updated swebench_docker/evaluate_instance.py into the docker images they have to be rebuilt. It takes some time to build everything, so I won't be able to do it right now. However, one way to test is to create new Docker images based on the existing ones and simply include this updated file.

authors: aorwall granawkins

repo: aorwall/SWE-bench-docker


text:
Caused by: java.lang.ClassNotFoundException: io.ap4k.deps.openshift.api.model.BuildSpec

This error is reported if I change the version of the ap4k deps to use 1.0-SNAPSHOT
and execute mvn clean install
[INFO] --- maven-install-plugin:2.4:install (default-install) @ ap4k-micronaut ---
[INFO] Installing /Users/dabou/Code/snowdrop/ap4k/ap4k-annotations/frameworks/micronaut/target/ap4k-micronaut-1.0-SNAPSHOT.jar to /Users/dabou/.m2/repository/io/ap4k/ap4k-micronaut/1.0-SNAPSHOT/ap4k-micronaut-1.0-SNAPSHOT.jar
[INFO] Installing /Users/dabou/Code/snowdrop/ap4k/ap4k-annotations/frameworks/micronaut/pom.xml to /Users/dabou/.m2/repository/io/ap4k/ap4k-micronaut/1.0-SNAPSHOT/ap4k-micronaut-1.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] AP4K 1.0-SNAPSHOT .................................. SUCCESS [  0.306 s]
[INFO] AP4K :: Core ....................................... SUCCESS [ 13.281 s]
[INFO] AP4K :: Annotations ................................ SUCCESS [  0.011 s]
[INFO] AP4K :: Annotations :: Kubernetes .................. SUCCESS [  3.253 s]
[INFO] AP4K :: Annotations :: Openshift ................... SUCCESS [  7.634 s]
[INFO] AP4K :: Annotations :: Istio ....................... SUCCESS [  4.190 s]
[INFO] AP4K :: Annotations :: Service Catalog ............. SUCCESS [  4.754 s]
[INFO] AP4K :: Annotations :: Component Operator .......... SUCCESS [  6.043 s]
[INFO] AP4K :: Frameworks ................................. SUCCESS [  0.013 s]
[INFO] AP4K :: Frameworks :: Spring Boot .................. SUCCESS [  1.701 s]
[INFO] AP4K :: Examples ................................... SUCCESS [  0.008 s]
[INFO] AP4K :: Examples :: Service Catalog ................ SUCCESS [  1.211 s]
[INFO] AP4K :: Examples :: Kubernetes ..................... SUCCESS [  2.825 s]
[INFO] AP4K :: Examples :: Openshift ...................... SUCCESS [  3.932 s]
[INFO] AP4K :: Examples :: Istio .......................... SUCCESS [  1.298 s]
[INFO] AP4K :: Examples :: Spring Boot on Kubernetes ...... SUCCESS [  2.922 s]
[INFO] AP4K :: Examples :: Spring Boot on Openshift ....... SUCCESS [  3.164 s]
[INFO] AP4K :: Examples :: Component Operator ............. SUCCESS [  0.856 s]
[INFO] AP4K :: Testing .................................... SUCCESS [  0.007 s]
[INFO] AP4K :: Testing :: Openshift ....................... SUCCESS [  0.890 s]
[INFO] AP4K :: Examples :: Source to Image Example ........ SUCCESS [ 54.237 s]
[INFO] AP4K :: Frameworks :: Thorntail .................... SUCCESS [  0.557 s]
[INFO] AP4K :: Frameworks :: Micronaut 1.0-SNAPSHOT ....... SUCCESS [  0.489 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:53 min
[INFO] Finished at: 2018-11-21T18:18:12+01:00
[INFO] ------------------------------------------------------------------------
Applied: s2i-java
Applied: source-to-image-example
Applied: source-to-image-example
Applied: source-to-image-example
Exception in thread "Thread-35" java.lang.NoClassDefFoundError: Lio/ap4k/deps/openshift/api/model/BuildSpec;
        at java.lang.Class.getDeclaredFields0(Native Method)
        at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
        at java.lang.Class.getDeclaredFields(Class.java:1916)
        at io.ap4k.deps.jackson.databind.util.ClassUtil$ClassMetadata.getDeclaredFields(ClassUtil.java:1087)
        at io.ap4k.deps.jackson.databind.util.ClassUtil.getDeclaredFields(ClassUtil.java:386)
        at io.ap4k.deps.jackson.databind.introspect.AnnotatedClass._findFields(AnnotatedClass.java:809)
        at io.ap4k.deps.jackson.databind.introspect.AnnotatedClass.resolveFields(AnnotatedClass.java:575)
        at io.ap4k.deps.jackson.databind.introspect.AnnotatedClass.fields(AnnotatedClass.java:353)
        at io.ap4k.deps.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:350)
        at io.ap4k.deps.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:283)
        at io.ap4k.deps.jackson.databind.introspect.POJOPropertiesCollector.getPropertyMap(POJOPropertiesCollector.java:248)
        at io.ap4k.deps.jackson.databind.introspect.POJOPropertiesCollector.getProperties(POJOPropertiesCollector.java:155)
        at io.ap4k.deps.jackson.databind.introspect.BasicBeanDescription._properties(BasicBeanDescription.java:142)
        at io.ap4k.deps.jackson.databind.introspect.BasicBeanDescription.findProperties(BasicBeanDescription.java:217)
        at io.ap4k.deps.jackson.databind.deser.BasicDeserializerFactory._findCreatorsFromProperties(BasicDeserializerFactory.java:330)
        at io.ap4k.deps.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:312)
        at io.ap4k.deps.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:252)
        at io.ap4k.deps.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:221)
        at io.ap4k.deps.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143)
        at io.ap4k.deps.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:406)
        at io.ap4k.deps.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:352)
        at io.ap4k.deps.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
        at io.ap4k.deps.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at io.ap4k.deps.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at io.ap4k.deps.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
        at io.ap4k.deps.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:3908)
        at io.ap4k.deps.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3803)
        at io.ap4k.deps.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2874)
        at io.ap4k.deps.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:235)
        at io.ap4k.deps.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:190)
        at io.ap4k.deps.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:383)
        at io.ap4k.deps.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:360)
        at io.ap4k.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromInputStream(BuildConfigOperationsImpl.java:274)
        at io.ap4k.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromFile(BuildConfigOperationsImpl.java:231)
        at io.ap4k.deps.openshift.client.dsl.internal.BuildConfigOperationsImpl.fromFile(BuildConfigOperationsImpl.java:68)
        at io.ap4k.openshift.hook.JavaBuildHook.run(JavaBuildHook.java:70)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: io.ap4k.deps.openshift.api.model.BuildSpec
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 37 more

The maven build no longer performs an s2i build by default. It has to be explicitly specified.

authors: cmoulliard iocanel


text:
Remove property tserver.compaction.major.delay

The tserver no longer does compactions and those were removed in #3604, need to remove this property.

In general, are the properties (and metrics) that are being removed, are there equivalents in the new code?  We should look to provide a mapping from previous -> new so that users that may need these setting know how to map them to the new equivalent.
If it is a straight removal - then that should be noted too.
Maybe we start a section in the release notes that we update as we go along?  It may be easier than trying for identify or remember all of the changes at the end?

Closed by #3950

authors: EdColeman jmark99 keith-turner

repo: apache/accumulo


text:
Create Config page on Monitor

Displays non sensitive properties on the Monitor. Link is under REST

What is the use case for this?

Can you supply a screenshot of the page?

Here is a screenshot of part of the page @Manno15. I don't think its meant to be human readable. I think its meant to serve as an endpoint where one can fetch the config in XML form.

Yeah it will show up here: http://localhost:9995/rest/config
It could be both human or machine readable. I just picked XML because my browser displays it more neatly than JSON.

For example:
<conf> <property>compaction.coordinator.compaction.finalizer.check.interval=60s</property> <property>compaction.coordinator.compaction.finalizer.threads.maximum=5</property> <property>compaction.coordinator.compactor.dead.check.interval=5m</property> <property>compaction.coordinator.message.size.max=10M</property> <property>compaction.coordinator.port.client=9132</property> <property>compaction.coordinator.port.search=false</property> <property>compaction.coordinator.threadcheck.time=1s</property> <property>compaction.coordinator.threads.minimum=1</property> <property>compaction.coordinator.threads.timeout=0s</property> <property>compaction.coordinator.tserver.check.interval=1m</property>

It also makes for easy copy and pasting.

When I visit the page, my browser doesn't display it so nicely (view my screenshot from earlier as an example). I only see the xml when I inspect or wget/curl the page. We could potentially add two pages, one used to fetch the data (JSON or XML) and another used to view it by a user which could be in the form of a table or something more readable than JSON or XML.

When I visit the page, my browser doesn't display it so nicely (view my screenshot from earlier as an example). I only see the xml when I inspect or wget/curl the page. We could potentially add two pages, one used to fetch the data (JSON or XML) and another used to view it by a user which could be in the form of a table or something more readable than JSON or XML.

You can use jq to format the output: curl <url> | jq .; stuff like this doesn't generally need to be human-readable.

I'm not convinced this is a good idea. At best, it will show the configuration of the monitor itself. It won't show the configuration of tservers, which may have different properties configured, or of the manager or gc. It also intentionally doesn't show sensitive properties.
So, it only shows a partial view of one server's configuration, and misrepresents it as the monolithic configuration for the entire cluster.
That is not only misleading, but it also goes against some of the previous work done to migrate to commons-configuration2 to support more modular configuration management at the service level. There isn't a "one global configuration" (there never has been, but now users have even more options with commons-configuration2 to manage their configuration in a modular way per server instance).

I pulled the code from ConfigCommand so we can easily do anything we are doing there. And we could display config by different servers and have the server name as a node in the output.

I pulled the code from ConfigCommand so we can easily do anything we are doing there. And we could display config by different servers and have the server name as a node in the output.

The ConfigCommand has some of the same issues (as does the public API it calls). But, at the very least, you can't view any configuration in the shell unless you've done some kind of authentication. That's not true of this change to the monitor. This just provides it without any kind of authentication.
To partially address the misleading information problem, you could display the config for each server, but that would be a bit of information overload. Or you could just display the one, but with "as reported by serverX". However, either way, you'd have to either modify the public API, or bypass it to get the server information and talk directly via Thrift. I don't think that extra code is worth it when the benefits are marginal. And, ultimately, I don't think it would better than an authorized sysadmin securely grabbing a copy of the config files from each server's class path and dumping the ZK config to get the authoritative information, that they can already do today.
Overall, though, I don't think the monitor's scope should grow to be a tool for configuration management, even if it is just used to report configuration. I'm also deeply uncomfortable exposing all this through the monitor without authentication. I think users have much more reliable and secure means for viewing the configuration they've deployed to their own servers than anything we'd add insecurely to the monitor.

The ConfigCommand has some of the same issues (as does the public API it calls).

What kind of issues?

authors: DomGarguilo ctubbsii milleruntime


text:
Make use of GitHub Actions environment files

set-output was previously used to create the IT matrix, but it is now deprecated. So, this change makes use of the environment files instead.
See https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#environment-files

I'm troubleshooting an error that seems to be a data type issue:
Error when evaluating 'strategy' for job 'mvn'. .github/workflows/maven-full-its.yaml (Line: 88, Col: 15): Error parsing fromJson,.github/workflows/maven-full-its.yaml (Line: 88, Col: 15): Error reading JToken from JsonReader. Path '', line 0, position 0.,.github/workflows/maven-full-its.yaml (Line: 88, Col: 15): Unexpected type of value '', expected type: Mapping.

It looks like GitHub is parsing environment variables as String types, and I can't seem to convert it to a "Mapping" type.

Ignore my previous observations. I forgot to update the job output mapping to reflect the new step output name. I'm testing now, but I think the current bits will work.

Okay, got it working. Apparently, you're not supposed to quote the output value, but leave the raw JSON... which is weird, because it makes an invalid environment variable assignment in a shell script... but apparently that's what it wants.


text:
adds validation of classloader context

This adds validation of the classloader context without adding a new SPI method and without changing the existing classloading behavior in a bug fix release.

These changes look okay - but I am lost were these fit into relation to the other, similar changes (https://github.com/apache/accumulo/pull/3682 in particular)

@EdColeman  I added a comment on #3682

I created two commits (a revert and then the changes) and pushed those to the 2.1 branch manually (can not do a fast forward merge in GH).


text:
Added property to configure TCP backlog for Thrift server sockets

Closes #3751

Kicked off full IT build

Thrift was originally passing 0 for the backlog. Since ServerSocket uses 50 if backlog value is less than 1, this change shouldn't modify existing behavior.
https://github.com/openjdk/jdk/blob/a9679acd5c85b88afe827dc55dbe9a78b00c8918/src/java.base/share/classes/java/net/ServerSocket.java#L272-L273
Changes look good to me!

authors: ddanielr dlmarion


text:
Artemis-2563 Add option to create queue on all active clustered nodes

This is to help with redistribution as mentioned in this Jira:
https://issues.apache.org/jira/browse/ARTEMIS-2563
Address option added to enable queue creation on all active nodes in a cluster, which happens on BINDING_ADDED. Since this creates a local binding for remote queues, a redistributor will also get created once a consumer registers

I will close this for now, got some issues rebasing and formatting. I will open a new one later

repo: apache/activemq-artemis

license_source: bigquery


text:
ARTEMIS-4658 Prevent reflections when using dual address federation

When federation is configured in two directions between nodes for an address the message can reflect from one node to another if max hops is not set or not set correctly and in some federation topologies the max hops value can't solve the issue and still result in a working configuration. This reflection should be prevented at the federation consumer level for address consumers.

I've done a few full runs of the tests to check for regressions


text:
Support outgoing AMQP connections

@mtaylor This is the initial version. The first commit is something that I was surprised that was not there, so let me know if it's bad. Without it, I have to call connectionCreated explicitly.
The second commit fixes an issue I saw when testing, where it looked like onFlow was called on a session that had already been closed.
The third commit is the actual implementation of outgoing connections. The code was originally a part of the AMQP connector service that is left out of this commit, but I thought it was a better fit inside the AMQP implementation. In the integration test you can see that its a bit awkward in its use, so I think it would benefit from a refactoring at some point.

@mtaylor @lulf , shouldn't we squash them all into a single commit?

@lulf the first two commits are definitely a keeper... the third one I'm not so sure.
I had in the past tried to implement core over AMQP ,which is something I gave up given the qpid-proton-jms being much more mature. I don't really want to invest all the time @tabish121  already invested (think about a good 1 year and half to 2 years) to make his client be matured enough to support production scenarios.
Just a FYI, I had recently removed a lot of code that was supporting client AMQP over the Proton client to simplify things around the server. so I'm really thinking this is not the right place to go.
if you want and have the stamina (think of a multi year commitment) to support core over AMQP (which is the ClientProtocolManager) then we could consider it as an option, otherwise I am really thinking this is not the right way to go.
@lulf  / @mtaylor what is the use case here? maybe I'm mistaken and missing something here.

@clebertsuconic In EnMasse(messaging-as-a-service), we use artemis for addresses that require store-and-forward semantics. Today, in order to have the broker connected to the router network, we co-locate a dispatch router together with the artemis instance. This limits our scalability and adds an extra hop for messages. We want to have the broker connect to the router network instead using a connector service, so that we can get rid of the co-located router.
This PR contains the changes I had to make in artemis for the connector service to work. It currently lacks the capability to establish sessions and outgoing links, which we might need later on. That being said, the only changes strictly required is the open() method on the ProtonHandler and AMQPConnectionContext. I can leave the  ProtonClientConnectionLifeCycleListener and ProtonClientProtocolManager inside the connector service instead if that is desired.

@lulf ok, I would merge it with this commit also:
https://github.com/clebertsuconic/activemq-artemis/commit/efc6e2dccf8ba8ae3e3909ba26ff8c4122bb8d0e
Did you consider using the Client API from qpid-proton-jms? or open a JMS connection from qpid-proton?

@clebertsuconic Added, thanks.
I actually started out trying both the current 'integrated' approach and using vertx-proton. I decided to go down the current path based after initial feedback from @andytaylor and @mtaylor on the 'integrated' approach. Using the artemis implementation gives me the connection management and integration with queues/topics, which I think would require more work if going through a client library path.
That being said, more work is probably required if/when we need to establish outgoing amqp links.

@lulf  ok.. will merge it then

authors: clebertsuconic lulf


text:
[AIRAVATA-2662] Adding Migrator class to create everyone group in all domains

One-time migration to create and populate the “everyone” group for all existing domains and share all the existing application deployments with the group.

@tilaks26 , can you also change the sharing registry so that whenever a new user is created it is added to the everyone group?  The migration script will add all existing users to the everyone group but we need to also add any new users that get created after the migration script runs to the everyone group.

@machristie Should that be a part of the createUser() method in the SharingRegistryServerHandler or do you want me to make a new method for adding new users to the everyone group?

@machristie , I've made the requested changes.

Changes to the AiravataDataMigrator.java class:

Removed the EveryoneGroupMigrator.java class and included the changes in AiravataDataMigrator.java class
Modified the pom.xml to include the security services stubs as well
Retrieving groupOwner and ApplicationDeploymentOwner from the KeyCloakSecurityManager.java class
Added a new PermissionType - EXEC for Application Deployments

Changes to the AiravataServerHandler.java class:

Added a new PermissionType - EXEC for methods - initSharingRegistry(), addGateway(), shareResourceWithUsers() and revokeSharingOfResourceFromUsers()
Modified getAccessibleApplicationDeployments() to first retrieve all the Group Resource Profiles accessible by the user in the gateway and extracting all the Compute Resource IDs from the list of Group Resource Profiles which is then passed to the getAccessibleApplicationDeployments() of the RegsitryServerHandler.java class

Changes to the registry-api.thrift, RegistryServerHandler.java class and ApplicationDeployment.java interface:

Modified getAccessibleApplicationDeployments() to accept the accessible Compute Resource IDs
Regenerated the Thrift files for Registry

Changes to the AppDeploymentResource.java class:

Modified getAll() to also filter by the accessible Compute Resource IDs

This Pull Request contains the fixes and suggestions for [AIRAVATA-2690] ticket as well.

@tilaks26 this looks good, I've reviewed the changes.  Two other things:
Looks like there is a build error, the error message is
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project airavata-registry-core: Compilation failure: Compilation failure:
[ERROR] /home/travis/build/apache/airavata/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/app/catalog/impl/ApplicationDeploymentImpl.java:[38,8] org.apache.airavata.registry.core.app.catalog.impl.ApplicationDeploymentImpl is not abstract and does not override abstract method getAccessibleApplicationDeployements(java.lang.String,java.util.List<java.lang.String>,java.util.List<java.lang.String>) in org.apache.airavata.registry.cpi.ApplicationDeployment
[ERROR] /home/travis/build/apache/airavata/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/app/catalog/impl/ApplicationDeploymentImpl.java:[398,5] method does not override or implement a method from a supertype
[ERROR] /home/travis/build/apache/airavata/modules/registry/registry-core/src/main/java/org/apache/airavata/registry/core/app/catalog/impl/ApplicationDeploymentImpl.java:[404,21] cannot find symbol
[ERROR]   symbol:   method setAccessibleAppDeploymentIds(java.util.List<java.lang.String>)
[ERROR]   location: variable resource of type org.apache.airavata.registry.core.app.catalog.resources.AppDeploymentResource
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :airavata-registry-core

Also, can you update getAccessibleAppModules in the AiravataServerHandler, similar to what you did with getAccessibleApplicationDeployments?

repo: apache/airavata


text:
AMBARI-24885 Ambari Files view not working when User doesnt have access to any group and just have access to Filesview

AMBARI-24885 Ambari Files view not working when User doesnt have access to any group and just have access to Filesviewss to any group and just have access to Filesview
What changes were proposed in this pull request?
Fix of AMBARI-24885
(Please fill in changes proposed in this fix)
How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)
Please review Ambari Contributing Guide before opening a pull request.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/4514/
Test PASSed.

authors: Akhilsnaik asfgit

repo: apache/ambari


text:
AMBARI-25526. Typo in the "NameNode" turn off maintenance mode dialog…

… box (dpidhaiets)
What changes were proposed in this pull request?
Typo in the "NameNode" turn off maintenance mode dialog box
How was this patch tested?
manually
Please review Ambari Contributing Guide before opening a pull request.

retest this please

retest this please

retest this please

authors: hapylestat pidhaietsdp


text:
feat: add support for Endpointslices

Please answer these questions before submitting a pull request

Why submit this pull request?

[ ] Bugfix

[X] New feature provided

[ ] Improve performance

[ ] Backport patches

Related issues

#201

@tokers @membphis PTAL, thanks!

@lianghao208 PR is good to me, but we need some e2e cases to cover it.

/retest

@lianghao208 Any updates?

@tokers
I haven't figured out a proper way to add e2e test case yet . To enable and disable endpointSlice feature, it needs to run on different versions of minikube, which needs to modify the github workflow, any suggestions?

@tokers
I haven't figured out a proper way to add e2e test case yet . To enable and disable endpointSlice feature, it needs to run on different versions of minikube, which needs to modify the github workflow, any suggestions?

Just add another workflow with different minikube version, own test cases about endpointSlices will be run under this minikube, while in the normal environment, cases about endpointSlices should be skipped, you can check out the ginkgo help documents about how to skip or focus on specific test cases.

Codecov Report

Merging #252 (5a55ab8) into master (fe68c79) will increase coverage by 48.34%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##           master      #252       +/-   ##
============================================
+ Coverage   51.65%   100.00%   +48.34%
============================================
  Files          34         1       -33
  Lines        2358         1     -2357
============================================
- Hits         1218         1     -1217
+ Misses        982         0      -982
+ Partials      158         0      -158

Impacted Files
Coverage Δ

pkg/apisix/service.go

pkg/api/server.go

pkg/apisix/apisix.go

pkg/version/version.go

pkg/api/router/router.go

pkg/apisix/nonexistentclient.go

pkg/seven/state/service_worker.go

pkg/ingress/apisix/plugin.go

pkg/id/idgen.go

pkg/ingress/apisix/annotation.go

... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe68c79...5a55ab8. Read the comment docs.

Need to due with some Conflicting files. @lianghao208

Ping @lianghao208

Close stale PR, feel free to reopen.

authors: codecov-io gxthrj lianghao208 tokers

repo: apache/apisix-ingress-controller


text:
refactor: endcta

Fixes: #[Add issue number here]
Changes:

Screenshots of the change:
new:

old:

@juzhiyuan @bzp2010 Hi, Let's merge this.

repo: apache/apisix-website


text:
Add ability to cache DataFrame in memory

In stack overflow question https://stackoverflow.com/questions/73015779/how-to-persist-datafusion-dataframe-in-memory there is a question about how to persist a DataFrame in memory. It would be nice to have an easy way to do this from the Python API.
DataFusion has support for MemTable which could be used to achieve this.

When this PR(https://github.com/apache/arrow-datafusion/pull/3512) is merged, I will submit a PR to support this feature.

authors: andygrove francis-du

repo: apache/arrow-datafusion-python


text:
feat: add case function (#447)

Which issue does this PR close?
Closes #447
What changes are included in this PR?
Besides exposing the case function, this PR also adds the PyCaseBuilder class (to parallel Rust's CaseBuilder)
Are there any user-facing changes?
Yes, but the documentation should be generated automatically

Looks like there is one failing CI check

Yes, @mesejo there is a check-style test failing if you don't mind fixing. If not I can make the commit to resolve

Thanks to both of you @alamb and @jdye64 for the review 🙏. The check-style test should be fixed now!

Thanks again @mesejo

authors: alamb jdye64 mesejo


text:
CI should run Ballista integration tests

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We are seeing PRs merged that cause regressions in Ballista because we currently rely on manual integration testing.
Describe the solution you'd like
CI should run the integration tests:
./dev/integration-tests.sh

Describe alternatives you've considered
None
Additional context
None

The lack of ballista coverage in CI came up again with @gaojun2048 @liukun4515  and @mingmwang
on https://github.com/apache/arrow-datafusion/pull/2070#discussion_r834104400

I can take this task in the Q2

Thank you @liukun4515 ! In case you missed it, the setup I had in https://github.com/apache/arrow-datafusion/pull/688 already works and was only paused pending optimizations to reduce cache size. I imagine you might be able to reuse some of the setup there when you start working on this this task.

Should we consider running the integration tests in CI without creating docker images? Just run the native binaries for scheduler and executor?

authors: alamb andygrove houqp liukun4515

repo: apache/arrow-datafusion


text:
Replace python based integration test with sqllogictest

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We have a great  integration test from @Jimexist 🦾  https://github.com/apache/arrow-datafusion/tree/master/integration-tests which runs  a limited number of queries against data in both postgres and datafusion and compares the results
The major downside is that it does not get updated with new coverage very often. I believe this is due to two factors:

It is not run with cargo test
It requires additional system setup above and beyond for developing DataFusion (postgres, python, datafusion-python, etc)

Describe the solution you'd like
I would like to port all the existing coverage in integration-test to the sqllogictest framework aded in https://github.com/apache/arrow-datafusion/pull/4395 and remove the python based version
In order to run the same tests against postgres, we could reuse some of the code from sqllogictest-bin to implement a postgres driver: https://github.com/risinglightdb/sqllogictest-rs/blob/main/sqllogictest-bin/src/engines/postgres.rs
I think we would likely have to add some sort of annotation to the tests like
# compare_to(postgres)

And then not run tests annotated like that by default
Describe alternatives you've considered
Keep the existing
Additional context
Add any other context or screenshots about the feature request here.

I think this would be a nice project for someone to get to know DataFusion better -- I think it would involve learning about sqllogictest-rs and porting the existing code from there.

Hi, @alamb.I want to work on this if it's not an urgent task. I'm new to the arrow-datafusion and wonder whether I can work it out, But I'd like to try on this and learn about datafusion.

Hi there! I am new to Datafusion and would like to learn more about it.
And I found this issue, which I'd like to tackle.
I did some investigation regarding sqllogictest, sqllogictest-rs and datafusion. And I now see some ways to implement a solution.
My main question is: what level of compatibility between Postgres and Datafusion would you like to check?
Currently, the tests in integration-test compare values within an error using np.testing.assert_allclose.
But while values are close, they are not always fully equal in terms of representation.
For example, the test simple_window_partition_aggregation.sql result has an avg value which Postgres represents as -13.8571428571428571, but Datafusion does as -13.857142857142858. The values are close, but not in a text form representation.

Regarding implementation I imagine some ways:
1. The very basic way without changes to sqllogictest-rs would be to define sqllogic tests with results explicitly. Each test would be created twice: once for Postgres, and once for Datafusion. Tests would differ
For example,
onlyif postgres
query R
select 2.0;
----
2.0

onlyif datafusion
query R
select 2.0;
----
2

Pros:

each output could be different. For example, as in the example above the values are the same, though their representation is different.
this doesn't change the idea of sqllogictest, and uses out-of-the-box features

Cons:

all outputs have to be created manually.
in many cases there would be just duplicate queries and duplicate sets of results.

2. Another option that uses existing features is:

create a test.slt test without outputs
run Postgress to complete the results. Do it the same way the --complete parameter works now. Output this file into a test.tmp.slt
Run Datafusion over test.tmp.slt

Pros:

no new sqllogictest-rs features required

Cons:

there only way to compare results is to compare their text representations

3. Use query labels to compare of labeled queries, but run multiple database engines at the same time.
The queries would look like
onlyif postgres
query R nosort q42
select 2.0;

onlyif datafusion
query R nosort q42
select 2.0;

Pros:

results are generated by sqllogictest. There is no need to define them manually
there is no new sqllogictest syntax features
the tool would be pretty generic. It could be reused for different database combinations

Cons:

sqllogictest-rs currently doesn't support comparison of labeled queries. This has to be implemented.
this is not really an intended use of onlyif. The sqllogictest in general and its implementations suppose that there is one database at a time. The same suite of tests can run on many different databases, but not at the same time. In my suggesting, we suppose that there are 2 databases. This fast is not necessarily bad but stretches the current idea of sqllogictest a little
there is no way to define comparison rules, like in option 2

4. Create an sqllogictest::Runner with a custom Validator.
The Validator is currently defined as
pub type Validator = fn(actual: &[Vec<String>], expected: &[String]) -> bool;

If Validator is updated to contain the original query record, then implementation can actully run Postgres, and disregard whatever expected value is passed to the validator.
pub type Validator = fn(record: Record, actual: &[Vec<String>], expected: &[String]) -> bool;

...
fn postgres_validator(record: Record, actual: &[Vec<String>], expected: &[String]) {
// let postgres_result = call_postgres(record.sql);
// postgres_result == actual
}

Pros:

relatively small number of changes to sqllogictest-rs
provides a more elaborate Validator that could be used for other purposes

Cons:

comparing float values with error would still require more changes. sqllogictest-rs treats results in actual/expected field as String now. So there is no way to do number rounding. Another enum, for example, DBOutput has to be introduced instead of String to carry type information. See this issue for a discussion of non-text comparisons.
value Validator would likely need to be merged with a type validator. Otherwise, there could be a need to call Postgress twice: in value validation, and in type validation

5. Introduce a way to run a statement/query on multiple databases at the same time.
This can be either achieved by using labels
query R nosort postgres datafusion
select 2.0;

or the way that is similar to what was proposed in the description of this
# compare(datafusion, postgres)
query R
select 2.0;

Pros:

this is a new generic feature that would allow having multiple databases at the same time

Cons:

sqllogictest-rs has to be updated to support having multiple runners
# compare(datafusion, postgres) would require updating not only execution, but also parsing logic

To summarize my understanding of sqllogictest:

tests check text representations of results
if comparing of same, but not exactly equal value is required, then they have to be transformed to the same text representation. For example, comparing floating numbers could require using sql functions like round or truncate.

In order to demonstrate how Postgres compatibility could look like using sqllogictest I created an implementation based on the 2nd approach https://github.com/apache/arrow-datafusion/pull/4834.
I would be glad to hear some feedback regarding my pr and thoughts about which way to proceed.
Thank you in advance!

Hi @melgenek

Hi there! I am new to Datafusion and would like to learn more about it.

That is great! Welcome!

My main question is: what level of compatibility between Postgres and Datafusion would you like to check?

I think for this ticket we should strive for "the same level as is currently verified using the python integration tests" as much as possible.

The values are close, but not in a text form representation.

yes  this is a classic "floating point rounding error" type situation and why it is typically not a great idea to directly compare floating point values.
In terms of how to implement this, would something like this work (initially):

For some specific files (maybe those that start with postgres_vaildated_ the datafusion sqllogic-rs test runner would run the file both against postgres and against datafusion and compare the results. This file might initially only contain the existing queries in the python based integration test but we could expand it over time.

For both datafusion and postgres, apply some sort of normalization to floating point values prior to printing (round them to a smaller number of significant figures https://github.com/apache/arrow-datafusion/blob/3cc607de4ce6e9e1fd537091e471858c62f58653/datafusion/core/tests/sqllogictests/src/normalize.rs#L78-L95, for example). Maybe we could add that to the compare directive

I also left some feedback on https://github.com/apache/arrow-datafusion/pull/4834#pullrequestreview-1239686856 -- does that make sense?

I also left some feedback on https://github.com/apache/arrow-datafusion/pull/4834#pullrequestreview-1239686856 -- does that make sense?

Thank you for the feedback.

For both datafusion and postgres, apply some sort of normalization to floating point values prior to printing (round them to a smaller number of significant figures

I think this is a good suggestion. I'll update the Postgres/Datafusion clients to have a normalization step for results.

In order to proceed further, I'd like to clarify the way how you'd like to define .slt tests.
In https://github.com/apache/arrow-datafusion/pull/4834 I implemented the tests similar to the existing integragration-tests: there are only queries and results are generated and compared by the test runner.
Regular .slt are supposed to have expected outputs. And in this comment https://github.com/apache/arrow-datafusion/pull/4834#discussion_r1064024225 you're suggesting having the same approach for Postgres compatibility tests.
Having explicit results is a totally viable approach, especially having --complete option for running sqllogictests.
So do we agree, that the way to go is to force all .slt tests to have expected outputs, and then simply verify the same tests with Postgres/Datafusion runnens?

So do we agree, that the way to go is to force all .slt tests to have expected outputs, and then simply verify the same tests with Postgres/Datafusion runnens?

Yes, that would be my preferred solution -- it will ensure that DataFusion is producing consistent answers with Postgres and that the answers can also be verified by looking at the slt file
Just so it is clear, I do not expect every single  .slt file now (or ever) to produce the same answers as postgres (as datafusion supports different syntax and likely also has existing discrepancies, etc). I think we should start simple with a few explicitly marked files used to do "with postgres verification" and we can expand the scope of which files are tested against slt files over time.

authors: Aitozi alamb melgenek


text:
Issue 2393: Support glob patterns for (local) file(s)

Which issue does this PR close?
Closes #2393.
What changes are included in this PR?
When the provided prefix is not a valid path (file/directory), we will now attempt to "glob" over the prefix.
Eg: when the path is
/testdata/blah.txt -> return /testdata/blah.txt (as before)
/testdata -> return all files that exist in /testdata (as before)
/testdata/x_z -> return all files that match the given pattern (assuming that there is no file/directory '/testdata/x_z')

Resolved failing clippy, lint and cargo fmt errors

I think the code now mixes blocking IO, as performed by glob, and non-blocking IO as performed by tokio::fs. I think we should consistently use one or the other...

Had a look at the implementation of tokio fs operations, and copied their (non-public) asyncify method to run the globbing on...

Trying to find the location where to put tests to verify that globbing works:
ctx.read_csv("globbedpath", CsvReadOptions::new()) -> should return data as expected
ctx.register_csv("aggregate_test_100", "globbedpath", CsvReadOptions::new()) -> should return data as expected (when queried)

Afaik this should be good to go now.

Tests pass now on my windows machine as well :)

Fixed clippy issues + documented the checks that are executed during a PR build (+ added script to execute them).

Perhaps we could split out the formatting and CI changes into a separate PR, as I think they're someone parallel to the changes in this PR?

Thanks again :+1:

authors: timvw tustvold


text:
Minor: Encapsulate LeftJoinData into a struct (rather than anonymous enum) and add comments

Which issue does this PR close?
Related to https://github.com/apache/arrow-datafusion/issues/8130
Rationale for this change
I am going through this code to understand what is happening, and one thing I think would help would be to use names to refer to fields rather than data.0, data.1, and that could be commented
What changes are included in this PR?

Make fields of JoinData non pub
Add doc comments
Make LeftJoinData a struct with named fields and accessors

Are these changes tested?
Existing tests
Are there any user-facing changes?
No, this is entirely internal code refactoring

@Dandandan  or @metesynnada  I wonder if you might have time to review this (I am in the process of working on https://github.com/apache/arrow-datafusion/issues/8130 and am trying to encode my findings in comments)

Thank you @alamb

authors: Dandandan alamb


text:
refactor: Apply minor refactorings to functions-array crate

Which issue does this PR close?
Closes #9787.
What changes are included in this PR?
This PR aims to do following refactoring:
1- Being added make_scalar_function support to missed invoke functions,
2- Aligning rust-doc across array functions,
3- Removing redundant imports,
4- Fixing typo problems,
5- Rename core.rs -> make_array.rs.
Are these changes tested?
Yes, all array.slt tests pass.
Are there any user-facing changes?
No

Thanks @alamb and @jayzhan211 for the reviews and feedbacks. I have also some questions on https://github.com/apache/arrow-datafusion/issues/9787 when you have time to have a look.


text:
[JDBC] Timezone improvements for the JDBC driver

Describe the enhancement requested
Currently, our Timezone handling is pretty iffy as identified in a few different threads. I've been doing some digging and would like to propose a few changes. I will make a draft PR with how I think we should implement these changes so we can open up discussions on each point.
Support java.time objects
The JDBC 4.2 spec specifies users can fetch java.time objects using resultSet.getObject(..., <java.time>.class). We should support OffsetDateTime, ZonedDateTime, Instant, and LocalDateTime for timestamps. We should also support LocalDate for dates vectors, and LocalTime for times.
Support TIMESTAMP_WITH_TIMEZONE JDBC type
Timestamp vectors that include a timezone should be treated as TIMESTAMP_WITH_TIMEZONE

When retrieved as OffsetDateTime or ZonedDateTime, use the TZ included in the vector
When retrieved as Timestamp with a Calendar, adjust the value to display time in the TZ specified by the Calendar, assuming the value in the vector is in the vector specified TZ
When retrieved as LocalDateTime, do same as Timestamp but assume the Calendar specifies UTC.
Still not sure how Instant plays into all of this

Timestamp vectors without a timezone should be treated as TIMESTAMP

When retrieved as OffsetDateTime or ZonedDateTime, throw an error?
When retrieved as Timestamp or LocalDateTime, take the value as-is and don't do any TZ conversion. The assumption should be that if the vector didn't include TZ, we're just dealing with "wall-clock" time so it should never be adjusted
Still not sure how Instant plays into all of this

If we get a new vector type that includes TZ/Offset per record (https://github.com/apache/arrow/issues/44248), it should also be treated as TIMESTAMP_WITH_TIMEZONE

This is still a hypothetical, but I think if this comes to fruition we should treat it similar to Timestamp vectors with TZ, but use the TZ per record instead of for the whole vector.

Don't do any TZ adjustments for regular TIMESTAMP w/o TZ

We currently assume UTC for vectors w/o a TZ and still do an offset conversion when requested with a given Calendar. We should not do any TZ conversion for these since they should be treated as "wall-clock" time.

Parameter binding
We probably also need to deal with relevant changes when binding parameters, but I'll leave that as a separate change since we also need to deal with #153.
Did I miss anything? Let me know!

@jduo would certainly love your feedback since I know you have lots of experience in this space

repo: apache/arrow-java


text:
parquet-rs: Support for Time Microseconds

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

As the author of the upstream odbc2parquet I would like for parquet to support writing Time columns with MicroSeconds precision, so I can offer more precise types to my users. This is especially helpful if odbc2parquet is used as part of a data pipeline in which the next steps relies on legacy converted types, since these do not have pendant for NanoSeconds precision.
Currently if I try to write a column of that type parquet emits a panic.
not implemented: Conversion for physical type INT64, converted type TIME_MICROS, value 334000000

Describe the solution you'd like

If I am not mistaken the interface should just be analogous to writing Time columns with NanoSeconds precision.
Describe alternatives you've considered

Lacking support in the parquet crate, I am unlikely to offer that improvement.
Additional context

Here is the original issue that triggered support for Time in odbc2parquet: https://github.com/pacman82/odbc2parquet/issues/285
Sadly so far I can only offer it with Nanoseconds precision.

The record API does not appear to support time types at all - https://docs.rs/parquet/latest/parquet/record/enum.Field.html
As a workaround I would suggest using either the arrow APIs or lower-level writer APIs as they support this and are more actively maintained. Otherwise I would also be happy to review a PR adding support for this, but am unlikely to have time to work on this myself.

Hello @tustvold ,

I would suggest using either the arrow APIs or lower-level writer API

I do not use the record API, but am using the lower-level writer API, as returned by:
Int64Type::get_column_writer_mut(column_writer).unwrap();
// ...
cw.write_batch(values, Some(def_levels), None)?;

Do you have a reproducer, the following works correctly
#[test]
    fn test_time() {
        let schema = Arc::new(
            types::Type::group_type_builder("schema")
                .with_fields(&mut vec![Arc::new(
                    types::Type::primitive_type_builder("col1", Type::INT64)
                        .with_repetition(Repetition::REQUIRED)
                        .with_converted_type(ConvertedType::TIME_MICROS)
                        .build()
                        .unwrap(),
                )])
                .build()
                .unwrap(),
        );

        let mut out = Vec::with_capacity(1024);
        let props = Arc::new(WriterProperties::builder().build());
        let mut writer = SerializedFileWriter::new(&mut out, schema, props).unwrap();

        let mut row_group = writer.next_row_group().unwrap();
        let mut column = row_group.next_column().unwrap().unwrap();
        column
            .typed::<Int64Type>()
            .write_batch(&[1, 2, 3], None, None)
            .unwrap();
        column.close().unwrap();
        row_group.close().unwrap();

        writer.close().unwrap();
    }

Hello @tustvold ,
thanks for your response. I apologize, for stating previously that I did not use the record API. While the production code of odbc2parquet does not it uses parquet-read in its integration tests which throws the error. I am sorry for the time you had to spend chasing this down.
Best, Markus

No worries at all, glad we got to the bottom of it :+1:

Thank you a lot and a merry Christmas!

repo: apache/arrow-rs


text:
Correct example code for column (#5560)

Which issue does this PR close?

Closes #5560 .
Rationale for this change
The example is incorrect as it is. This changes the assertion to reflect the actual behavior.

What changes are included in this PR?

Only documentation changes.
Are there any user-facing changes?

Thank you

authors: tustvold zgershkoff


text:
[MATLAB] Implement the isequal() method on arrow.array.Array

Describe the enhancement requested
Currently, it's not possible to determine if two arrow.array.Array instances are equal because the isequal() method always returns false by default:
Example
>> a = arrow.array([1 2 3])

a =

[
  1,
  2,
  3
]

% Compare a with itself.
>> tf = isequal(a, a)

tf =

  logical

   0

Component(s)
MATLAB

take

repo: apache/arrow


text:
[Java] Implement TransferPair functionality for BinaryView

Describe the enhancement requested
At the moment in the StringView implementation, transferPair functionality is not included. This must be implemented with corresponding test cases.
Component(s)
Java

Issue resolved by pull request 41980
https://github.com/apache/arrow/pull/41980

authors: lidavidm vibhatha


text:
ARROW-15700: [C++] Compilation error on Ubuntu 18.04

Modified Substrait consumer interaction with libprotobuf to support versions down to 3.0.0, which is the minimum required due to Substrait's usage of proto3 syntax.
Tested locally with:
export ARROW_PROTOBUF_URL=https://github.com/protocolbuffers/protobuf/releases/download/v3.0.0/protobuf-cpp-3.0.0.tar.gz
cmake \
  --preset ninja-debug \
  -DProtobuf_SOURCE=BUNDLED \
  -DARROW_PROTOBUF_BUILD_VERSION=v3.0.0 \
  -DARROW_PROTOBUF_BUILD_SHA256_CHECKSUM=318e8f375fb4e5333975a40e0d1215e855b4a8c581d692eb0eb7df70db1a8d4e

(Is there an easier way to do this without modifying versions.txt or 751fb9d7? Also, the env var is needed only because Google isn't at all consistent with their release file naming that far back.)
It'd also be nice to add this to CI, but it's probably excessive to always run for a PR, unless combined with some other run.

Presumably 2144b75 will solve the above, but I wasn't able to reproduce the problem locally.

@github-actions crossbow submit -g cpp


text:
ARROW-382: Extend Python API documentation

Fix numpydoc compilation
Add simple examples to the API
Move away from deprecated Cython-property declaration
Add basic descriptions with return types to functions

+1

authors: wesm xhochy


text:
GH-15287: [Ruby] Merge column and add suffix in Table#join

Rationale for this change
Current implementation is always preserve column of join key. It is convenient if columns are merged.
What changes are included in this PR?

Columns from left and right are marged if;

Join key is a String or a Symbol (<= incompatible Change)
Join key is nil (natural join) (<= change in unreleased feature)

New options left_suffix="" and right_suffix="" are introduced.

If it is empty (by default), join key(s) do not change.
If it is not empty, the suffix is appended to join key(s).

Are these changes tested?
Yes.
Are there any user-facing changes?
There are incompatible change when join key is a String or a Symbol.

Closes: #15287

Comment for reviewers

Renaming feature by suffix is introduced only for the keys=[key1, key2] case. I think keys={hash} case does not require this feature.
I added a test for right_anti. I think there may be a bug when it creates empty ChunkedArray data in the Table.

table1 = Arrow::Table.new(key: [1])
table = table1.join(table1, "key", type: :right_anti)
#=>
# returns a Table with empty column. This is the expected result.
#<Arrow::Table:0x10c6bf730 ptr=0x7fb62708f530>
	key

# But its content is strange ChunkedArray;
table[:key].data

#=>
#<Arrow::ChunkedArray:0x10c6946c0 ptr=0x7fb6276aead0 [

]>

# Empty ChunkedArray is this;
Arrow::ChunkedArray.new([[]])
#<Arrow::ChunkedArray:0x1202e4610 ptr=0x7fb627a59240 [
  []
]>

# Or if it contains null;
Arrow::ChunkedArray.new([[nil]])
#<Arrow::ChunkedArray:0x1203060d0 ptr=0x7fb627a58a40 [
  [
    null
  ]
]>

I'm not sure this is a bug.

Rebased after #33677, #33675

Thanks!


text:
WIP: [Release] Verify release-15.0.2-rc3

PR to verify Release Candidate

@github-actions crossbow submit  --group verify-rc-source --param release=15.0.2 --param rc=3

Revision: e03105efc38edca4ca429bf967a17b4d0fbebe40
Submitted crossbow builds: ursacomputing/crossbow @ release-15.0.2-rc3-0

Task
Status

almalinux-8-amd64

almalinux-8-arm64

almalinux-9-amd64

almalinux-9-arm64

amazon-linux-2023-amd64

amazon-linux-2023-arm64

centos-7-amd64

centos-8-stream-amd64

centos-8-stream-arm64

centos-9-stream-amd64

centos-9-stream-arm64

conan-maximum

conan-minimum

debian-bookworm-amd64

debian-bookworm-arm64

debian-bullseye-amd64

debian-bullseye-arm64

debian-trixie-amd64

debian-trixie-arm64

java-jars

nuget

python-sdist

r-binary-packages

test-ubuntu-22.04-docs

ubuntu-focal-amd64

ubuntu-focal-arm64

ubuntu-jammy-amd64

ubuntu-jammy-arm64

ubuntu-mantic-amd64

ubuntu-mantic-arm64

wheel-macos-big-sur-cp310-arm64

wheel-macos-big-sur-cp311-arm64

wheel-macos-big-sur-cp312-arm64

wheel-macos-big-sur-cp38-arm64

wheel-macos-big-sur-cp39-arm64

wheel-macos-catalina-cp310-amd64

wheel-macos-catalina-cp311-amd64

wheel-macos-catalina-cp312-amd64

wheel-macos-catalina-cp38-amd64

wheel-macos-catalina-cp39-amd64

wheel-manylinux-2-28-cp310-amd64

wheel-manylinux-2-28-cp310-arm64

wheel-manylinux-2-28-cp311-amd64

wheel-manylinux-2-28-cp311-arm64

wheel-manylinux-2-28-cp312-amd64

wheel-manylinux-2-28-cp312-arm64

wheel-manylinux-2-28-cp38-amd64

wheel-manylinux-2-28-cp38-arm64

wheel-manylinux-2-28-cp39-amd64

wheel-manylinux-2-28-cp39-arm64

wheel-manylinux-2014-cp310-amd64

wheel-manylinux-2014-cp310-arm64

wheel-manylinux-2014-cp311-amd64

wheel-manylinux-2014-cp311-arm64

wheel-manylinux-2014-cp312-amd64

wheel-manylinux-2014-cp312-arm64

wheel-manylinux-2014-cp38-amd64

wheel-manylinux-2014-cp38-arm64

wheel-manylinux-2014-cp39-amd64

wheel-manylinux-2014-cp39-arm64

wheel-windows-cp310-amd64

wheel-windows-cp311-amd64

wheel-windows-cp312-amd64

wheel-windows-cp38-amd64

wheel-windows-cp39-amd64

@github-actions crossbow submit  --group verify-rc-binaries --group verify-rc-wheels --param release=15.0.2 --param rc=3

For verify-rc-binaries-wheels-macos-11-arm64 see: https://github.com/apache/arrow/issues/40546

@github-actions crossbow submit verify-rc-binaries-jars-* --group verify-rc-wheels --param release=15.0.2 --param rc=3

@github-actions crossbow submit verify-rc-binaries-jars-* --param release=15.0.2 --param rc=3

@ursabot please benchmark

authors: kou raulcd


text:
ARROW-5937: [Release] Stop parallel binary upload

It's fragile.

+1


text:
ARROW-9338: [Rust]  Add clippy instructions

Add instructions on how to run clippy locally.

I tested the instructions and they work (obviously) for me on a mac.

I'm unable to merge, something wrong with my local setup. @andygrove may you please merge if you see this message

authors: alamb nevi-me


text:
ARROW-10109: [Rust] Add support to the C data interface for primitive types and utf8

This PR is a proposal to add support to the C data interface by implementing the necessary functionality to both consume and produce structs with its ABI and lifetime rules.
This is for now limited to primitive types and strings (utf8), but it is easily generalized for all types whose data is encapsulated in ArrayData (things with buffers and child data). For types where this does not happen (such as StructArray, DictionaryArray and ListArray, where data also resides on their specific struct implementations instead of ArrayData), I believe that more work is needed. IMO we should strive to have all data in ArrayData, as it makes it significantly easier to export it via the C Data Interface, as well as understanding what they physically contain.
Some design choices:

import and export does not care about the type of the data that is in memory (previously BufferData, now Bytes) - it only cares about how they should be converted from and to ArrayData to the C data interface.
import wraps incoming pointers on a struct behind an Arc, so that we thread-safely refcount them and can share them between buffers, arrays, etc.
export places Buffers in private_data for bookkeeping and release them when the consumer releases it via release.

I do not expect this PR to be easy to review, as it is touching sensitive (aka unsafe) code. However, based on the tests I did so far, I am sufficiently happy to PR it. I tried to comment as much as possible, which I will continue to do so in the following commits.
This PR has three main parts:

Addition of an ffi module that contains the import and export functionality
Add some helpers to import and export an Array from C Data Interface
A crate to test this against Python/C++'s API

It also does a small refactor of BufferData, renaming it to Bytes (motivated by the popular bytes crate), and moving it to a separate file.
What is tested:

round-trip Python -> Rust -> Python (new separate crate, arrow-c-integration)
round-trip Rust -> Python -> Rust  (new separate crate, arrow-c-integration)
round-trip Rust -> Rust -> Rust

What is not tested yet:

memory allocation counts (I am still trying to find a way of doing this in Rust)

Things to do:

[ ] CI for the Python tests: it requires different compilation flags, which requires compiling the whole thing. I excluded it from the workspace as it does not behave well with rust-analyzer, but we need to add it to the CI nevertheless.
[ ] Add more comments
[ ] Add more tests
[ ] Extend for all types that only use buffers
[ ] Error on types that are not supported

Finally, this PR has a large contribution of @pitrou , that took a lot of his time to explain to me how the C++ was doing it and the main things that I had to worry about here.

Thanks @jorgecarleitao this is really interesting. I've had a first pass through to get familiar with this and I will try building locally sometime this weekend hopefully.

memory allocation counts (I am still trying to find a way of doing this in Rust)

Just FTR, in C++ we have a global function that returns the number of currently Arrow-allocated bytes. This helps us write crude resource allocation tests (here through the Python wrapper pyarrow.get_total_allocated_bytes):
https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_cffi.py#L116-L131
Another possibility would be to use the callback facility on your Bytes object to check that the destructor was called at some point.

Some changes since last time:

added a small tool to verify memory allocations and verified no leaks
added the test that @pitrou suggested to verify that we call the c++'s release.

The tool counts all memory allocations and deallocations, like in C++. It is used as a test at the end of all tests, as a final validation that the test suite does not leak.
I've placed it behind a feature gate as it current only works in single-threaded programs. Suggestions are welcomed to improve it further.
I think that this is now ready to review.

@nevi-me , @andygrove @pitrou @alamb , I have rebased this PR.
I need your guidance here:

is this something that we still want to pursue, or should we close this? If open:
Currently the memory track is done with a feature gate. This is faster, but requires a new compilation to run the tests with that feature gate.
Currently it tests memory leaks via a test at the end of all tests (and under the feature gate). This covers all tests implicitely, but tests that panic are intrinsically leaky and thus there is a non-trivial interaction between tests and the memory check test.
The integration test with Python/C++ requires another compilation, with other compiler flags, which is an extra CI burden.

Some ideas:

Refactor the memory test check to be on a per-test basis, so that we do not have interactions between tests. The positive is that we won't get tests interactions. The downside is that we need to be explicit in performing the memory check on each test we want.
Make the allocation/deallocation outside the feature gate. This may make the code slower (I need to profile) as we need to handle an thread-atomic lock of a counter, but it significantly reduces the complexity around CI (no need to re-compile with a different feature gate).
Make the Python/C++ tests use C++ libs and headers instead of running against pyarrow. This is something that I do not know how to do, so it will take me some time.

I made the following changes:

Made the allocation counter a AtomicIsize and removed it from the feature gate
Added CI
Renamed the crate interacting with Python to arrow-pyarrow-integration-testing

The CI will likely fail, as I will need to fiddle with it. There will be some spam. Sorry guys!

Codecov Report

Merging #8401 (e0b1fe4) into master (be13bf5) will decrease coverage by 0.11%.
The diff coverage is 70.53%.

@@            Coverage Diff             @@
##           master    #8401      +/-   ##
==========================================
- Coverage   84.43%   84.32%   -0.12%
==========================================
  Files         178      189      +11
  Lines       44144    45186    +1042
==========================================
+ Hits        37275    38104     +829
- Misses       6869     7082     +213

Impacted Files
Coverage Δ

rust/arrow/src/array/array.rs
83.49% <0.00%> (-7.04%)
:arrow_down:

rust/arrow/src/error.rs
5.00% <0.00%> (-0.27%)
:arrow_down:

rust/arrow/src/bytes.rs
54.05% <54.05%> (ø)

rust/arrow/src/ffi.rs
70.61% <70.61%> (ø)

rust/arrow/src/array/ffi.rs
88.63% <88.63%> (ø)

rust/arrow/src/buffer.rs
98.21% <100.00%> (+2.56%)
:arrow_up:

rust/arrow/src/memory.rs
100.00% <100.00%> (ø)

rust/datafusion/src/logical_plan/builder.rs
85.09% <0.00%> (-2.48%)
:arrow_down:

rust/datafusion/src/logical_plan/plan.rs
81.56% <0.00%> (-1.72%)
:arrow_down:

rust/datafusion/src/physical_plan/planner.rs
77.45% <0.00%> (-0.62%)
:arrow_down:

... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be13bf5...9569214. Read the comment docs.

@kszucs , I am trying to fix the Dev / Source Release and Merge Sctript, but I have been unable: I do not understand what I need to do besides what I already did.
From what I understood, this script is used to replace versions throughout the project. I added a new section for the new crate, but I can't find understand why is the test failing.

@alamb @pitrou , @paddyhoran @andygrove , the integration with CI is in place, the tests pass, and all comments from @alamb and @pitrou were addressed. If anyone wants to take a final pass, please let me know, otherwise, for me this is ready to merge.

I merged this and marked all associated issues as resolved.
We can now FFI to and from c++ and pyarrow 🎉

Wahoo! This is a great milestone! You could announce it on the ML.

authors: andygrove codecov-io jorgecarleitao pitrou


text:
AVRO-3904: [Rust] Change the return type of SchemaCompatibility::can_read() from bool to AvroResult<()>

AVRO-3904
API break
What is the purpose of the change

Do not panic when schemata are incompatible. Return Err and let the user deal with it.

Verifying this change

Run the build and test

Documentation

Does this pull request introduce a new feature? no

The unit tests still fail! Needs debugging!

This looks really similar to https://github.com/apache/avro/compare/main...marcosschroh:avro:fix/compatibility-read-when-field-is-subset?expand=1. I push literal an hour ago, I stopped to solve an issue in my personal work and I see this... I have never seen such a thing... incredible

Next time I will make sure to create a PR a microsecond after pushing to my fork to avoid  people stealing things. This one would have been my first contribution to a project using rust as I am a newie in rust , thanks a lot @martin-g really appreciate it.

Next time I will make sure to create a PR a microsecond after pushing to my fork to avoid people stealing things. This one would have been my first contribution to a project using rust as I am a newie in rust , thanks a lot @martin-g really appreciate it.

Sorry for stealing your joy! :-)
I had free time and decided to work on this...

@marcosschroh You can still review and approve the PR! Thanks!

As I mentioned: I am a newbie in rust, so another person with more experience should look at it. Despite my lack of knowledge on rust, yesterday we agreed in the jira ticket that I had to send a PR and then you (@martin-g ) would look at it (not the other way around). Involve another developer with rust experience to get feedback.

I have no time for drama!
I will close the PR and let you open a new one!

authors: marcosschroh martin-g

repo: apache/avro


text:
Fix AVRO-2235

Fixed typo in record_with_logical_types.asvc, then regenerated TestRecordWithLogicalTypes.java

I see this was partially fixed by a recent commit.  Will redo my fix.

Ref #342

authors: kojiromike rstata


text:
[Feature Request]: Optimize [multi]map side inputs.

What would you like to happen?
Currently the only way to read a map is key-by-key lookup. While this works great for too-large-to-fit-into-memory maps, for smaller maps (which are commonly used) this is very inefficient.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

@robertwb is work completed on this task?

The attached PRs should resolve the issue for the SDK and local runners.

authors: jrmccluskey robertwb

repo: apache/beam


text:
Bump python container versions to beam-master-20210526

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

---

Java

Python
---

---

---

XLang

---

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

LGTM, m

LGTM


text:
TEST hbase stuff

cc: @iemejia

Looks like it got farther -- master sets up, but the regionserver is what fails.

Coverage decreased (-0.003%) to 69.224% when pulling 8f2be281b84d3d47e5d7db8bf7c1f8d20013e209 on dhalperi:hbase-help into 36ed3f6c54d35ac48241789fd77e2657d54c5831 on apache:master.

Closing, now that I got the HBaseIOTest to pass (https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7544/org.apache.beam$beam-sdks-java-io-hbase/testReport/org.apache.beam.sdk.io.hbase/HBaseIOTest/)

Wow, thanks a lot for finding the issue, that was great !!!
I will fix and PR to see if the full build passes now, excellent!

authors: coveralls dhalperi iemejia


text:
Remove Direct Runner "doneness" configuration

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

@jkff

Coverage decreased (-0.02%) to 70.177% when pulling b1fd8487887ca47c435a57552fabe02c28f10d1f on tgroh:remove_direct_shutdown_configuration into 118da81716ef3f8fc4dc035bd998eff821bb9b2d on apache:master.

Coverage decreased (-0.07%) to 70.128% when pulling cdcf1bafd59244480273f60189201e3fb23965f3 on tgroh:remove_direct_shutdown_configuration into 118da81716ef3f8fc4dc035bd998eff821bb9b2d on apache:master.

authors: coveralls tgroh


text:
Fix typo in join explanation.

Please add a meaningful description for your change here
Fix typo in join explanation.

Run Website_Stage_GCS PreCommit

Thanks.


text:
[BEAM-1402] Makes TextIO comply with PTransform style guide

Migration guide: TextIO.Read.from() -> TextIO.read().from; TextIO.Write.to() -> TextIO.write().to()
Also converts both to AutoValue and moves some utility classes to upper level to declutter TextIO.java.
Each individual commit should be trivial to review.
R: @reuvenlax

Please, please no churn on FileBased-Write transforms while I'm still hacking on FileBasedSink.

Hmm ,well, that's unfair. This is a lot of work, too.

This PR doesn't appear to touch the FileBasedSink base class.
On Apr 28, 2017 9:37 PM, "Daniel Halperin" notifications@github.com wrote:

Hmm ,well, that's unfair. This is a lot of work, too.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/beam/pull/2774#issuecomment-298139029, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AUGE1aed1dSRcxJT5UFH_gu26I5eLGN5ks5r0pRDgaJpZM4NMGeb
.

Changes Unknown when pulling 03b5df3f3094a68b9100a30a1e1908c2c17772a6 on jkff:text-style into ** on apache:master**.

retest this please

Coverage increased (+0.007%) to 69.822% when pulling 99547e9b2dfabfa68c281909f5f9fce70ca15eef on jkff:text-style into f5e3f5230af35da5a03ba9740f087b0f22df6dca on apache:master.

retest this please

retest this please

Coverage increased (+0.02%) to 69.836% when pulling 674336bd4121dd1178c4152886623a9091c8763c on jkff:text-style into 14d60b26e7e9b8a578038599341e66ccd99d012b on apache:master.

Coverage increased (+0.02%) to 69.832% when pulling 674336bd4121dd1178c4152886623a9091c8763c on jkff:text-style into 14d60b26e7e9b8a578038599341e66ccd99d012b on apache:master.

Here too, the test failures are flakes - it passes mvn verify -Prelease locally.

retest this please

Coverage increased (+0.02%) to 69.832% when pulling 674336bd4121dd1178c4152886623a9091c8763c on jkff:text-style into 14d60b26e7e9b8a578038599341e66ccd99d012b on apache:master.

retest this please

Coverage increased (+0.01%) to 69.824% when pulling 674336bd4121dd1178c4152886623a9091c8763c on jkff:text-style into 14d60b26e7e9b8a578038599341e66ccd99d012b on apache:master.

Since Reuven is traveling
R: @jbonofre
(though if you approve, please don't merge: I'll merge myself, upon coordinating with @dhalperi)

Coverage increased (+0.03%) to 70.412% when pulling 541611fcb872ccab53a35f31f3403f96c463f6f9 on jkff:text-style into c2a5557839c6146e5a3f386172d4d068c67ac656 on apache:master.

retest this please

R: @jbonofre

Coverage increased (+0.03%) to 69.868% when pulling 60861867157bf5808f02d6d605fea5d82483e51d on jkff:text-style into eec903fb538eb9c3d41d737dba94a41f7c1c469e on apache:master.

authors: coveralls dhalperi jbonofre jkff reuvenlax


text:
Always proxy RunInference with large models

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Run Python 3.9 PostCommit

Run Python 3.9 PostCommit

Run Python 3.9 PostCommit

Run Python 3.9 PostCommit

Run Python 3.9 PostCommit

Run Python 3.9 PostCommit

Run Python 3.9 PostCommit

Run Python_Runners PreCommit

Run Python_PVR_Flink PreCommit

Run Python_Coverage PreCommit

Run Python 3.9 PostCommit

Run Python 3.9 PostCommit

R: @riteshghorse


text:
Fix verifyJavaVersion wrong filter

It is found that after #28918, the java version verification test only checks that the source code was compile with Java8, but not test code compiled/running on certain Java version
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Run SQL_Java8 PreCommit

Run SQL_Java17 PreCommit

PreCommit Java Example Java21 now verifies java version:
https://github.com/apache/beam/actions/runs/10393658121?pr=32183
:sdks:java:testing:test-utils:verifyJavaVersion21
PASSED	2.146s	1.953s	2.146s
org.apache.beam.sdk.testutils.jvmverification.JvmVerification
PASSED	0.179s	0.121s	0.179s
verifyCodeIsCompiledWithJava8
PASSED	0.045s
verifyRunningJVMVersionIs21
PASSED	0.011s
verifyTestCodeIsCompiledWithJava21
PASSED	0.002s

assign set of reviewers

R: @tvalentyn @damccorm

The latest commit should fix https://github.com/apache/beam/actions/workflows/beam_PreCommit_Xlang_Generated_Transforms.yml currently failing on master

Run Xlang_Generated_Transforms PreCommit


text:
[BEAM-3339] Mobile gaming automation for Java nightly snapshot on core runners

DESCRIPTION HERE
In this change, we added test automation of mobile-gaming (UserScore, HourlyTeamScore, LeaderBoard) on Direct and Dataflow running against nightly snapshot.

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand:

[ ] What the pull request does
[ ] Why it does it
[ ] How it does it
[ ] Why this approach

[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Seed Job

Run Dataflow PostRelease

Run Python ReleaseCandidate

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

+R: @alanmyrvold

+R: @lukecwik

@lukecwik This is the code change of java mobile-gaming automation on core runners (direct, dataflow). Please review it when you get a moment. Thanks~

+R: @tgroh

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Changes are made base on comments. @tgroh

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

@lukecwik Changes are made base on comments. Please review. Thanks.

rename task names and related methods.
re-formatting code follows google styles.

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Changes are made:

add comments in Injector that indicate the team list should sync to the list in MobileGamingJavaUtils.
Use map to create arguments for each examples.
Rewrite command generation method in MobileGamingJavaUtils.
Add "-q" option for each quickstart command that stop writing [Info] messages into console during maven compile. This make the console output cleaner.
PTAL @tgroh

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

run java precommit

run java precommit

Run Seed Job

Run Dataflow PostRelease

Run Seed Job

Run Dataflow PostRelease

run java precommit

Jenkins: retest this please.

Jenkins: retest this please.

@tgroh  Please merge if you are happy with this :)   Thanks!

authors: robertwb tgroh yifanzou


text:
fix: reconciliation loops behaviors

With this PR we aim to improve the reconciliation loop of Integrations and KameletBindings. The Integration now fails and report the error that may happen during the execution of traits. We also change the reconciliation loop for KameletBindings in order to detect any failure (failing traits, or wrong spec), report an error and stop the reconciliation loop.
Closes #3389
Closes #3010

Release Note
fix: reconciliation loops behaviors

Still working on more E2E test to have a proper coverage. But I'd like to collect early feedback in the while.

repo: apache/camel-k


text:
Avoid overwriting the Qute json which breaks Camel K catalogs

This should solve apache/camel-k-runtime#677
Without this, the catalog seems to generate its own bogus qute.json. Then, during camel-k-runtime build, it gets picked instead of the one available in camel-quarkus-qute-component jar.

Looking at the CI failure - maybe we need to split the prepare-catalog-quarkus and check-extension-pages goals into separate executions with their own configuration. Otherwise the extension docs will get removed for qute.

Looking at the CI failure - maybe we need to split the prepare-catalog-quarkus and check-extension-pages goals into separate executions with their own configuration. Otherwise the extension docs will get removed for qute.

Thanks.
Do you have something like this in mind?
<execution>
                        <!-- prepare the catalog and update doc files, etc. -->
                        <goals>
                            <goal>prepare-catalog-quarkus</goal>
                        </goals>
                        <phase>process-resources</phase>
                        <configuration>
                            <extensionDirectories>
                                <extensionDirectory>${maven.multiModuleProjectDirectory}/extensions-core</extensionDirectory>
                                <extensionDirectory>${maven.multiModuleProjectDirectory}/extensions</extensionDirectory>
                                <extensionDirectory>${maven.multiModuleProjectDirectory}/extensions-jvm</extensionDirectory>
                            </extensionDirectories>
                            <skipArtifactIdBases>
                                <skipArtifactIdBase>http-common</skipArtifactIdBase>
                                <skipArtifactIdBase>qute</skipArtifactIdBase>
                            </skipArtifactIdBases>
                        </configuration>
                    </execution>
                    <execution>
                        <!-- prepare the catalog and update doc files, etc. -->
                        <goals>
                            <goal>check-extension-pages</goal>
                        </goals>
                        <phase>process-resources</phase>
                        <configuration>
                            <extensionDirectories>
                                <extensionDirectory>${maven.multiModuleProjectDirectory}/extensions-core</extensionDirectory>
                                <extensionDirectory>${maven.multiModuleProjectDirectory}/extensions</extensionDirectory>
                                <extensionDirectory>${maven.multiModuleProjectDirectory}/extensions-jvm</extensionDirectory>
                            </extensionDirectories>
                            <skipArtifactIdBases>
                                <skipArtifactIdBase>http-common</skipArtifactIdBase>
                            </skipArtifactIdBases>
                        </configuration>
                    </execution>

@ppalaga @jamesnetherton I've test the updated PR, based on your comments and it worked for the Camel K Runtime. It is, indeed, caused by the bogus qube.json file on the catalog jar.
That said, I understand it may not be desirable from Camel Quarkus perspective. So I'd like to hear your insights: do you feel this is an appropriate approach or should I pursue a different strategy for dealing with this?

@ppalaga how do you want to handle this? This would remove remove the broken qute.json from the catalog. Is there a simple means of getting a correct JSON file included?

Agreed to fix differently: https://github.com/apache/camel-quarkus/issues/2701

authors: jamesnetherton orpiske ppalaga

repo: apache/camel-quarkus