text:
plot is not the same between APDL and Pymapdl_reader

🤓 Before submitting the issue

[X] I have visited the Troubleshooting section.
[X] I have searched among the existing issues.
[X] I am using a Python virtual environment.
[X] I am using the latest version of PyMAPDL and its dependencies (fully updated virtual environment). You can update them using pip install --upgrade --upgrade-strategy eager ansys-mapdl-core in your activated virtual environment.

🔍 Description of the bug
Hello everybody,
When i plot my rst file with pyansys, it is not the same result as APDL plotting

Thanks in advance for your help
🕵 Steps To Reproduce
from ansys.mapdl import reader as pymapdl_reader

model = "C:/test_ansys/MODEL.rst"
result = pymapdl_reader.read_binary(model)
print(result)
result.plot_nodal_solution(0, 'norm', label='Displacement')

💻 Which Operating System are you using?
Windows
🐍 Which Python version are you using?
3.10
📝 PyMAPDL Report
Show the Report!

# PASTE HERE THE OUTPUT OF `python -c "from ansys.mapdl import core as pymapdl; print(pymapdl.Report())"` here

📝 Installed packages
Show the installed packages!

# PASTE HERE THE OUTPUT OF `python -m pip freeze` here

I have attached the .rst file here:
MODEL.txt
Just change the extension ".txt" to ".rst" ( github doesn't support .rst file)
📝 Logger output file
Show the logger output file.

# PASTE HERE THE CONTENT OF THE LOGGER OUTPUT FILE.

@mikerife updated


text:
picker选择器，用在FormItem中时，设置必选无法检测

组件
Picker
基础库版本
2.7.12
设备机型
开发者工具
组件库版本
最新版
预期行为

复现步骤

选择picker对应的值
点击提交出现如下情况

复现 demo
axml代码
<view class="buy-content">
  <form form="{{form}}" initialValues="{{initialValues}}" onFinish="handleSubmit">
    <form-item label="产品类型" name="productType" required form="{{form}}">
      <picker onOk="pickerConfirm" placeholder="请选择" value="{{productType1}}"
        title="请选择" data="{{productTypeList}}">
      </picker>
      <!-- <view class="hidden-class">
        <input-item mode="form" value="{{productType}}" placeholder="请输入金额" maxLength="11" type="digit" hidden />
      </view> -->
    </form-item>
    <button slot="footer" type="primary" mode="form" form="{{form}}" htmlType="submit">提交
    </button>
  </form>
</view>

选中picker后，点击提交会提示“请输入必填项”

有没有人呀，这个是bug还是代码错误呀

尝试下在picker中加mode="form" 呢，详细请见https://mini.ant.design/components/form

尝试下在picker中加mode="form" 呢，详细请见https://mini.ant.design/components/form

我尝试了，不行。输入框可以，但是picker不行，而且我看到文档 picker也没有 mode 属性

https://github.com/ant-design/ant-design-mini/issues/305

source: gharchive/issue

repo: ant-design/ant-design-mini

license: MIT

license_type: permissive

license_source: github-api


text:
🧐[问题]导航菜单项无children时icon与label紧贴

🧐 问题描述
使用pro-layout的自定义菜单形式，渲染中通过Link跳转菜单指定路由，代码如下
menuItemRender = (menuItemProps, defaultDom) => {
if (menuItemProps.isUrl || menuItemProps.children) {
return defaultDom
}
return {defaultDom}
}
这段代码在使用pro-layout V4版本时没有问题。升级至V5版本后出现菜单在没有children的情况下，icon与label之间紧贴，margin被覆盖的情况。

如果删除Link组件，则渲染正确，但无法跳转路由。
menuItemRender = (menuItemProps, defaultDom) => {
return defaultDom
}

当前使用pro-layout版本5.0.6，请问该问题如何解决

Any solutions?

antd@4.0.3 将会解决
https://github.com/ant-design/ant-design/pull/22217

authors: chenshuai2144 guodong johnnylc

repo: ant-design/ant-design-pro-layout


text:
When I use the @ vueuse/nuxt terminal, tell me a [warn]

Describe the bug
[warn] [nuxt] You are using a plugin that has not been wrapped in `defineNuxtPlugin`.
 It is advised to wrap your plugins as in the future this may enable enhancements: () => { }

when i install yarn add @vueuse/nuxt @vueuse/core.in nuxt.config.ts:
1. don't  modules: ['@vueuse/nuxt']. don't have warn.

2. modules: ['@vueuse/nuxt']`. have a warn.

Reproduction
https://github.com/RSS1102/for-nuxt3-vueuse
System Info
System:
    OS: Windows 10 10.0.22000
    CPU: (16) x64 AMD Ryzen 7 5800H with Radeon Graphics
    Memory: 3.01 GB / 13.86 GB
  Binaries:
    Node: 18.1.0 - D:\nvm\node\node.EXE
    Yarn: 1.22.19 - D:\nvm\node\yarn.CMD
    npm: 8.8.0 - D:\nvm\node\npm.CMD
  Browsers:
    Edge: Spartan (44.22000.120.0), Chromium (104.0.1293.70)
    Internet Explorer: 11.0.22000.120

Used Package Manager
yarn
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

"devDependencies": {
"@vueuse/core": "^9.1.1",
"@vueuse/nuxt": "^9.1.1",
"nuxt": "3.0.0-rc.8"
},

my warn!
maybe shoundn't
https://vueuse.org/guide/#installation
   vueuse: {
        ssrHandlers: true,
    },


text:
Timed out after 30000 ms while trying to connect to the browser, Only Chrome at revision r856583 is guaranteed to work

Describe the bug
Timed out after 30000 ms while trying to connect to the browser!
Only Chrome at revision r856583 is guaranteed to work.

Reproduction
Run Browse Lite: Open... command to start the browser
System Info
System:
    OS: macOS 13.0
    CPU: (8) arm64 Apple M1
    Memory: 116.45 MB / 16.00 GB
    Shell: 3.2.57 - /bin/bash

Binaries:
    Node: 20.5.1 - /opt/homebrew/bin/node
    Yarn: 1.22.19 - /opt/homebrew/bin/yarn
    npm: 9.8.0 - /opt/homebrew/bin/npm

Browsers:
    Chrome: 116.0.5845.110
    Safari: 16.1

Used Package Manager
npm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

This is my personal usage issue, after I restart compute it works well.

repo: antfu/vscode-browse-lite


text:
Update to create 0.5.1.e

Hello,
Create has launched a new version, and it breaks a lot of add-ons, including this one.
Could you fix it ?
Here the logs
[21/09/2023 23:22:04 PM] Caused by: java.lang.NoSuchFieldError: MAIN_TAB
[21/09/2023 23:22:04 PM] 	at TRANSFORMER/balancedflight@1.20.1_v2.1.0/com.vice.balancedflight.AllBlocks.<clinit>(AllBlocks.java:11)
[21/09/2023 23:22:04 PM] 	at TRANSFORMER/balancedflight@1.20.1_v2.1.0/com.vice.balancedflight.BalancedFlight.<init>(BalancedFlight.java:54)
[21/09/2023 23:22:04 PM] 	... 14 more
[21/09/2023 23:22:04 PM] [modloading-worker-0/ERROR] Failed to create mod instance. ModID: balancedflight, class com.vice.balancedflight.BalancedFlight
[21/09/2023 23:22:04 PM] 	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[21/09/2023 23:22:04 PM] 	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
[21/09/2023 23:22:04 PM] 	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[21/09/2023 23:22:04 PM] 	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
[21/09/2023 23:22:04 PM] 	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
[21/09/2023 23:22:04 PM] 	at LAYER PLUGIN/javafmllanguage@1.20.1-47.1.47/net.minecraftforge.fml.javafmlmod.FMLModContainer.constructMod(FMLModContainer.java:68)
[21/09/2023 23:22:04 PM] 	at LAYER PLUGIN/fmlcore@1.20.1-47.1.47/net.minecraftforge.fml.ModContainer.lambda$buildTransitionHandler$10(ModContainer.java:123)
[21/09/2023 23:22:04 PM] 	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
[21/09/2023 23:22:04 PM] 	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1796)
[21/09/2023 23:22:04 PM] 	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
[21/09/2023 23:22:04 PM] 	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
[21/09/2023 23:22:04 PM] 	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
[21/09/2023 23:22:04 PM] 	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
[21/09/2023 23:22:04 PM] 	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

Neeed this update :o

yes an update would be much appreciated

@nthxny
https://github.com/anthxnymc/BalancedFlight/pull/24

authors: Fundryi Siryak billjk2013

repo: anthxnymc/BalancedFlight


text:
[Fix] Fixed getting index for amplitude array.

It was crash due to going beyond the array bounds if there were less than 7 members in self.amplitudeArray

Awesome! thanks @AndrewPetrov  for the fix! :)

Thank you! I'll send you one more soon)
On Mon, Mar 21, 2016 at 3:04 PM, Bryan Antigua notifications@github.com
wrote:

Merged #39 https://github.com/antiguab/BAFluidView/pull/39.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/antiguab/BAFluidView/pull/39#event-597076741

sweet!! :+1:

source: gharchive/pull-request

authors: AndrewPetrov antiguab

repo: antiguab/BAFluidView

license: mit

license_source: bigquery


text:
World Of Warcraft

What are you experiencing an issue with?

[ ] Aurora
[ ] Aurora supported game, (Fill in the game)
[ ] Aurora Wrapper
[ ] Aurora Profile
[ ] Aurora supported device, (Fill in problematic devices)
[ ] Aurora Scripts
[ ] Aurora Updater
[ ] Aurora Lighting issue
[x] Other, (Fill in another issue here)

What kind of an issue is it?

[ ] Crash
[ ] Performance problems
[ ] Visual bug
[ ] Localization issue
[ ] Enhancement suggestion
[ ] Device issue
[x] Other, (Fill in another type of issue here)

Expected behavior:
Explain here what should happen.
Actual behavior:
Explain here what happened instead.
How often does this problem occur?

This problem occurs... (Fill in here)
Please describe the steps to reproduce this issue:
The issue can be reproduced by doing following steps:

What is the version of Aurora you are experiencing this issue with?

My version is... (Fill in here)
Does the previous version of Aurora have this issue?
The previous version, (Fill in the version here), did/did not have this issue.
Please include any logs, screenshots, or files that are related to this issue:

For logs, you can use: http://pastebin.com/
For screenshots, you can use: http://imgur.com/
For files, you can use: http://www.tinyupload.com/ , https://www.dropbox.com/ , or attach them to the issue.

It would be a good idea to make a project about World Of Warcraft.

Duplicate of #183 , also take a look at an already existing RGB project for WOW that should work with Aurora. https://github.com/antonpup/Aurora/issues/64#issuecomment-232565417

repo: antonpup/Aurora


text:
Automated cherry pick of #3670: Fix broken Markdown links check for main branch

Cherry pick of #3670 on release-1.5.
#3670: Fix broken Markdown links check for main branch
For details on the cherry pick process, see the cherry pick requests page.

/skip-all

repo: antrea-io/antrea


text:
我使用的是4.1.34版本，当我用npm audit 我发现了高危漏洞关于 d3-color

我又试了  4.2.0 和 4.2.8 版本，都有问题。
这是检查报告
High            d3-color vulnerable to ReDoS

  Package         d3-color

  Patched in      >=3.1.0

  Dependency of   @antv/g2

  Path            @antv/g2 > @antv/g-base > d3-interpolate > d3-color

  More info       https://github.com/advisories/GHSA-36jr-mh4h-2g58

  High            d3-color vulnerable to ReDoS

  Package         d3-color

  Patched in      >=3.1.0

  Dependency of   @antv/g2

  Path            @antv/g2 > @antv/component > @antv/g-base > d3-interpolate >
                  d3-color

  More info       https://github.com/advisories/GHSA-36jr-mh4h-2g58

@xiaoiver G 层升级一下吧！

已升级 v3 的 d3-interpolate

@antv/g-base => 0.5.12
@antv/g-canvas => 0.5.13
@antv/g-math => 0.1.9
@antv/g-svg => 0.5.7

已升级 v3 的 d3-interpolate，G 4.0 相关依赖已更新：

@antv/g-base => 0.5.12
@antv/g-canvas => 0.5.13
@antv/g-math => 0.1.9
@antv/g-svg => 0.5.7

需要 G2 4.0 发布新版解决。

应该用户直接update 就可以直接生效吧，G2 没有写死依赖的 G 版本。

嗯是的，NPM 方式使用的话现在重新安装依赖即可。

authors: adseng hustcc xiaoiver


text:
feat(pie): optimize spider

[x] code
[x] test
[ ] example

Before
After

export function mockPieSpider(): G2Spec {
  return {
    type: 'interval',
    data: [
      { type: '微博', value: 93.33 },
      { type: '其他', value: 6.67 },
      { type: '论坛', value: 4.77 },
      { type: '网站', value: 1.44 },
      { type: '微信', value: 1.12 },
      { type: '客户端', value: 1.05 },
      { type: '新闻', value: 0.81 },
      { type: '视频', value: 0.39 },
      { type: '博客', value: 0.37 },
      { type: '报刊', value: 0.17 },
    ],
    encode: {
      y: 'value',
      color: 'type',
    },
    labels: [
      {
        position: 'spider',
        text: (obj) => `${obj.type} (${obj.value})`,
      },
    ],
    transform: [{ type: 'stackY' }],
    coordinate: { type: 'theta' },
  };
}

Pull Request Test Coverage Report for Build 9906138480
Details

32 of 32 (100.0%)  changed or added relevant lines in 3 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.02%) to 86.564%

Files with Coverage Reduction
New Missed Lines
%

src/shape/text/advance.ts
5
90.0%

Totals

Change from base Build 9902373963:
-0.02%

Covered Lines:
10467

Relevant Lines:
11712

💛  - Coveralls

authors: coveralls pearmini


text:
path 没有二次贝塞尔，只有三次

Describe the feature you'd like to request
虽然三次够了，但是svg有的功能，库里应该要有
Describe the solution you'd like
'S' should be included in Path
Describe alternatives you've considered
'S' should be included in Path
Your Example Website or App
No response
Screenshots or Videos
No response

二次的贝塞尔是不是可以直接将两个控制点合并成一个（也就是两个控制点坐标重合）就行了？

是可以，但是源码功能上要有这个属性，不是技巧上

authors: komagic lloydzhou


text:
使用react自定义节点后，节点无法手动连线，节点只能拖拽，改为非自定义的节点就正常。

Describe the bug
使用react自定义节点后，节点无法手动连线，节点只能拖拽，改为非自定义的节点就正常。
Your Example Website or App
https://codesandbox.io/s/upbeat-wood-tvnvub?file=/src/App.js
Steps to Reproduce the Bug or Issue
鼠标连线两个节点，没法生成连线
Expected behavior
期望使用react的节点可以手动连线。
Screenshots or Videos
No response
Platform

OS: [e.g. macOS, Windows, Linux]
Browser: [e.g. Chrome, Safari, Firefox]
Version: [e.g. 91.1]

Additional context
No response

我尝试把这个节点内部样式（宽和高）改动了一下，就可以在节点边缘部分拉出连接线

x6这边连线是在.x6-node [magnet='true']的元素上监听事件进行处理的。
而你的demo里面，react定义的在foreignObject内部的元素，把这个节点内部的rect这个[magnet='true']的元素给盖住了。导致不能响应拖拽的事件。
你可以尝试将自己定义的react节点内部的元素控制一下大小，不要盖住rect这个元素。就正常了。

我尝试把这个节点内部样式（宽和高）改动了一下，就可以在节点边缘部分拉出连接线

x6这边连线是在.x6-node [magnet='true']的元素上监听事件进行处理的。 而你的demo里面，react定义的在foreignObject内部的元素，把这个节点内部的rect这个[magnet='true']的元素给盖住了。导致不能响应拖拽的事件。
你可以尝试将自己定义的react节点内部的元素控制一下大小，不要盖住rect这个元素。就正常了。

这样改了在边缘部分连线挺不友好的，官方是不是使用react节点本身就不支持手动连线。

我不是要你把样式改成我这样，我这样改动只是为了好演示！
你可以通过自己处理css样式，尽量不让react渲染出来的元素覆盖到rect，这样就不会影响连线操作了。

我不是要你把样式改成我这样，我这样改动只是为了好演示！
你可以通过自己处理css样式，尽量不让react渲染出来的元素覆盖到rect，这样就不会影响连线操作了。

好的，谢谢。

authors: lloydzhou misslinge


text:
支持多引擎切换

目前 G 支持 svg 和 canvas 两个版本，在 G2 层面要提供这两个引擎的切换方案，到时候也需要 G 的响应配合。

ref: https://www.yuque.com/antv/g2-docs/mquk09

authors: hustcc simaQ


text:
按需加载

你好，我们项目只有一个地方用到g2，但是g2的文件占比16.6%,请问如何实现部分文件加载？
下面是引入方式：
import {Stat} from 'g2'

你好，Stat 是 2.x 暴露的接口，2.x 后续不会做打的改动了，如果要使用按需加载特性，请等待 3.1 的 beta 版本

好的，幸苦你们啦！

tooltip 和 legend 能否也进一部分离呢

authors: Airkro lchunxiu leungwensen


text:
自定义 node shape 时，不显示 label

比如如下自定义 shape 的代码（参考了该示例）：
G6.registerNode('customNode', {
    draw(cfg, group) {
        const shape = group.addShape('rect', {
            attrs: {
                x: cfg.x-40,
                y: cfg.y-40,
                width: 80,
                height: 80,
                fill: '#fff',
                stroke: 'green'
            }
        });
        return shape;
    },
});

当绘制该 customNode 时，node 中并不会显示 label，但是直接添加 rect 时，可以正常显示 label。
$('#addCustom').on('click', () => {
    net.beginAdd('node', {
        label:'this is a label',
        shape: 'customNode'         /*不会显示 label */
        shape: 'rect'               /*正常显示 label */
    });
});

之所以叫自定义节点，就是给你最大的自由，什么都是你自己画呢

嗯呢，明白你的意思。但是如何画一个 label 上去呢？
对象的 label 属性是添加了，通过 console.log 也能显示值，但就是图像上渲染不出来。

authors: TomHuangCN lianchi


text:
tooltip显示

怎么使这个tooltip有x轴的时间

https://antv.alipay.com/zh-cn/f2/3.x/api/tooltip.html#_API
showTitle 自己先找下 api


text:
Alignment of the Contact us section.

Describe the feature
Hey @anuragverma108 ,
I want to correct the alignment of the Contact us section ,it is not properly visible .It is looking like this as mentioned in the below picture.

Can you please assign me this issue ?
I would like to work on this issue to showcase  my skills to improve the project .
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Hey @anuragverma108 ,
It seems that the issue has been resolved already .
Can you close the issue ?


text:
Feat: Add SignIn with google option

Describe the feature
We can add google authentication to directly sign in using google firebase.
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@anuragverma108 Please look at this issue raised


text:
enhanced the login page

issue no :- #728
This pull request focuses on enhancing the user interface of the login page by adjusting the layout of the social media login buttons. The goal is to create a more user-friendly and visually appealing experience for users attempting to log in via social media accounts.
Key Changes:
Button Alignment:
Adjusted the alignment of the social media buttons to be centrally aligned, providing a more balanced look.
Added consistent spacing between buttons to prevent a cluttered appearance.
Button Styling:
Updated the button styles to match the overall theme of the login page, ensuring that they are visually cohesive with other UI elements.
Implemented hover effects for buttons to enhance interactivity and provide feedback to users.
Responsive Design:
Ensured that the layout is responsive, making it accessible on various devices, including mobile and tablet screens.
Utilized media queries to adjust button sizes and spacing on smaller screens for optimal usability.

Attach screenshots of the changed sections After and before.  @Kedar-sonavani

@Kedar-sonavani check this PR #863 , don't you think those social media buttons without text are more attractive than yours with text ? Just wander to know.

@Kedar-sonavani check this PR #863 , don't you think those social media buttons without text are more attractive than yours with text ? Just wander to know.

Yeah but I think so my blur effect is better than that

@Kedar-sonavani I appreciate your design bro !


text:
gexf file from gephi to 3d graph?

First of all: amazing work!
here you say that the ngraph generated from a .gexf file could also easily be rendered by using e.g. ngraph.three as the renderer. I changed ngraph.pixi to ngraph.three, but it doesn't seem to work. Could you please help?

Here is some raw code doing what you're (were) looking for
module.exports.main = async function() {
    let query = require('query-string').parse(window.location.search.substring(1));
    let graph = await getGraphFromDisk(query);
    let createThree = require('ngraph.three');
    let graphics = createThree(graph, {
        interactive: true
    });
    graphics.run(); // begin animation loop:
    graphics.camera.position.z = getNumber(query.z, 2000);
};

function getGraphFromDisk(query) {
    let gexf = require('ngraph.gexf');
    let filename = query.filename || 'graph.gexf';
    return new Promise(function(resolve) {
        let req = new XMLHttpRequest();
        req.open('GET', filename);
        req.onload = function() {
            let content = this.responseText;
            let graph = gexf.load(content);
            resolve(graph);
        };
        req.send();
    });
}

function getNumber(string, defaultValue) {
    let number = parseFloat(string);
    return (typeof number === 'number') && !isNaN(number) ? number : (defaultValue || 10);
}

Starting a local web server like python3 -m http.server and browsing to http://localhost:8000

authors: sandboxdp ticapix

repo: anvaka/ngraph


text:
Add popcnt target feature

Problem
Without popcnt feature the bitwise operations will use the standard library implementation instead of the hardware supported popcnt instruction.
Instances where it will affect:
bloom/src/bloom.rs:223:        let num_bits_set = bits.iter().map(|x| x.count_ones() as u64).sum();
core/src/banking_stage/transaction_scheduler/thread_aware_account_locks.rs:394:        self.0.count_ones()
core/src/banking_stage/transaction_scheduler/thread_aware_account_locks.rs:399:        (self.num_threads() == 1).then_some(self.0.trailing_zeros() as ThreadId)

On x86 popcnt is a single instruction but without this target-feature rustc will generate something like this: (https://godbolt.org/z/69oc98chW)
; Without the patch
example::count1::h62041e16c115ac6a:
        mov     eax, edi
        shr     eax
        and     eax, 1431655765
        sub     edi, eax
        mov     eax, edi
        and     eax, 858993459
        shr     edi, 2
        and     edi, 858993459
        add     edi, eax
        mov     eax, edi
        shr     eax, 4
        add     eax, edi
        and     eax, 252645135
        imul    eax, eax, 16843009
        shr     eax, 24
        ret

; With the patch
example::count1:
 popcnt eax,edi
 ret

Summary of Changes
Add target-feature popcnt to generate single instruction for popcnt, count leading zeros etc.

I’m happy to help integrate this into the ci process, including tesing and releasing. however, I’ll defer to another domain expert for the reviewing.

The std library implementation uses the hardware instruction if it is available.
User can always specify the -C target-cpu=native or their specifici cpu architecture to give the compiler knowledge of instruction set.
I'm guess I'm not sure I understand why we'd specifically enable this feature, only for x86.
There are other instruction features I'd expect to have much more significant impact than popcnt, so why this one?
And since popcnt is not part of the x86 baseline (see https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels) I'm not sure how rustc handles if this is not present.
Does it just fail to compile? I don't think anyone is running/compiling on such old targets, so maybe that's fine?

I don't think we should enable popcnt explicitly. If we wanted to do something, we could bump to a more recent target-cpu family (that among other things will include popcnt). But I'm not sure it's worth doing, I don't think that many people are using our binaries, most people compile from source with -C target-cpu=native which will automatically select the best depending on the host machine.

Agree with this.
Additionally target-feature is considered unsafe according to rustc, so probably don't want to use that directly anyway.
From rustc -C help
    -C           target-feature=val -- target specific attributes. (`rustc --print target-features` for details). This feature is unsafe.

If this is the case, then all good. But how do we ensure they are doing it?

That's the best part, we don't need to ensure they are doing it. That's why there is a fallback

authors: apfitzge ksolana yihau

repo: anza-xyz/agave


text:
Firebase auth

The signin dynamic function seems like a good place to start but I am at a loss to where to start with firebase authentication with this wallet.
Is there anybody who implemented this solution?
Can we use firebase authentication with message signing so users are logged in to firebase also?

Wallet-adapter is not a wallet. You'd need to create a wallet that works with Firebase auth. See https://github.com/anza-xyz/wallet-adapter/blob/master/WALLET.md for information on creating a wallet that is compatible with wallet-adapter.

repo: anza-xyz/wallet-adapter


text:
Would Moloch work with an existing ES cluster that already has some indices in it?

I already have an ES cluster, holding some other indices. There's plenty of capacity, so I'm thinking of using it for moloch capture as well. Would the additional indices cause a problem, say with the viewer or the API queries?

yes it will

authors: awick thongsia

repo: aol/moloch


text:
Failed with Drone: Please provide compiled classes of your project with sonar.java.binaries property

ERROR: Error during SonarScanner execution
64 | org.sonar.java.AnalysisException: Please provide compiled classes of your project with sonar.java.binaries property
65 | at org.sonar.java.JavaClasspath.init(JavaClasspath.java:64)
66 | at org.sonar.java.AbstractJavaClasspath.getElements(AbstractJavaClasspath.java:280)
67 | at

I use the lastest plugin image, and my .drone.yml just like this:
  - name: sonar
    image: aosapps/drone-sonar-plugin
    pull: if-not-exists
    settings:
      sonar_host:
        from_secret: sonar_host
      sonar_token:
        from_secret: sonar_token
    when:
      event:
        - push

Please help.

OK, I fix this by adding a new environment parameter JAVA_BINARIES and build my own image.
Check it here: mailbyms/drone-sonar-plugin

repo: aosapps/drone-sonar-plugin


text:
🛑 Main Site is down

In 37fea88, Main Site (https://aosp.app) was down:

HTTP code: 502
Response time: 617 ms

Resolved: Main Site is back up in f665d50 after 17 minutes.

repo: aospapp/upptime


text:
Bump plugins and build-related deps

Bump plugin versions to latest
Bump build related deps and plugin deps to latest (errorprone, easymock, checkstyle, slf4j)
Ignore license-maven-plugin execution when using Eclipse M2E (broken by https://github.com/eclipse-m2e/m2e-core/issues/1332)
Update source with modernizer recommendations to use Optional.orElseThrow instead of Optional.get (this is also recommended in the Optional javadoc)
Add comment for why exec-maven-plugin can't be updated (they broke -p again)

The updated errorprone found a legitimate bug in the Precondition pattern message added in https://github.com/apache/accumulo/commit/57ed0ec503159e23d608628aa856ca8a3f164465#diff-23a22b999a2787a6c8a0be675465f31b71067f68de3ba5f633c0a73da8780b2fR328-R330 for #1389

repo: apache/accumulo


text:
ARTEMIS-3501 Added exception handling on #handleAddMessage to not sto…

…p broker from starting with currupted messages

I've been unable to write a proper test for this as I cannot simulate a journal broken in the "correct" way. This did however solve the issue I was seeing with an actual broken journal. It will discard and log the corrupted message and proceed with loading the rest of the journal

I don't think it will discard.. the message would still be in the journal until the next restart... right?

At least in my case I end up in #handleNoMessageReferences (from AbstractJournalStorageManager) and the message is removed by:
storageManager.deleteMessage(msg.getMessageID());
And the following log:
AMQ221019: Deleting unreferenced message id=$MESSAGEID from the journal

authors: AntonRoskvist clebertsuconic

repo: apache/activemq-artemis


text:
Make start script work on busybox

The current version of the activemq start script uses options to ps which are unavailable
in busybox.
Suggested change refactors the start script to remove duplicated code to find a java process
given a pidfile and makes it work on busybox.
I have limited possibilities of verifying the change on different platforms but it is confirmed to
work on at least Mac OS X in addition to busybox and should work on platforms where
previous "ps -p" command worked.

Can one of the admins verify this patch?

authors: fjollberg lucastetreault

repo: apache/activemq


text:
AMBARI-25705. Bump spring to 5.3.22 to address the CVE

What changes were proposed in this pull request?
Bump Spring to the latest versions
How was this patch tested?
Verified it was built ok. Existing UT should cover it enough.

@jojochuang  any chance test after this upgrade..?

Build succeed in trunk branch, I'm assuming there are no runtime errors, will be fully tested after we add BIGTOP stack.

Thanks @jojochuang

Build succeed in trunk branch, I'm assuming there are no runtime errors, will be fully tested after we add BIGTOP stack.

It will be good, once it's tested we can merge. when are you planning test against BIGTOP stack ? and once can you please update the test results..?

Thank you @kevinw66!!
Just to understand better, do you have some docker based script that can bring up ambari server and few agents that we can test locally on dev machine itself? Unless such script already exists somewhere (with mpacks) that I might not be aware of.
Otherwise building rpms and deploying to test cluster might be perhaps too much efforts even for small changes. Curious what your thoughts are. Thanks

Thank you @kevinw66!!
Just to understand better, do you have some docker based script that can bring up ambari server and few agents that we can test locally on dev machine itself? Unless such script already exists somewhere (with mpacks) that I might not be aware of. Otherwise building rpms and deploying to test cluster might be perhaps too much efforts even for small changes. Curious what your thoughts are. Thanks

Hi @virajjasani
For develop and test Bigtop Mpack
Please take a look at this PR: https://github.com/apache/bigtop/pull/940
And set CentOS7 repo to this: https://bigtop-snapshot.s3.amazonaws.com/centos-7/$basearch (which is using BIGTOP 3.2.0 snapshot packages, you can find discussion details in here: https://issues.apache.org/jira/browse/BIGTOP-3745)
BTW, MPack it's now using no-prefix version of components
For Ambari itself
I'll work on it sooner or later, I'm not sure if previous scripts still works, I haven't try it, you can take a look at it if you got time, and give some feedback if possible, thanks!!
https://github.com/kevinw66/ambari/blob/trunk/dev-support/docker/README.md

Thanks a lot @kevinw66, you have provided many refs, let me go through it. Having dockerized setup would greatly benefit new devs as well as those devs who are not using exact latest versions of branch-2.7 for instance and are still using HDP stack backed deployment in their test/prod clusters.
Thanks

Thanks a lot @kevinw66, you have provided many refs, let me go through it. Having dockerized setup (specifically with mpack installation) would greatly benefit new devs as well as those devs who are still using HDP stack backed deployment in their test/prod clusters.
Thanks

Thanks a lot @kevinw66, you have provided many refs, let me go through it. Having dockerized setup (specifically with mpack installation) would greatly benefit new devs as well as those devs who are still using HDP stack backed deployment in their test/prod clusters. Thanks

My pleasure, thanks for your contributions on Ambari, ping me anytime if you need help!

authors: brahmareddybattula jojochuang kevinw66 virajjasani

repo: apache/ambari


text:
AMBARI-25802: Add group functionality is not working for Mysql8

Verified using mysql-8.0.30.
Able to create group from Ambari web UI.

@kevinw66 Kindly review.

lgtm. @jojochuang can you also check once..?

Looks good. @bhavikpatel9977 Why do we have to use quotes with escape. Is it because of keyword group?

+1. going to commit, as there is no concerns on this.


text:
APEXCORE-711 create a new attribute CUSTOM_SSL_SERVER_CONFIG and use its value to set custom ssl server config

@PramodSSImmaneni  pls review and merge as appropriate

also requesting @devtagare to review and merge ...

@PramodSSImmaneni  incorporated your feedback. Pls review and if okay I will squash

@PramodSSImmaneni  if there are no more comments can you merge now?

@vrozov are you fine with @sanjaypujare's comments on the JIRA. Also YARN-6457 has been resolved.

YARN-4562 is also resolved and both YARN-4562 and YARN-6457 have been merged into 2.7 and later branches by the Hadoop committers based on our requests - I was impressed by the speed. It is time this PR gets merged now.

@sanjaypujare can you rebase

See contributor guidelines for git setup:
Sanjay Pujare sanjaypujare@Sanjay-DT-Mac2.local

authors: PramodSSImmaneni sanjaypujare tweise

repo: apache/apex-core


text:
APEXMALHAR-2314 Improper functioning in partitioning for sequentialFileRead for FSRecord

Fix the StreamCodec for FSRecordReader, initially it was hashcode of blockId's mostly always unique.
Hence unable to satisfy the sequentialFileRead property. Now the StreamCodec is modified to work
with hashcode of filePath. So all blocks related to a file would be partitioned on same operator.
Tested with recordReader and verified for sequentialFileRead that all blocks related to a file are partitioned to single operator.

@yogidevendra: Could you please review ?

Let us reuse SequentialFileBlockMetadataCodec from FSInputModule instead of defining separate one.

@yogidevendra : Thanks for suggestion. Makes sense. Only thing that worried me about  creating unnecessary dependency of FSInputModule's StreamCodec on FSRecordReaderModule. So if someone changes FSInput StreamCodec he must consider same in FSRecoredReaderModule.

This dependency should be OK. FSRecordReader is closely aligned with FSInputModule.

Incorporated Yogi's suggestion.

Also please fix the spelling mistake in the JIRA title and commit message.
sequencialFileRead =>sequentialFileRead.
Check for spellings in the code, javadoc as well.

Done. Thanks !
On Wed, Oct 26, 2016 at 11:15 AM, yogidevendra notifications@github.com
wrote:

Also please fix the spelling mistake in the JIRA title and commit message.
sequencialFileRead =>sequentialFileRead.
Check for spellings in the code, javadoc as well.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/apex-malhar/pull/468#issuecomment-256255096,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASvTtiq6elsdBijUCNmb4Zeci-sfmslBks5q3uj8gaJpZM4Keh-S
.

--
Thanks & Regards
Deepak Narkhede

[ERROR] src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java:[32,8] (imports) UnusedImports: Unused import - com.datatorrent.common.partitioner.StatelessPartitioner.
[ERROR] src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java:[33,8] (imports) UnusedImports: Unused import - com.datatorrent.lib.io.block.FSSliceReader.
please fix above checkstyle error.

Sorry my bad. After rebase it occurred. Will resolve the conflicts.
On Nov 7, 2016 3:33 PM, "Tushar R. Gosavi" notifications@github.com wrote:

[ERROR] src/main/java/org/apache/apex/malhar/lib/fs/
FSRecordReaderModule.java:32,8 http://imports UnusedImports: Unused
import - com.datatorrent.common.partitioner.StatelessPartitioner.
[ERROR] src/main/java/org/apache/apex/malhar/lib/fs/
FSRecordReaderModule.java:33,8 http://imports UnusedImports: Unused
import - com.datatorrent.lib.io.block.FSSliceReader.
please fix above checkstyle error.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/apache/apex-malhar/pull/468#issuecomment-258793883,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASvTtqvJp4rc28gSu76eARinF0EAytHpks5q7vdpgaJpZM4Keh-S
.

repo: apache/apex-malhar


text:
Support SHOW command

What feature or improvement would you like to see?
In postgres, the show command returns server/session configuration attributes:
SHOW all

The show command doesn't appear to be recognized by abdc:
>>> cur.execute("show all")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Projects\cqn-bur\.venv_cqn\Lib\site-packages\adbc_driver_manager\dbapi.py", line 698, in execute
    handle, self._rowcount = _blocking_call(
                             ^^^^^^^^^^^^^^^
  File "adbc_driver_manager\\_lib.pyx", line 1590, in adbc_driver_manager._lib._blocking_call
  File "adbc_driver_manager\\_lib.pyx", line 1213, in adbc_driver_manager._lib.AdbcStatement.execute_query
  File "adbc_driver_manager\\_lib.pyx", line 260, in adbc_driver_manager._lib.check_error
adbc_driver_manager.ProgrammingError: INVALID_ARGUMENT: [libpq] Failed to execute query: could not begin COPY: ERROR:  syntax error at or near "show"
LINE 1: COPY (show all) TO STDOUT (FORMAT binary)

Thank you!

repo: apache/arrow-adbc


text:
feat: Support Emit::First for SumDecimalGroupsAccumulator

Which issue does this PR close?

Closes #46.
Rationale for this change

What changes are included in this PR?

How are these changes tested?

CI tests are passed. The failure is due to
WARNING: /Users/runner/hostedtoolcache/Java_Adopt_jdk/17.0.10-7/x64/Contents/Home/bin/java is loading libcrypto in an unsafe way

@viirya I think you need to rebase this PR

Hmm, I think I already rebased on latest main.

Hmm not sure. The same issue happened to https://github.com/apache/arrow-datafusion-comet/pull/40 and it is OK now after the rebasing.

I re-triggered the pipeline.

Hmm, it is actually flaky. Retriggered run is okay.

Merged and added details in the PR description. Thanks.

authors: sunchao viirya

repo: apache/arrow-datafusion-comet


text:
Update repository url

Which issue does this PR close?
Closes #16.
Rationale for this change
We moved to a new repo and need to update the URL.
What changes are included in this PR?
Update repo URL in Cargo.toml
Are there any user-facing changes?
No

Codecov Report

Merging #233 (57cdc95) into master (c945b03) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 57cdc95 differs from pull request most recent head c832aa5. Consider uploading reports for the commit c832aa5 to get more accurate results

@@           Coverage Diff           @@
##           master     #233   +/-   ##
=======================================
  Coverage   76.46%   76.46%
=======================================
  Files         135      135
  Lines       23250    23250
=======================================
  Hits        17777    17777
  Misses       5473     5473

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c945b03...c832aa5. Read the comment docs.

There's one more in 'ballista-ui'
=> https://github.com/apache/arrow-datafusion/blob/master/ballista/ui/scheduler/src/components/Header.tsx#L70
Please check 🙏

authors: andygrove codecov-commenter djKooks

repo: apache/arrow-datafusion


text:
Fix the test failure of size_of_scalar failure on M1/M2 Mac

Signed-off-by: remzi 13716567376yh@gmail.com
Which issue does this PR close?

Closes #3383.
Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Codecov Report

Merging #3398 (4090903) into master (81addf7) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #3398      +/-   ##
==========================================
- Coverage   85.68%   85.68%   -0.01%
==========================================
  Files         298      298
  Lines       54654    54653       -1
==========================================
- Hits        46833    46832       -1
  Misses       7821     7821

Impacted Files
Coverage Δ

datafusion/common/src/scalar.rs
85.11% <100.00%> (-0.01%)
:arrow_down:

datafusion/expr/src/logical_plan/plan.rs
77.02% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

I think we need to merge this before releasing 12.0.0.
@alamb @Dandandan any objection to merging this?

Go ahead :+1:

authors: Dandandan HaoYang670 andygrove codecov-commenter


text:
Make ObjectStoreScheme in the object_store crate public

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to parse an object store URL into a type. Using parse_url and parse_url_opts casts the result into a Box<dyn ...> which makes it impossible to get the underlying type from.
Describe the solution you'd like
Making ObjectStoreScheme public will allow me to do:
let (scheme, path) = ObjectStoreScheme::parse("s3://foo/bar")

+1, I want to support all the different cloud providers in my app, but I need more fine-grained control over how the ObjectStore is constructed, so I can't just use parse_url -- having this public would be very useful, I literally just came here to submit a PR to do the same thing :)

label_issue.py automatically added labels {'object-store'} from #5912

authors: alamb drmorr0 orf

repo: apache/arrow-rs


text:
Wrap all tokio::io::AsyncWrite types to a specific type which is implemented AsyncFileWriter

In current, parquet defines all AsyncWrite implement AsyncFileWriter:
impl<T: AsyncWrite + Unpin + Send> AsyncFileWriter for T {
    ... ...
}

which would in conflict with:
impl<T: AsyncFileWriter + ?Sized + Unpin + Send> AsyncFileWriter for &mut T {
    ... ...
}

this definition is not included right now, however, it should be helpful in more general cases, numerous of similar cases choose to implement this, such as:

tokio::io::AsyncWrite
futures::io::AsyncWrite

Describe the solution you'd like
Wrap all tokio::io::AsyncWrite into a proxy struct, then we can define &mut AsyncFileWriter implements AsyncFileWriter:
pub struct TokioFileWriter<F: AsyncWrite> {
    file: F,
}

impl<F: AsyncWrite> From<F> for TokioFileWriter {
    ... ...
}

impl<F: AsyncWrite + Unpin + Send> for TokioFileWriter {
    ... ...
}

Describe alternatives you've considered
I don't think there is an alternatives.
Additional context
This is a breaking change, but it values to be considered.

Hi, could you share the use case for &mut AsyncFileWriter?

But AsyncArrowWriter is created from AsyncFileWriter, creating an AsyncArrowWriter from &mut AsyncFileWriter is exactly what I want.
struct CustomFile {
    // ...
}

impl AsyncFileWriter for CustomFile {
    fn write(
        &mut self,
        bs: bytes::Bytes,
    ) -> futures::future::BoxFuture<'_, parquet::errors::Result<()>> {
        todo!()
    }

    fn complete(&mut self) -> futures::future::BoxFuture<'_, parquet::errors::Result<()>> {
        todo!()
    }
}

fn test() {
    let mut file = CustomFile {};
    {
        let mut writer =
            AsyncArrowWriter::try_new(&mut file, SchemaRef::from(Schema::empty()), None).unwrap();
        // write record batch
        writer.close()
    }
    // do something with file after writes
}

Also it is helpful to make boxed AsyncFileWriter general, compare with impl AsyncFileWriter for Box<dyn AsyncFileWriter> currently:
impl<T: AsyncFileWriter + ?Sized + Unpin + Send> AsyncFileWriter for Box<T> {
    fn write(&mut self, bs: Bytes) -> BoxFuture<'_, Result<()>> {
        self.as_mut().write(bs)
    }

    fn complete(&mut self) -> BoxFuture<'_, Result<()>> {
        self.as_mut().complete()
    }
}

// do something with file after writes

From my current understanding, you should not do anything to this file after closing it. Can you provide a detailed list of actions you plan to take?

It is my mistake, actually I just want to flush the AsyncArrowWriter, then seek and read this file as native file rather than a AsyncFileWriter, I have fixed the example code.

Understood. I believe you need AsyncArrowWriter::into_inner.

First of all, allowing users to manipulate the writer after AsyncArrowWriter has written the header and data could be error-prone.
Additionally, it doesn't make sense given our context could involve S3, Azure Blob, or GCS storage services. In these contexts, the file doesn't exist before close has been called, so you can't perform seek/read or other actions.
If you simply want to use this API in the local filesystem context, would you consider using try_clone instead?

Additionally, it doesn't make sense given our context could involve S3, Azure Blob, or GCS storage services.

It does make sense in local filesystems, which allows to implements AsyncFileWriter.

First of all, allowing users to manipulate the writer after AsyncArrowWriter has written the header and data could be error-prone

However, &mut tokio::io::AsyncWriter is a valid AsyncFileWriter in the current, if you think parquet should not allow &mut AsyncFileWriter be an AsyncFileWriter, then I think &mut tokio::io::AsyncWriter should be fixed. What ever happends, the behavior of tokio::io::AsyncWriter and non-AsyncWriter should be aligned.

It does make sense in local filesystems, which allows to be a AsyncFileWriter, don't give up eating for fear of choking.

I agree with you that it useful in the local fs context. Therefore, I propose a new API called AsyncArrowWriter::into_inner. I believe this can address your needs in a compatible manner.

Understood. I believe you need AsyncArrowWriter::into_inner.

I think into_inner has different behaviors and takes different overheads, it would move the AsyncFileWriter but &mut AsyncFileWriter is in-placed. But I think it should be nice to have AsyncArrowWriter::into_inner method.

authors: Xuanwo ethe


text:
feat: Add Commands enum to decode prost messages to strong type

Which issue does this PR close?
Closes #3874
Rationale for this change
Per the issue, this PR introduces a Commands enum to allow client to match on all supported messages.
The following minor improvements were made to the prost_message_ext macro:
1. Generate a constant for the type URLs:
const ACTION_CLOSE_PREPARED_STATEMENT_REQUEST_TYPE_URL: &'static str = concat!( "type.googleapis.com/arrow.flight.protocol.sql.", stringify!( ActionClosePreparedStatementRequest ) );
const ACTION_CREATE_PREPARED_STATEMENT_REQUEST_TYPE_URL: &'static str = concat!( "type.googleapis.com/arrow.flight.protocol.sql.", stringify!( ActionCreatePreparedStatementRequest ) );
// Remainder omitted for brevity

2. Commands enum to unpack any of the supported FlightSQL commands:
pub enum Commands {
    ActionClosePreparedStatementRequest(ActionClosePreparedStatementRequest),
    ActionCreatePreparedStatementRequest(ActionCreatePreparedStatementRequest),
    // Remainder omitted for brevity
}

with an associated unpack function:
impl Commands {
    pub fn unpack(any: Any) -> Result<Commands, ArrowError> {
        match any.type_url.as_str() {
            ACTION_CLOSE_PREPARED_STATEMENT_REQUEST_TYPE_URL => {
                let m: ActionClosePreparedStatementRequest = Message::decode(&*any.value).map_err(|err| {
                    ArrowError::ParseError({
                        let res = ::alloc::fmt::format(format_args!("Unable to decode Any value: {err}", err = err));
                        res
                    })
                })?;
                Ok(Self::ActionClosePreparedStatementRequest(m))
            }
            ACTION_CREATE_PREPARED_STATEMENT_REQUEST_TYPE_URL => {
                let m: ActionCreatePreparedStatementRequest = Message::decode(&*any.value).map_err(|err| {
                    ArrowError::ParseError({
                        let res = ::alloc::fmt::format(format_args!("Unable to decode Any value: {err}", err = err));
                        res
                    })
                })?;
                Ok(Self::ActionCreatePreparedStatementRequest(m))
            }

            // Remainder omitted for brevity

            _ => Err(ArrowError::ParseError({
                let res = ::alloc::fmt::format(IntellijRustDollarCrate::__export::format_args!("Unable to decode Any value: {}", any.type_url));
                res
            }))
        }
    }
}

3. Update other generated code to use the shared constant for the type URL:
impl ProstMessageExt for ActionClosePreparedStatementRequest {
    fn type_url() -> &'static str {
        ACTION_CLOSE_PREPARED_STATEMENT_REQUEST_TYPE_URL
    }

    fn as_any(&self) -> Any {
        Any {
            type_url: <ActionClosePreparedStatementRequest>::type_url().to_string(),
            value: self.encode_to_vec().into(),
        }
    }
}
impl ProstMessageExt for ActionCreatePreparedStatementRequest {
    fn type_url() -> &'static str {
        ACTION_CREATE_PREPARED_STATEMENT_REQUEST_TYPE_URL
    }

    fn as_any(&self) -> Any {
        Any {
            type_url: <ActionCreatePreparedStatementRequest>::type_url().to_string(),
            value: self.encode_to_vec().into(),
        }
    }
}
// Remainder omitted for brevity

What changes are included in this PR?

Are there any user-facing changes?
API improvements for users of the FlightSQL module.

I plan to review this shortly

Here are some suggested improvements: https://github.com/stuartcarnie/arrow-rs/pull/2# (docs + revamp some of the server.rs code to use the new dispatch logic 👍 )

I pushed a few minor doc updates directly to this branch.

@alamb see ea18fa9697d1face7945e40dc6542b59c3890614 for the addition of the Unknown(Any) variant, which was a great suggestion. I've also updated do_get and do_put, and the simple test now passes ✅

Thanks -- I plan to come back to this PR over the weekend but got overloaded with other things this week

I am sorry for the delay -- I will get this done / merged this week

Github seems to have some issue -- restarting

I propose some additional documentation for this feature here https://github.com/apache/arrow-rs/pull/4012

authors: alamb stuartcarnie


text:
[C++][Python] Support start == stop in list_slice kernel

[GitHub PR 14395 | https://github.com/apache/arrow/pull/14395] adds the list_slice kernel, but does not implement the case where stop == stop, which should return empty lists.
Reporter: Miles Granger / @milesgranger
Assignee: Miles Granger / @milesgranger
Related issues:

[C++] Add kernel for slicing list values (relates to)

PRs and other links:

GitHub Pull Request #14836

Note: This issue was originally created as ARROW-18281. Please see the migration documentation for further details.

take

authors: asfimport felipecrv

repo: apache/arrow


text:
[Java] Update Unit Tests for Compression Module

Describe the enhancement requested
This is a sub-issue of the issue below.

#41680

The unit tests should use the org.junit.jupiter.api package. This issue deals with the compression module.
Component(s)
Java

Issue resolved by pull request 42044
https://github.com/apache/arrow/pull/42044


text:
ARROW-13976: [C++] Add path to libjvm.so in ARM CPU

resolve issue ARROW-13976, add path in hdfs_internal.cc to find libjvm.so in ARM CPU.

@cyb70289 Would you know if this looks good?

MinGW failures tracked at https://issues.apache.org/jira/browse/ARROW-13999

@pitrou, do you have comments?

No :-)

Thanks @wuzhuoming !

authors: cyb70289 pitrou wuzhuoming


text:
ARROW-16918: [Gandiva][C++] Adding UTC-local timezone conversion functions

Adding functions
to_utc_timezone : Converts a timestamp from local timezone to UTC time
from_utc_timezone : Converts a timestamp from UTC time to local time

We can fix style by cmake --build ${BUILD_DIR} --target format but you need to install clang-format-12.
If it's difficult to prepare clang-format-12, I can fix style and push the fix to this branch.


text:
ARROW-2868: [Packaging] Fix Apache Arrow ORC GLib related problems

.pc file name was wrong: arrow-glib-orc.pc -> arrow-orc-glib.pc

We use arrow-XXX-glib.pc rule such as arrow-gpu-glib.pc.

.spec didn't include arrow-orc-glib.pc.
.spec included duplicated -DARROW_ORC=ON

https://github.com/apache/arrow-dist/commit/6580a6b7f7fd3d2e5f06b68fe606767db72faf9e was backported.

Codecov Report

Merging #2280 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2280      +/-   ##
==========================================
+ Coverage   84.24%   84.25%   +<.01%
==========================================
  Files         290      290
  Lines       44297    44297
==========================================
+ Hits        37320    37321       +1
+ Misses       6946     6945       -1
  Partials       31       31

Impacted Files
Coverage Δ

cpp/src/arrow/util/thread-pool-test.cc
99.45% <0%> (+0.54%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ba8769...73cd564. Read the comment docs.

Testing it in build: https://travis-ci.org/kszucs/crossbow/builds/405266989

The build passed, successfully uploaded the artifacts too: https://github.com/kszucs/crossbow/releases/tag/build-232-centos-7
Thanks @kou!

authors: codecov-io kou kszucs


text:
GH-35926: [C++][Parquet] Testing disable ColumnIndex by disable statistics

Rationale for this change

Allow only build Offset Index for some column by disable statistics
What changes are included in this PR?

Fix a tiny bug
Are these changes tested?

Yes
Are there any user-facing changes?
Actually no change?

Closes: #35926

cc @wgtmac @pitrou

@pitrou Mind take a look?

@pitrou @wgtmac Do you still have any comments?

The remaining CI failures are unrelated, will merge.

authors: mapleFU pitrou


text:
GH-43228: [C++] Fix Abseil compile error on GCC 13

Rationale for this change
When trying to compile Arrow with GCC 13, it fails due to ABSEIL missing a <cstdint> include, this PR addresses the issue by adding the missing include.
There have been past reports for this issue too: https://github.com/apache/arrow/issues/36969
This is a more minimal fix that tries to avoid the complexity of previous attempts like  https://github.com/apache/arrow/pull/43147 and https://github.com/apache/arrow/pull/37066 which involved updating Abseil and facing additional issues to fix.
What changes are included in this PR?
Add the missing include when GCC>=13
Are these changes tested?
They are tested by the existing compile infrastructure and testsuite.
We don't seem to have GCC 13 in our testsuite, I could only verify on my own PC that it compiled successfully
Are there any user-facing changes?
No, all behaviours should remain the same

GitHub Issue: #43228

@github-actions crossbow submit -g cpp -g r

@raulcd @assignUser this should be ready for review, I tried a more conservative approach as upgrading Abseil introduced more problems than it fixed even on the most conservative version upgrade.

Could you open a new issue for this instead of reusing existing closed issue?

Could you open a new issue for this instead of reusing existing closed issue?

Given it's the same exact error message shouldn't we preserve a consolidated history by keeping the original issue?
Happy to open a new issue if that's the preference

The problem is that the original issue is in the 13.0.0 milestone and all our current release scripts take that into account to generate release notes, etcetera. If we reuse the same issue it would be as if this PR was released on 13.0.0 and if we require to back port this issue it gets messy.
It has it's pros and cons but the current process doesn't account for two PRs pointing to the same GH issue.

The problem is that the original issue is in the 13.0.0 milestone

Makes sense, I didn't check and thought it was closed without being done. I'll open a new one.

@github-actions crossbow submit test-r-gcc-13

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13-bundled

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13-bundled

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13-bundled

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13-bundled

Hmm. Could you also add -e ARROW_VERBOSE_THIRDPARTY_BUILD=ON?

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13-bundled

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13-bundled

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13-bundled

@kou test-ubuntu-24.04-cpp-gcc-13-bundled now compiles correctly. It fails the test, but it seems unrelated as they are timezone related tests. I think that this is ready for final review.

@github-actions crossbow submit test-ubuntu-20.04-cpp-bundled

@github-actions crossbow submit test-ubuntu-24.04-cpp-gcc-13-bundled

The timestamp tests now pass too by installing tzdata-legacy

@github-actions crossbow submit test-r-library-r-base-latest test-r-depsource-bundled

@github-actions crossbow submit test-r-depsource-bundled

Looks good, thanks! Could you also remove the workaround we added to r/configure?

Done and tested on test-r-depsource-bundled

@github-actions crossbow submit r-binary-packages
Nice, thanks. I'll run this as it has a build on 24.04 which comes with the new gcc. I'll merge after

Ugh, turns out ubuntu-latest is still 22.04, I modified the crossbow workflow to re-run on 24.04

authors: amol- assignUser kou raulcd


text:
ARROW-6031: [Java] Support iterating a vector by ArrowBufPointer

Provide the functionality to traverse a vector (fixed-width vector & variable-width vector) by an iterator. This is convenient for scenarios when accessing vector elements in sequence.

Codecov Report

Merging #4950 into master will decrease coverage by 23.5%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master    #4950       +/-   ##
===========================================
- Coverage   88.53%   65.02%   -23.51%
===========================================
  Files         912      488      -424
  Lines      116620    64921    -51699
  Branches     1418        0     -1418
===========================================
- Hits       103255    42218    -61037
- Misses      13003    22703     +9700
+ Partials      362        0      -362

Impacted Files
Coverage Δ

cpp/src/arrow/util/memory.h
0% <0%> (-100%)
:arrow_down:

cpp/src/gandiva/date_utils.h
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/util/memory.cc
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/filesystem/util-internal.cc
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/util/sse-util.h
0% <0%> (-100%)
:arrow_down:

cpp/src/gandiva/decimal_type_util.h
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/compute/logical_type.h
0% <0%> (-100%)
:arrow_down:

cpp/src/parquet/hasher.h
0% <0%> (-100%)
:arrow_down:

cpp/src/gandiva/basic_decimal_scalar.h
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/compute/kernels/boolean.cc
0% <0%> (-100%)
:arrow_down:

... and 664 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 141a213...29f63d2. Read the comment docs.

@jacques-n thanks a lot for your comments.
I am convinced that these APIs should not be added to the core APIs. In fact, they are not intended to be the primary way of accessing vector data elements.
The iterator can be something nice to have, but not essential.
I can think of two applications of this iterator:

quickly calculating/combining hash code & determining element equality.
for traversing some "vectors" that does not support random access, like compressed vectors.

@jacques-n @emkornfield
I have removed the methods out of the core APIs. Please check if it looks good. Thank yo so much.

@jacques-n any more concerns if the APIs are not directly on Vector classes?

I think @jacques-n is out this week, if there are no more comments next week we can probably merge this.

I think @jacques-n is out this week, if there are no more comments next week we can probably merge this.

@emkornfield Thanks for your comments.
I have revised the PR a little, since now we have a common super interface for fixed width vectors and the variable width vectors.
@jacques-n 's comments make sense. However, I think this is at least useful for variable width vectors.

+1.

authors: codecov-io emkornfield liyafan82


text:
ARROW-9210: [C++] Use BitBlockCounter in array/visitor_inline.h

This significantly speeds up processing of mostly-not-null or mostly-null data, while having almost no overhead for the other scenarios where you rarely have a word-sized run of all-not-null or all-null-data. Because BitUtil::GetBit is used for bit-checking in the scenario where you need to check every bit in the whole array individually I show slight but inconclusive perf regression similar with the perf difference we've seen comparing BitmapReader with the naive approach calling GetBit inside a loop. This small perf degradation seems to be present mostly with gcc and not meaningfully with clang on Linux.
For data with null_count 0, data is processed in blocks of INT16_MAX values at a time, so this adds no meaningful overhead for this case either.
I modified the hash benchmarks where this code is used to exhibit both the cases that benefit from this optimization as well as the ones that don't.

Here's a benchmark run with gcc-8
---------------------------------------------------------------
Benchmark                        Time           CPU Iterations
---------------------------------------------------------------
BuildDictionary            3219443 ns    3219440 ns        218   1.21215GB/s
BuildStringDictionary      3692881 ns    3692881 ns        192   81.7532MB/s
UniqueInt64/0             14413456 ns   14413251 ns         48 null_percent=0   2.16814GB/s
UniqueInt64/1             15516052 ns   15515737 ns         45 null_percent=0.1   2.01408GB/s
UniqueInt64/2             17031282 ns   17031266 ns         41 null_percent=1   1.83486GB/s
UniqueInt64/3             20680114 ns   20680064 ns         34 null_percent=10   1.51112GB/s
UniqueInt64/4             12018069 ns   12017844 ns         57 null_percent=99    2.6003GB/s
UniqueInt64/5              9179953 ns    9179946 ns         77 null_percent=100   3.40416GB/s
UniqueInt64/6             15501523 ns   15501496 ns         45 null_percent=0   2.01593GB/s
UniqueInt64/7             16482935 ns   16482300 ns         41 null_percent=0.1   1.89597GB/s
UniqueInt64/8             18349988 ns   18349317 ns         38 null_percent=1   1.70306GB/s
UniqueInt64/9             21439268 ns   21439244 ns         32 null_percent=10   1.45761GB/s
UniqueInt64/10            12530067 ns   12529871 ns         55 null_percent=99   2.49404GB/s
UniqueInt64/11             9167314 ns    9167365 ns         75 null_percent=100   3.40883GB/s
UniqueString10bytes/0     43535899 ns   43535846 ns         16 null_percent=0   918.783MB/s
UniqueString10bytes/1     45130595 ns   45129634 ns         16 null_percent=0.1   886.336MB/s
UniqueString10bytes/2     45249034 ns   45247983 ns         15 null_percent=1   884.017MB/s
UniqueString10bytes/3     45101533 ns   45100209 ns         16 null_percent=10   886.914MB/s
UniqueString10bytes/4      4316048 ns    4316019 ns        163 null_percent=99   9.05059GB/s
UniqueString10bytes/5      1435781 ns    1435763 ns        485 null_percent=100   27.2068GB/s
UniqueString10bytes/6     59100344 ns   59098817 ns         12 null_percent=0   676.832MB/s
UniqueString10bytes/7     59797544 ns   59795857 ns         12 null_percent=0.1   668.943MB/s
UniqueString10bytes/8     61024697 ns   61023090 ns         11 null_percent=1    655.49MB/s
UniqueString10bytes/9     59817211 ns   59816339 ns         12 null_percent=10   668.714MB/s
UniqueString10bytes/10     4950387 ns    4950242 ns        134 null_percent=99   7.89103GB/s
UniqueString10bytes/11     1443482 ns    1443434 ns        446 null_percent=100   27.0622GB/s
UniqueString100bytes/0    95609006 ns   95606132 ns          7 null_percent=0   4.08577GB/s
UniqueString100bytes/1    96850582 ns   96849441 ns          7 null_percent=0.1   4.03332GB/s
UniqueString100bytes/2    95404742 ns   95404634 ns          7 null_percent=1    4.0944GB/s
UniqueString100bytes/3    89401775 ns   89401006 ns          8 null_percent=10   4.36936GB/s
UniqueString100bytes/4     4705868 ns    4705746 ns        148 null_percent=99   83.0102GB/s
UniqueString100bytes/5     1434077 ns    1434055 ns        486 null_percent=100   272.392GB/s
UniqueString100bytes/6   206155133 ns  206148425 ns          3 null_percent=0   1.89487GB/s
UniqueString100bytes/7   204661287 ns  204653659 ns          3 null_percent=0.1   1.90871GB/s
UniqueString100bytes/8   205941884 ns  205941271 ns          3 null_percent=1   1.89678GB/s
UniqueString100bytes/9   192074501 ns  192073431 ns          4 null_percent=10   2.03373GB/s
UniqueString100bytes/10    6180349 ns    6180227 ns        111 null_percent=99   63.2056GB/s
UniqueString100bytes/11    1474565 ns    1474564 ns        482 null_percent=100   264.909GB/s
UniqueUInt8/0              1990025 ns    1990023 ns        348 null_percent=0   1.96292GB/s
UniqueUInt8/1              2594146 ns    2594089 ns        272 null_percent=0.1   1.50583GB/s
UniqueUInt8/2              4726027 ns    4726053 ns        145 null_percent=1   846.372MB/s
UniqueUInt8/3              9465222 ns    9465126 ns         75 null_percent=10   422.604MB/s
UniqueUInt8/4              3557141 ns    3557135 ns        195 null_percent=99    1124.5MB/s
UniqueUInt8/5              2259664 ns    2259664 ns        314 null_percent=100   1.72869GB/s

Here is the % diff versus the baseline.

Cases 1 and 7 are the mostly-not-null cases. This shows a 15-20% perf improvement
Cases 5 and 11 are the all-null cases.
Case 4 is the 99% null case
The "BuildDictionary" case at the bottom with the perf regression is one of the "worst case scenarios". 89% of the values are null and so we almost never observe an all-null or all-not-null block. The use of BitUtil::GetBit over BitmapReader causes this slightly regression since nearly every validity bit must be checked separately. I don't think it's worth optimizing for this case since the others are more empirically representative of real world data

                  benchmark          baseline        contender  change %  regression
8    UniqueString100bytes/5    40.668 GiB/sec  272.392 GiB/sec   569.787       False
37    UniqueString10bytes/5     4.064 GiB/sec   27.207 GiB/sec   569.456       False
33   UniqueString10bytes/11     4.065 GiB/sec   27.062 GiB/sec   565.751       False
12  UniqueString100bytes/11    40.578 GiB/sec  264.909 GiB/sec   552.841       False
0     UniqueString10bytes/4     3.568 GiB/sec    9.051 GiB/sec   153.692       False
36   UniqueString100bytes/4    34.408 GiB/sec   83.010 GiB/sec   141.252       False
19   UniqueString10bytes/10     3.375 GiB/sec    7.891 GiB/sec   133.794       False
24            UniqueUInt8/1   677.981 MiB/sec    1.506 GiB/sec   127.435       False
5   UniqueString100bytes/10    30.775 GiB/sec   63.206 GiB/sec   105.381       False
27            UniqueUInt8/5  1000.163 MiB/sec    1.729 GiB/sec    76.989       False
13            UniqueUInt8/2   650.819 MiB/sec  846.372 MiB/sec    30.047       False
29           UniqueInt64/11     2.703 GiB/sec    3.409 GiB/sec    26.126       False
7             UniqueInt64/5     2.704 GiB/sec    3.404 GiB/sec    25.903       False
18            UniqueUInt8/4   932.926 MiB/sec    1.098 GiB/sec    20.535       False
23            UniqueInt64/1     1.681 GiB/sec    2.014 GiB/sec    19.840       False
21            UniqueInt64/7     1.628 GiB/sec    1.896 GiB/sec    16.476       False
31            UniqueInt64/2     1.658 GiB/sec    1.835 GiB/sec    10.651       False
20    UniqueString10bytes/7   612.647 MiB/sec  668.943 MiB/sec     9.189       False
16            UniqueInt64/3     1.386 GiB/sec    1.511 GiB/sec     9.053       False
38    UniqueString10bytes/8   601.259 MiB/sec  655.490 MiB/sec     9.019       False
1             UniqueUInt8/0     1.808 GiB/sec    1.963 GiB/sec     8.588       False
41            UniqueInt64/9     1.355 GiB/sec    1.458 GiB/sec     7.562       False
14    UniqueString10bytes/1   830.614 MiB/sec  886.336 MiB/sec     6.709       False
4             UniqueInt64/8     1.603 GiB/sec    1.703 GiB/sec     6.260       False
32    UniqueString10bytes/2   847.018 MiB/sec  884.017 MiB/sec     4.368       False
42            UniqueInt64/4     2.508 GiB/sec    2.600 GiB/sec     3.701       False
39    UniqueString10bytes/3   855.985 MiB/sec  886.914 MiB/sec     3.613       False
28           UniqueInt64/10     2.413 GiB/sec    2.494 GiB/sec     3.360       False
34   UniqueString100bytes/3     4.254 GiB/sec    4.369 GiB/sec     2.722       False
11   UniqueString100bytes/2     3.993 GiB/sec    4.094 GiB/sec     2.544       False
9     UniqueString10bytes/9   654.257 MiB/sec  668.714 MiB/sec     2.210       False
35    UniqueString10bytes/6   662.915 MiB/sec  676.832 MiB/sec     2.099       False
6     BuildStringDictionary    80.971 MiB/sec   81.753 MiB/sec     0.966       False
22   UniqueString100bytes/1     4.002 GiB/sec    4.033 GiB/sec     0.783       False
25            UniqueInt64/0     2.153 GiB/sec    2.168 GiB/sec     0.697       False
17    UniqueString10bytes/0   917.726 MiB/sec  918.783 MiB/sec     0.115       False
43            UniqueInt64/6     2.017 GiB/sec    2.016 GiB/sec    -0.071       False
40   UniqueString100bytes/0     4.091 GiB/sec    4.086 GiB/sec    -0.130       False
3    UniqueString100bytes/7     1.938 GiB/sec    1.909 GiB/sec    -1.519       False
26   UniqueString100bytes/8     1.954 GiB/sec    1.897 GiB/sec    -2.935       False
2    UniqueString100bytes/9     2.114 GiB/sec    2.034 GiB/sec    -3.782       False
30   UniqueString100bytes/6     2.008 GiB/sec    1.895 GiB/sec    -5.649        True
10            UniqueUInt8/3   474.468 MiB/sec  422.604 MiB/sec   -10.931        True
15          BuildDictionary     1.776 GiB/sec    1.212 GiB/sec   -31.742        True

Also on the binary size, these changes add about 75KB to libarrow.so. My guess is the difference is mostly coming from code inlining for the all-null case (which wasn't split out before)

FWIW on the "gcc/clang perf discussion", clang also shows performance benefits and limited downside
                  benchmark         baseline        contender  change %  regression
2            UniqueInt64/11    6.444 GiB/sec   18.511 GiB/sec   187.240       False
31            UniqueInt64/5    6.470 GiB/sec   18.390 GiB/sec   184.244       False
39            UniqueUInt8/5  810.180 MiB/sec    1.747 GiB/sec   120.867       False
26            UniqueUInt8/1  683.475 MiB/sec    1.430 GiB/sec   114.196       False
42            UniqueInt64/4    5.424 GiB/sec    6.965 GiB/sec    28.397       False
18            UniqueInt64/1    2.672 GiB/sec    3.411 GiB/sec    27.627       False
40            UniqueUInt8/2  654.320 MiB/sec  826.916 MiB/sec    26.378       False
33            UniqueUInt8/4  758.115 MiB/sec  947.360 MiB/sec    24.962       False
25           UniqueInt64/10    5.248 GiB/sec    6.426 GiB/sec    22.460       False
9    UniqueString100bytes/5   26.923 GiB/sec   32.142 GiB/sec    19.384       False
35   UniqueString10bytes/11    2.691 GiB/sec    3.207 GiB/sec    19.173       False
3     UniqueString10bytes/5    2.695 GiB/sec    3.200 GiB/sec    18.731       False
20  UniqueString100bytes/11   26.909 GiB/sec   31.831 GiB/sec    18.291       False
30            UniqueInt64/7    2.514 GiB/sec    2.890 GiB/sec    14.960       False
37            UniqueInt64/2    2.619 GiB/sec    2.975 GiB/sec    13.578       False
11    UniqueString10bytes/4    2.487 GiB/sec    2.700 GiB/sec     8.596       False
32   UniqueString10bytes/10    2.386 GiB/sec    2.589 GiB/sec     8.481       False
0    UniqueString100bytes/4   24.419 GiB/sec   26.365 GiB/sec     7.966       False
38  UniqueString100bytes/10   22.463 GiB/sec   24.128 GiB/sec     7.411       False
34            UniqueInt64/8    2.392 GiB/sec    2.563 GiB/sec     7.157       False
19    UniqueString10bytes/1  781.817 MiB/sec  835.760 MiB/sec     6.900       False
43            UniqueInt64/3    2.184 GiB/sec    2.331 GiB/sec     6.721       False
24    UniqueString10bytes/7  583.523 MiB/sec  621.007 MiB/sec     6.424       False
15   UniqueString100bytes/7    1.936 GiB/sec    2.024 GiB/sec     4.538       False
6     UniqueString10bytes/2  780.337 MiB/sec  805.686 MiB/sec     3.248       False
27   UniqueString100bytes/2    3.934 GiB/sec    4.059 GiB/sec     3.197       False
13   UniqueString100bytes/1    3.898 GiB/sec    3.995 GiB/sec     2.485       False
7     UniqueString10bytes/8  592.115 MiB/sec  604.865 MiB/sec     2.153       False
29   UniqueString100bytes/8    1.969 GiB/sec    2.011 GiB/sec     2.111       False
21            UniqueInt64/9    2.034 GiB/sec    2.048 GiB/sec     0.676       False
1     BuildStringDictionary   85.937 MiB/sec   85.928 MiB/sec    -0.010       False
41            UniqueUInt8/3  449.171 MiB/sec  448.844 MiB/sec    -0.073       False
28   UniqueString100bytes/0    4.084 GiB/sec    4.077 GiB/sec    -0.161       False
4    UniqueString100bytes/3    4.255 GiB/sec    4.235 GiB/sec    -0.450       False
5    UniqueString100bytes/6    2.054 GiB/sec    2.033 GiB/sec    -1.041       False
14   UniqueString100bytes/9    2.138 GiB/sec    2.107 GiB/sec    -1.449       False
8             UniqueUInt8/0    1.777 GiB/sec    1.750 GiB/sec    -1.487       False
23            UniqueInt64/0    3.860 GiB/sec    3.799 GiB/sec    -1.560       False
10    UniqueString10bytes/9  616.458 MiB/sec  605.470 MiB/sec    -1.782       False
22    UniqueString10bytes/3  799.494 MiB/sec  783.825 MiB/sec    -1.960       False
17    UniqueString10bytes/6  647.921 MiB/sec  631.631 MiB/sec    -2.514       False
36          BuildDictionary    1.539 GiB/sec    1.498 GiB/sec    -2.694       False
16            UniqueInt64/6    3.193 GiB/sec    3.077 GiB/sec    -3.634       False
12    UniqueString10bytes/0  881.975 MiB/sec  839.487 MiB/sec    -4.817       False

@ursabot benchmark --suite-filter=arrow-compute-vector-sort-benchmark

I see big performance drop from some counting sort cases, also tested on my local machine. Should be related to these visitor code: https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/kernels/vector_sort.cc#L133-L155

Sorting seems too important to leave it to these relatively complex templates, I would suggest implementing the counting sort without using VisitArrayDataInline

Also, I don't really understand the use of util::optional in these templates. The user should pass separate lambdas for the not-null and null cases

I'm refactoring to nix util::optional.

FWIW the performance issue seems to be more pronounced on gcc than clang, here is the benchmark comparison on my machine with clang-8
                                              benchmark         baseline        contender  change %                                       counters
1     SortToIndicesInt64Count/32768/10000/min_time:1.000    1.560 GiB/sec    2.000 GiB/sec    28.163    {'iterations': 70030, 'null_percent': 0.01}
15  SortToIndicesInt64Compare/32768/10000/min_time:1.000  145.735 MiB/sec  158.918 MiB/sec     9.046     {'iterations': 6654, 'null_percent': 0.01}
5     SortToIndicesInt64Compare/32768/100/min_time:1.000  149.117 MiB/sec  159.609 MiB/sec     7.036      {'iterations': 6545, 'null_percent': 1.0}
7       SortToIndicesInt64Compare/32768/0/min_time:1.000  153.027 MiB/sec  162.227 MiB/sec     6.012      {'iterations': 6862, 'null_percent': 0.0}
4      SortToIndicesInt64Compare/32768/10/min_time:1.000  160.419 MiB/sec  167.725 MiB/sec     4.554     {'iterations': 6934, 'null_percent': 10.0}
2       SortToIndicesInt64Compare/32768/2/min_time:1.000  255.024 MiB/sec  260.284 MiB/sec     2.063    {'iterations': 11390, 'null_percent': 50.0}
9       SortToIndicesInt64Count/32768/100/min_time:1.000    1.486 GiB/sec    1.458 GiB/sec    -1.912     {'iterations': 66757, 'null_percent': 1.0}
10        SortToIndicesInt64Count/32768/0/min_time:1.000    2.143 GiB/sec    2.067 GiB/sec    -3.568     {'iterations': 98191, 'null_percent': 0.0}
13      SortToIndicesInt64Count/8388608/1/min_time:1.000    4.215 GiB/sec    3.813 GiB/sec    -9.531     {'iterations': 762, 'null_percent': 100.0}
11        SortToIndicesInt64Count/32768/2/min_time:1.000  679.023 MiB/sec  609.379 MiB/sec   -10.256    {'iterations': 29602, 'null_percent': 50.0}
0       SortToIndicesInt64Count/1048576/1/min_time:1.000    4.487 GiB/sec    4.021 GiB/sec   -10.400    {'iterations': 6550, 'null_percent': 100.0}
12    SortToIndicesInt64Compare/8388608/1/min_time:1.000    4.250 GiB/sec    3.762 GiB/sec   -11.476     {'iterations': 766, 'null_percent': 100.0}
6         SortToIndicesInt64Count/32768/1/min_time:1.000    4.758 GiB/sec    4.185 GiB/sec   -12.040  {'iterations': 217705, 'null_percent': 100.0}
8       SortToIndicesInt64Compare/32768/1/min_time:1.000    4.730 GiB/sec    4.125 GiB/sec   -12.780  {'iterations': 213908, 'null_percent': 100.0}
3     SortToIndicesInt64Compare/1048576/1/min_time:1.000    4.556 GiB/sec    3.953 GiB/sec   -13.228    {'iterations': 6539, 'null_percent': 100.0}
14       SortToIndicesInt64Count/32768/10/min_time:1.000    1.316 GiB/sec    1.051 GiB/sec   -20.108    {'iterations': 59539, 'null_percent': 10.0}

I'm refactoring to nix util::optional. I'm too tired to finish it tonight so I'll work on it tomorrow morning. If the perf regression isn't gone I'll rewrite the sort kernels.

Very likely I'm wrong. I remember util::optional is added due to CI failure https://github.com/apache/arrow/pull/6495#issuecomment-593732821
I think this patch is okay.
Sorting regression can be fixed (maybe improved). I'm okay to do the follow up changes.

Let's leave sorting optimizations for another PR. I'll review this one.

thanks @pitrou and @cyb70289 -- I will spend a little time on the count-sort implementation and post a new patch

authors: cyb70289 pitrou wesm


text:
AVRO-2211: SchemaBuilder equivalent or other means of schema creation

Make sure you have checked all steps below.
Jira

[x]  My PR addresses the following Avro Jira

https://issues.apache.org/jira/browse/AVRO-2211
In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

[x]  My PR adds the following unit tests OR does not need testing for this extremely good reason:
Add a new test case to the Avro.Test.CodeGenTest.TestCodeGen unit test
Enable the tests to run under .NET Core by adding compilation support with the Microsoft.CodeAnalysis.CSharp package

Commits

[x]  My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x]  In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain Javadoc that explain what it does

@yanivru Can you start by updating the description to follow the standards for the pull request?

Long awaited feature. Thanks for doing it! I assume https://issues.apache.org/jira/browse/AVRO-3003 is s typo in the PR description. The PR title has the proper tickjet number.

@KyleSchoonover Fixed all issues. I changed the constructors to factory methods. If you think it's ok I will leave it. If not I can revert it.
@zcsizmadia Added more unit tests

@yanivru Plz review the CodeQL warnings or suggestions and I think it is ready to be approved. Thanks for all the changes!

@zcsizmadia I think I fixed all CodeQL issues

Thank you, @yanivru !

authors: KyleSchoonover martin-g yanivru zcsizmadia

repo: apache/avro


text:
[BEAM-8666] Remove dependency between DataflowRunner and PortableRunner introduced by PR#9811

PR#9811 creates a new dependency between DataflowRunner and PortableRunner. DataflowRunner uses DockerEnvironment but there was no need for DataflowRunner to depend on the PortableRunner module before. We will refactor out default_docker_image() method to live in apache_beam.transforms.Environment.
JIRA issue: https://issues.apache.org/jira/browse/BEAM-8666

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

R: @mxm @chamikaramj @chadrik

LGTM. Thanks for the fix.
Will merge after tests pass.

authors: chamikaramj violalyu

repo: apache/beam


text:
[BEAM-9468]  Hl7v2 io

Add HL7v2IO connector for the Google Cloud Healthcare API.
Initial work for these connectors is being prioritized based to support various
customer use cases for HL7v2 -> FHIR ETL pipelines.
Healthcare clients are extremely concerned with not losing data in their pipelines and want dead letter queues for these sources / sinks.
With this in mind the HL7v2IO.Read is designed as a mini pipeline that:

Starts with an arbitrary PTransform to populate a PCollection of message IDs. This was chosen to give flexibility between reading the HL7v2 pubsub notifications, reading an entire HL7v2 store (with optional filter) or using some other method (e.g. a hand prepared subset of message IDs uploaded to GCS and read w/ TextIO). In the future we can gain efficiency here when the Healthcare API exposes a batch read method where we can batch up requests rather than making a separate call per element.
Then a DoFn that attempts to fetch the actual message contents from the HL7v2 store. Successfully fetched messages will be written to HL7v2IO.Read.OUT tag and Failures will be written to a HLv2IO.Read.DEAD_LETTER tag. The "expected" failure case would be attempting to fetch a message ID that doesn't exist.

Similarly, the HL7v2IO.Write is designed as a mini pipeline that

starts with a PTransform that attempts to ingest a PCollection of Messages

then applies the this.getDeadLetterPTransform() to the PCollection of messages that failed to ingest. This gives the customer the ability to capture dead letters in a system of their chosing (e.g. gcs, bq, pubsub) based on their needs.

HL7v2:

Unbounded Read: Uses PubsubIO to read notification subscription
Bounded Read: DoFn to get message IDs using the Messages.List REST API method

Future Work will include a very similar IO transform for FHIR store:

FhirIO

Write use the executeBundle REST API method to execute transactions on the FHIR Store
Read with read / search / from notification subscription

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[x] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@lastomato FYI Moved this to it's own PR.
Once we get feedback and iterate on this with beam reviewer, I can open a PR for FHIRIO.

R: @pabloem

retest this please

("retest this please" is a code phrase for jenkins to run precommits on the PR)

since your change is a Java change, you don't need to worry about the Python errors for now. They seem to come from upstream.

@lastomato
Thoughts on how we could rework the HCAPI client to get around this failing CI Check?
https://builds.apache.org/job/beam_PreCommit_Java_Commit/10418/testReport/org.apache.beam.sdk.io.gcp/GcpApiSurfaceTest/testGcpApiSurface/

let me know if this is ready for review or not

retest this please

retest this please

@pabloem sorry for the rage commits today.
This is now ready for review.

Open Questions:

Should we remove adaptive throttling?

Seems that we're using retries in the client request initializer and right now a "bad record" will slow down the Read / Write (even though the error has nothing to do with the HL7v2 store being overwhelmed). Originally we wanted to be safe with overwhelming QPS on the HL7v2 store in batch scenarios.

Should we add more to the HealthcareIOError?

Add (processing time) Timestamp?
Add a convenience DoFn HealthcareIOErrrorToTableRowFn to ease writing deadletter queue to BigQuery.

Would it me more useful to expose an error rate metric than an error count?

retest this please

retest this please

retest this please

retest this please

Future improvements:

Currently this uses the alpha API because the motivating use case of HL7v2 -> FHIR mapping which requires the schematizedData field (not yet available in beta API).
For backwards compatibility the ListHL7v2MessageIDs always uses Messages.List to get message IDs and relies on the the HL7v2IO.Read to fetch the message contents. This provides a consistent flow for real-time and batch and provides backwards compatibility. However, in the alpha API Messages.List returns the actual message contents (rather than just the message IDs). This leads to us reading the messages contents twice for alpha HL7v2 stores. I suggest we address optimizing this "double fetch" in a future PR as the alpha API stabilizes.

retest this please

Ok an updates here from an internal thread w/ API team.

Message.List returning message contents is available in beta API with the view parameter.
Schematized Data should be in next beta release roughly in ~2 weeks.
right now the sink is outputting schematized data json wrapped in "{data=<actual_valid_json>}"

In light of these I will do the following refactors:

how we batch read from to always avoid the double get. This will make it a completely parallel code path than the real-time path but I think that's ok.
refactor to use beta client library
I'll strip out that {data=} wrapper to make this easier for users.

retest this please

retest this please

Sorry for the delay. Please see my comments inline.

Open Questions:

Should we remove adaptive throttling?
I think it is fine to remove it since the API has quota enabled by default, and retry logic is in place.

Seems that we're using retries in the client request initializer and right now a "bad record" will slow down the Read / Write (even though the error has nothing to do with the HL7v2 store being overwhelmed). Originally we wanted to be safe with overwhelming QPS on the HL7v2 store in batch scenarios.

Should we add more to the HealthcareIOError?
The second point below would be very helpful, but I am fine with adding it to next PR (if that's easier).

Add (processing time) Timestamp?
Add a convenience DoFn HealthcareIOErrrorToTableRowFn to ease writing deadletter queue to BigQuery.

Would it be more useful to expose an error rate metric than an error count?
This functionality is probably already provided by services like stackdriver, we might want to wait until there is a concrete use case.

retest this please

I'm starting to take a good look at this now.

in case it's not obvious: feel free to ignore failures in unrelated test suites (e.g. communitymetrics)

Next Steps (based on offline feed):

[x] Improve API for users:

[x] Add static methods for common patterns with ListHL7v2Messages
[x] Add ValueProvider<String> support to ease use in the DataflowTemplates

[x] ListHL7v2Messages (hl7v2Store and filter)
[x] Write (hl7v2Store)

[ ] "standardize" integration tests

[ ] Remove hard coding of my HL7v2Store / project in integration tests.
[ ] Add HL7v2 store to Beam GCP IT project

@pabloem PTAL.
I've addressed all outstanding feedback and setup integration testing infrastructure
Note on integration tests:
Integration tests seem to be passing.
Created the following resources

gs://temp-storage-for-healthcare-io-tests

will house temp files for FHIR Import in upcoming FhirIO PR

projects/apache-beam-testing/locations/us-central1/datasets/apache-beam-integration-testing which houses ephemeral HL7v2 store per test class (and later ephemeral FHIR store per class.)

retest this please

retest this please

@pabloem
Questions:

How could / should we add this to this page for built in connectors? Are there other documentation things that I might have missed?
Should we file jira issues or buganizer  for future work, notably:

Using Batch import / export for read when that api is available
Updating to newer version of client library (coming soon, which will simplify the hack-y handling of schematized data in this PR)

What is the feasibility / process for back-porting this to prior beam releases for expediting use by users who want to use an already released version e.g. 2.19 ?

Run Java PostCommit

Run Java PostCommit

Java PostCommit is probably our slowest suite of all, but once it runs we should be good to go

3. What is the feasibility / process for back-porting this to prior beam releases for expediting use by users who want to use an already released version e.g. 2.19 ?

bumping this question.  would like to know timing on when I can see it in a released version... either backport, or a new one.

What is the feasibility / process for back-porting this to prior beam releases for expediting use by users who want to use an already released version e.g. 2.19 ?

bumping this question. would like to know timing on when I can see it in a released version... either backport, or a new one.

You would need to get support from the community to do the backport and release which seems unlikely based upon this being a new feature (vs a critical bug/security issue) and people are constrained time wise due to COVID-19.
Once this is merged it would be part of the subsequent release, the branch cut dates are here:  https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com (with 2.22.0 scheduled on May 20th)

retest this please

retest this please

@lukecwik I ran the linkeage check "as-is" (log) and IIUC this has no issues.
Is this acceptable or is it a hard requirement that I update google_clients_version?

@lukecwik I ran the linkeage check "as-is" (log) and IIUC this has no issues.
Is this acceptable or is it a hard requirement that I update google_clients_version?

This is a hard requirement. Either we are using the google_clients_version of shared deps or google_healthcare_clients_version for deps. We can't be using both.

Note that your dependency report seems to be showing that you should be able to update to the version needed by the healthcare lib.

@lukecwik that's what I thought to but when I do that I get the output in this comment

@lukecwik that's what I thought to but when I do that I get the output in this comment

A lot of google libraries have three parts to the version string (API major version, API minor version, shared lib version)
You have to find and upgrade the client libraries to versions that contain the 1.30.9 shared lib version like:
google-api-services-cloudresourcemanager:v1-rev20191206-1.30.4 becomes google-api-services-cloudresourcemanager:v1-rev20200311-1.30.9
Best way to find matching versions is to look at the index of the maven repo for each artifact (e.g.):
https://repo1.maven.org/maven2/com/google/apis/google-api-services-cloudresourcemanager/
Find ones that have the same API major version and any newer API minor version with the shard lib version being 1.30.9.
Major version changes are breaking API changes, minor version increases are meant to be safe to upgrade and shared lib version just needs to be the same across all versions.

Thanks for the pointer!
I will give that a try.

@lukecwik
some progress I was able to update storage / cloudresourcemanager that way.
However, in
https://repo1.maven.org/maven2/com/google/oauth-client/google-oauth-client/
https://repo1.maven.org/maven2/com/google/oauth-client/google-oauth-client-java6/
They stop at 1.30.6

@lukecwik would it be a reasonable compromise to introduce a google_oauth_clients_verion? If not I believe we are at an impass.
Reasoning / Research / Background:

features in healthcare API making this PR possible require >=1.30.9 (healthcare api absolute minimum version is 1.30.8)
all deps under com.google.apis are updatable to 1.30.9
the two deps under com.google.oauth-client have versions that stop at 1.30.6
I have a suspicion that this 1.30.9 for com.google.apis is not even really related to the 1.30.6 for com.google.oauth-clients but an unfortunate naming convention similarity

see that 1.30.6 oauth client updated fairly recently on 2020-03-05

I've prepared this in the latest commit to make this a concrete proposal to review / run pre/post commits on.

retest this please

retest this please

retest this please

now linkage checker is giving me no output:
jferriero@shadow-gallery:~/VersionControl/beam$ sdks/java/build-tools/beam-linkage-check.sh
Fri 10 Apr 2020 04:42:32 PM PDT: Installing artifacts of HL7v2IO(785a5937) to Maven local repository.
jferriero@shadow-gallery:~/VersionControl/beam$

Run Java PostCommit

Run Java PreCommit

Run Java PreCommit

Run Java PreCommit

Run Java PostCommit

Run Java PostCommit

retest this please

Run Java PostCommit

Run Java PostCommit

retest this please

retest this please

Run Java PostCommit

oops. looks like a merge issue
13:11:33 /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy: 386: The current scope already contains a variable of the name google_oauth_clients_version
13:11:33  @ line 386, column 9.
13:11:33        def google_oauth_clients_version = "1.30.6"
13:11:33            ^
13:11:33

retest this please

retest this please

Run Java PostCommit

Run Java PostCommit

this looks fine to me as long as the dependency changes look fine to @lukecwik

@lukecwik ptal so @jaketf won't have to rebase if changes look fine

@lukecwik please let me know if there's anything else I can do to give confidence in the dependency changes.

@lukecwik please let me know if there's anything else I can do to give confidence in the dependency changes.

I have a lot of reviews on my radar and will get to this one as I can. If you want, you can reach out to the community for another reviewer to double check the dependency work and/or have Pablo take it up.

@suztomo - Tomo are you able to review the dependency changes for this PR?

@suztomo would you be able take a look at the dependency changes? otherwise I will drop a note to the dev list.

retest this please

retest this please

retest this please

retest this please

retest this please

retest this please

retest this please

Run Java PostCommit

retest this please

retest this please

Run Java PreCommit

Run Java PostCommit

Yeah, this can be a performance bottleneck and this whole operation will be limited to a single machine. Usually sources need an additional level of parallelism due to being high fanout. BTW it might sense to add a Reshuffle at the end here just to allow any subsequent steps to parallelize.

@chamikaramj that sounds like a good idea. However, I get deprecation working on org.apache.beam.sdk.transforms.Reshuffle is there a new blessed way of doing this?

You can ignore the deprecation warning for Reshuffle.viaRandomKey(). The deprecation was just because the behavior across runners for Reshuffle is not well defined. Transform is not going away. Many runners add a fusion break after GBK. So this will allow subsequent steps to parallelize better.

You can also consider adding an option to not add the Reshuffle to avoid adding any additional shuffle for anyone who already will have a GBK downstream. See here: https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L823

@pabloem please do not retest this until I say so.
reshuffle is messing up something to do w/ coders in my ITs.
will investigate and let you know when it's safe to re run tests

@pabloem reshuffle added and ITs passing locally as of c50df5f

retest this please

retest this please

Run Java PostCommit

Run Java PostCommit

And liftoff

Seems like one of the new tests is flaky.
https://builds.apache.org/job/beam_PostCommit_Java/5947/
https://builds.apache.org/job/beam_PostCommit_Java/5943/
https://builds.apache.org/job/beam_PostCommit_Java/5942/

@chamikaramj looking into this now.
Struggling to reproduce locally.

authors: brianlucier chamikaramj jaketf lastomato lukecwik pabloem


text:
[BEAM-12246] Fix ib.collect(dataframe) indexing

Indexing wasn't being properly propagated. This also preserves any hierarchical indexing from the dataframes. There is one problem with passing dataframes, however, which is that the indexing isn't kept and is reset per bundle. This solves this by resetting the index once the DataFrame is collected but only if it is an unnamed index. This behavior can be overridden with the reset_unnamed_indexes=False flag.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunner compliance status (on master branch)

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---

---

---

Java

Python
---

---

---

XLang

---

---

Examples testing status on various runners

Lang
ULR
Dataflow
Flink
Samza
Spark
Twister2

Go
---
---
---
---
---
---
---

Java
---

---
---
---
---
---

Python
---
---
---
---
---
---
---

XLang
---
---
---
---
---
---
---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go
Java
Python

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Portable_Python PreCommit

Added TODOs, thanks Brian!