text:
ANS-006-98 - C2023003490

Ressources de conformité:

ajout des descriptions des différents éléments
Ressource Task

correction de l'élément text de la slice accueilSéquentiel : acceuilSequentiel -> accueilSequentiel
Ajout d'une restriction sur les codes à utiliser pour la slice ModePriseCharge
Corrections de plusieurs URL
Modification de la slice typeDroitPrestation
modification des types valueID en valueIdentifier pour pouvoir prendre en charge les caractères spéciaux
correction des formats des éléments "min" : suppression des guillemets autours de la valeur

Ressource Consent

correction du type de reference
suppression de l'URL de la ressource SDOTask dans l'élément provision.data.reference. Remplacé par un type générique DocumentReference

Ressource SDODocumentReference

ajout d'un warning indiquant que l'élément data est obligatoire pour le flux 1.4 (hors Bundle)
title passé en obligatoire

Ressource ESMSDocumentReference

Ajout d'une contrainte sur la valeur du title

Exemples FHIR

Suppresion des références "author" non utilisée dans les ressources DocumentReference
Suppresson des éléments "type.coding" pour chaque slice dans la ressource SDOTask
Mise à jour des noms des id des exemples pour un affichage plus explicite
Suppression des exemples non passants
report des évolutions de spécifications
Mise à jour des title de des ressources DocumentReference et Bundle liés
Mise à jour des codes et display pour temporaliteAccueil des ressource Task et Bundle associé

Exemples CDA

Suppression des exemples non passants
report des évolutions de spécifications

Code source

Annexes déplacées dans un onlget individuel
index : ajout d'une description de la partie CDA
Ajustement à la taille de l'écran du diagramme de séquence de bout en bout
correction des liens vers les zip de la page téléchargement + suppression des structures et exemples turtle
suppression de l'onglet tests
index : mise à jour du tableau ressources profilées
index : nettoyage du code et suppression des doublons
suppression des mentions du flux fonctionnel 6 et renommage du flux 5
ajout des dépendances NOS
index + liste_profils : ajout d'information concernant l'usage de chacune des ressources de conformité en fonction des volets
Suppression des fichiers dépréciés

Spécifications fonctionnelles

Mise à jour du pdf

Spécifications techniques CDA

ajout de la partie CDA en format tableau dans l'IG. Formatage à venir
Remplacement de la TRE civilite par le JDV_J245-Civilite-CISIS
Remplacement la TRE_R249 par le JDV_J143 CI-SIS
Modification de la gestion de l'id du document ClinicalDocument.id
Ajout des éléments de versionning dans le document CDA suivant les préconisations CI-SIS : versionId, setId, relatedDocument

Spécifications techniques FHIR

correction des requêtes des flux 1.3 et 5.3 (id n'est pas un paramètre)
Suppression des flux 5.3 et 5.4
Suppression du paramètre _elements dans le flux 5.1
ajou d'une limite temporelle sur le _lastUpdated des flux 1.1 et 5.1

Annexes

Mise à jour du diagramme de séquence de bout en bout
Mise à jour du fichier des nomenclatures

Il y a des choses à corriger dans le Q/A : https://ansforge.github.io/IG-fhir-medicosocial-suivi-decisions-orientation/ig/presta-98/qa.html
Mettre le package NOS dans les dépendances

C'est vraiment compliqué de review les json et les grosses PR comme ça @SEBELIN est-il prévu de passer à FSH ?
Côté Kereval, c'est une très mauvaise pratique de faire une PR de ce type, associée à une prestation :

On ne sait pas ce qui est fait (aucune description, le titre n'est pas clair)
Très difficile à review
In fine, on obtient un guide qui n'est pas très qualitatif

@nriss , j'ai ouvert une issue pour le passage à FSH : #67.
Nous n'avons jamais eu de prestation dédiée au passage en IG. L'IG a été créé principalement pour utiliser matchbox. Il reste un gros travail de mise en forme et mise à niveau pour en faire un support complet et facile à maintenir. Sujet partagé avec @SEBELIN .
Pour information, le titre correspond au nom de la prestation et du bon de commande ANS. Le besoin est donc borné par la propale.
Nous sommes encore en phase d'ajustement pour la gestion des PR. Je plussoie, il y a une marge d'amélioration

C'est vraiment compliqué de review les json et les grosses PR comme ça @SEBELIN est-il prévu de passer à FSH ? Plus on fait le passage à FSH tard, plus ça sera coûteux
Côté Kereval, c'est une très mauvaise pratique de faire une PR de ce type, associée à une prestation :

On ne sait pas ce qui est fait (aucune description, le titre n'est pas clair)
Très difficile à review
In fine, on obtient un guide qui n'est pas très qualitatif, les validations n'étant même pas faites : https://github.com/ansforge/IG-fhir-medicosocial-suivi-decisions-orientation/pull/49/checks

@nriss , vu ensemble, les vérifications sont bien faites. Les warning restant ont trait à un dossier FSH non existant puisque nous n'avons pas de ressources au format FSH. Devrait être traité lors du passage en FSH => issue #67

Il y a des choses à corriger dans le Q/A : https://ansforge.github.io/IG-fhir-medicosocial-suivi-decisions-orientation/ig/presta-98/qa.html Mettre le package NOS dans les dépendances

Tracé dans l'issue #100 et en partie traitée via les PR #80 , #94 et #87

Hello, merci pour la PR :)
Je vois encore deux choses à faire:

Editer le message de la PR pour expliquer tous les travaux menés
Résoudre les conflits de la PR

Historique ajouté, conflit résolu, dernières corrections ajoutées et validées. OK pour merge @nriss si OK pour toi

authors: TristanKEREVAL jguihot nriss


text:
No libselinux package found when using Python 3.8 (needed by Ansible Core 2.12)

I'm using AlmaLinux 8.6 and ansible-bender 0.9.0. If I install ansible it needs python 3.8 since it provides ansible-core-2.12.2. But there is no python38-libsselinux package for python 3.8, an according to this bugreport there never will be? ( https://bugzilla.redhat.com/show_bug.cgi?id=1932650 )
However I was not able to run ansible-bender since it is checking for selinux (_selinux) available
There was an error during execution:
This system is using selinux(8) and selinux python module is not installed. There is a known issue in ansible that it freezes in this setup:
  https://bugzilla.redhat.com/show_bug.cgi?id=1696706
Please install libselinux python bindings (on Fedora the package name is python3-libselinux).

So I just modified the function _check_selinux_iz_gud to just return. It seems to work. However I dont't know if it was just luck due to my minimalistic playbook.

I ran into this as well on fedora. My workaround was installing the selinux shim via pip/pypi in my virtualenv.
@TomasTomecek - in the bug report linked to in the error message shown, you advocated closing the bug report because, Screw these heisenbugs: I can't reproduce the freeze anymore: tried a clean VM and my dirty workstation and the freeze happened on neither. :)
I've opened #285 to address this.

source: gharchive/issue

authors: TheMasterFX quietjoy

repo: ansible-community/ansible-bender

license: MIT

license_type: permissive

license_source: github-api


text:
Amw 103

Issue: https://issues.redhat.com/browse/AMW-103

@rpelisse Can you please review the PR

source: gharchive/pull-request


text:
Memsource Credentials issue

Memsource credentials passed from variable files or externally from playbook (using command line or extra_vars file) is not working.

The issue has been fixed as per 6b3b7fee7b9702893f083c1f722ea5729bd8fc78

repo: ansible/ansible-collection-memsource


text:
Move pull_policy to pull.policy

This moves pull policy under the pull key in the settings file.
This will result in the next release being a major as it is a breaking change.
Fixes #951

ansible-language-server followed the similar for pull-policy as that of ansible-navigator.
https://github.com/ansible/ansible-language-server#language-server-settings
Should we do a similar change in ALS as well? It will require a major version change for both ALS and extension.

ansible-language-server followed the similar for pull-policy as that of ansible-navigator. https://github.com/ansible/ansible-language-server#language-server-settings Should we do a similar change in ALS as well? It will require a major version change for both ALS and extension.

I'd say if we were going to do it, we should do it fast before there is broad adoption of using EEs with both.  I'm guessing it is still low.
Maybe we could time it with a navigator release and the lint 6 release?

authors: cidrblock ganeshrn

repo: ansible/ansible-navigator


text:
Add PKGMGR_OPTS support

This allows users to pass flags to our package manager.
Signed-off-by: Paul Belanger pabelanger@redhat.com

Pull-request updated, HEAD is now aba3503d54f363240ff93ecaa8720f8f3472a113

repo: ansible/python-builder-image


text:
Images in the private mode can still be accessed when not logged in

私有状态下图片仍然可以在非登录状态下访问。
设置成必须登录才能查看后，图片仍然可以直接访问。

这里带来一个问题，如果图片存在第三方 OSS，该如何验证权限呢？
Here is a question: how to verify the permission of images if they are stored in OSS (Object Storage Service)?

参考minio和七牛云又拍云等的方案，通常是向oss服务申请一个key ，再通过303重定向到真实的带着key的资源链接实现的。
但这会带来一系列的问题，比如问题内容发布修改时的逻辑。 以及图片资源需要归档到数据库等。
具体参考文档  https://help.upyun.com/knowledge-base/cdn-token-limite/
（通常客户端会这么干 ，不知道老哥有没有什么骚操作）
Referring to solutions such as Minio and Qiniuyun Youpaiyun, it is usually achieved by applying for a key from the OSS service and redirecting it to a real resource link with the key through 303.
But this will bring a series of problems, such as the logic when publishing and modifying the problem content. And image resources need to be archived in the database.
Specific reference documents https://help.upyun.com/knowledge-base/cdn-token-limite/

This sounds like an interesting problem. I'd like to help out but since I'm very new and unfamiliar with the App, can someone provide a "steps to replicate" instruction so I can get started quickly?

Thanks @LinkinStars , I'm able to run the app in my local but was wondering if there's documentation/steps on how to replicate this exact scenario or the url/route of this specific feature?
If not, I will just explore and see if I can find it 👌

Got it. I think I will just look for other issues that I can help out then.
I'd like to help with the discussion but I'm not sure exactly what the bug is. From my understanding, the reporter has set their uploaded image to "private" but it's still viewable for unauthenticated users.

现在有一个新问题。当设置为私有后，登录用户后，用户上传的图片无法预览。

对不起，让我确认一下。因此，请清除浏览器缓存，注销，然后重新登录并刷新页面。

这个问题还是存在，是否和安装的版本有关？ 我这边是使用最新的docker镜像进行安装。

是直接使用ip访问的。
1Panel-answer-h3Zj-20231120163702.log

Fixed in v1.2.0

authors: AionKwong carlqt fenbox zhenruyan

repo: answerdev/answer


text:
Geometry Visualization Issue

🔍 Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

🐞 Description of the bug
Couldn't able to visualize the geometry, standalone trame, PyVista trame or provided TrameVisualizer Class,

from ansys.geometry.core.plotting.trame_gui import TrameVisualizer
view = TrameVisualizer()
view.set_scene(plotter=plotter)
view.show()

CRITICAL -  -  logger - handle_exception - Uncaught exception
Traceback (most recent call last):
  File "d:\Research\pygeom_plotter_test\geom_plotter.py", line 86, in <module>
    view.set_scene(plotter=plotter)
  File "C:\Users\username\.ansys_python_venvs\.venv_pyansys\lib\site-packages\ansys\geometry\core\plotting\trame_gui.py", line 64, in set_scene
    view = plotter_ui(plotter.scene)
AttributeError: 'PlotterHelper' object has no attribute 'scene'

📝 Steps to reproduce
Pass the plotter object to respective implementation, e.g.,
TrameVisualizer.set_scene(plotter)

💻 Which operating system are you using?
Windows
📀 Which ANSYS version are you using?
Ansys 24R1
🐍 Which Python version are you using?
3.12
📦 Installed packages
Installed all the pyansys meta-pacakages using Ansys Python Manager.

Additionally install following packages.
trame trame-vuetify trame-vtk

Hi @abhishekchitwar - the way to use the trame visualizer is not this one.
Please do as follows

import ansys.geometry.core as pygeo

pygeo.USE_TRAME = True

# Do your stuff...
#
#

design.plot()

Our PyAnsys Geometry main objects (design, component, body) all have a "plot()" method you can use easily this way with your intended backend

Have a look at https://geometry.docs.pyansys.com/version/stable/examples/01_getting_started/05_plotter_picker.html as well

Hi @RobPasMue
I attempted the following code snippet, but the design plot still appears in a desktop window. I'm looking for a way to display this design on a server or local URL instead.
from ansys.geometry.core import launch_modeler
from ansys.geometry.core.misc import UNITS, Distance, DEFAULT_UNITS
from ansys.geometry.core.math import Point2D
from ansys.geometry.core.sketch import Sketch
from ansys.geometry.core.plotting import PlotterHelper
import pyvista as pv
#from trame_app import TrameApp
DEFAULT_UNITS.LENGTH = UNITS.mm
modeler = launch_modeler(mode='spaceclaim')

launch_modeler.USE_TRAME=True

base_sketch = Sketch()
base_sketch.segment(Point2D([0,0]), Point2D([0,10]))
base_sketch.segment(Point2D([0,10]), Point2D([10,10]))
base_sketch.segment(Point2D([10,10]), Point2D([10,0]))
base_sketch.segment(Point2D([10,0]), Point2D([0,0]))

# Start by creating the Design
design = modeler.create_design("ModelingDemo")

# Create a body directly on the design by extruding the sketch
body = design.extrude_sketch(
    name="Design_Body", sketch=base_sketch, distance=Distance(10, unit=UNITS.mm)
)

# design plot
design.plot()

# plotter, this is for test
# plot_list = [body]
# plotter = PlotterHelper(use_trame=True)
# plotter.plot(plot_list)

Hi @RobPasMue,
I just updated the argument with design.plot(use_trame= True), getting following issue.
    raise TypeError(
TypeError: Server using client_type='vue3' while we expect 'vue2'

It seems, it requires update from TrameVisualizer.

Hi @RobPasMue,
I just updated the self.server = get_server(client_type='vue2') with TrameVisualizer, it worked as updated. Thanks!
Thanks!

Oh interesting! So you needed to pass the client type to the TrameVisualizer.. We might have to expose this somehow. Could you share your final script? I guess you had to modify your local version of PyAnsys Geometry, right?

Hi @RobPasMue,
yeah, I modified the local pygeometry installation.
It would be great if TrameVisualizer exposes several methods in context of remote visualization, e.g.
    def get_url(self):
        return f"http://{self.address}:{self.port}"

Ease for developer to configure address and port as per requirement.
Here is the final code! Example test case developed by @agvarghe
from ansys.geometry.core import launch_modeler
from ansys.geometry.core.misc import UNITS, Distance, DEFAULT_UNITS
from ansys.geometry.core.math import Point2D
from ansys.geometry.core.sketch import Sketch
from ansys.geometry.core.plotting import PlotterHelper
import pyvista as pv

DEFAULT_UNITS.LENGTH = UNITS.mm
modeler = launch_modeler(mode='spaceclaim')

# launch_modeler.USE_TRAME=True

base_sketch = Sketch()
base_sketch.segment(Point2D([0,0]), Point2D([0,10]))
base_sketch.segment(Point2D([0,10]), Point2D([10,10]))
base_sketch.segment(Point2D([10,10]), Point2D([10,0]))
base_sketch.segment(Point2D([10,0]), Point2D([0,0]))

# Start by creating the Design
design = modeler.create_design("ModelingDemo")

# Create a body directly on the design by extruding the sketch
body = design.extrude_sketch(
    name="Design_Body", sketch=base_sketch, distance=Distance(10, unit=UNITS.mm)
)

# design plot
design.plot(use_trame=True)

Thanks!

We will delegate the actual remote visualization to a different project we are working on. On this one we will just fix the inputs needed to use vue3 or vue2 =)
This issue will be closed by #1148 - thanks @AlejandroFernandezLuces

authors: RobPasMue abhishekchitwar

repo: ansys/pyansys-geometry


text:
Implement a function which helps debugging

Somehting like:
debug_pymapdl checks

It should use the CLI.

It should check the MAPDL starting point...

I think we should just use the report? And append things there...

I think the whole CLI could be revamped:
Launcher
Replace launch_mapdl https://mapdl.docs.pyansys.com/version/dev/user_guide/cli.html#pymapdl-command-line-interface with

pymapdl launch
pymapdl stop

converter
https://mapdl.docs.pyansys.com/version/dev/user_guide/convert.html

pymapdl convert

for debugging
we could have something like:

pymapdl debug should print the report?

Help

pymapdl help open the website for help and/or print useful information to the stdout.

repo: ansys/pymapdl


text:
form 中使用picker问题，如果data的数据通过远程获取并setData后，form表单所有验证失效问题

你好，我在form中使用picker ,但我的picker 的data数据源来自远程服务器中，在page的onshow钩子函数中远程获取数据。并通过setData设置picker的data数据，数据能够设置成功，并能界面也正常显示，但form中的所有控件验证都失效，这是什么原因？

有没有微信或者qq群问题讨论群？

我们正在对form进行全面重构，1.0版本预发下个月会发布

https://github.com/ant-design/ant-design-mini/issues/305

使用1.0没这个问题

repo: ant-design/ant-design-mini


text:
Write inside <# #> will  cause extra #>

When your writing comments inside <# #> and pressing enter it will add a extra #> by it self.

Duplicate of #35.

authors: ForNeVeR rstolpe

repo: ant-druha/intellij-powershell


text:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]

Describe the bug
failed to load config from ...\vite.config.ts
error when starting dev server:
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main defined in ...\node_modules@yankeeinlondon\happy-wrapper\package.json
help！thanks！
Reproduction
npm
System Info
system

Used Package Manager
npm
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Check that this is a concrete bug. For Q&A, please open a GitHub Discussion instead.
[X] The provided reproduction is a minimal reproducible of the bug.

我在我的package.json文件当中加入"type": "module",项目正常启动了，可能依赖包抛弃了以前的模式。
我用的是node 16 的版本
npm 8+

谢谢！

Just in case you couldn't understand our Chinese friend, here's a translation of his message:

I added "type": "module" to my package.json file, and the project started normally, maybe the dependent package abandoned the previous mode.
I am using the node 16 version

Having the same problem in a brand-new Vite/Vue project.
Any plans to fix this? Seems to make the plugin unusable with the default Vite setup. Adding "type": "module" to package.json doesn't work for me (and even if it did, I wouldn't see this as a long-term solution).

Another way.happy-wrapper is a pue esm package. So we can use dynamic import load this plugin.
Here is example:

import { defineConfig } from 'vite';

export default (async () => {
  const Markdown = (await import('vite-plugin-md')).default;

  return defineConfig({
     plugins:[others plugin,Markdown(...yourOption)]
  })

})();

我vite-plugin-md版本降到0.10.0版本就可以了

Translation:
My vite-plugin-md version just needs to be reduced to version 0.10.0.
I really feel an obligation to do this.

authors: MaGaroo XiaoYang943 biancangming callmesoul drbarto nonzzz

repo: antfu/vite-plugin-md


text:
added CORS to the API Server and WASM Server

Added CORS Support for WASM and API Server
This PR introduces Cross-Origin Resource Sharing (CORS) support for both the WebAssembly (WASM) module and the API server.
Changes:

Added CORS middleware to the API server: The server.go file now includes a middleware function that sets appropriate CORS headers in the HTTP response. This allows client-side web applications to interact with the API server.

Added CORS middleware to the WASM server: The wasm.go file now includes a middleware function that sets appropriate CORS headers in the HTTP response. This allows client-side web applications to interact with the WASM server.

Sorry for all the PRs. I'm relatively new to projects where multiple people make changes. But this should be the last one... at least for this function.

repo: anthdm/ffaas


text:
Connecting to MechanicalTurk

Hi,
How do we connect this to mechanical turk? Is there some missing documentation?
Here's what we've done so far:

Created an AWS account
Created an AWS user with full turk access only
Created environment variables with the AWS_ID and AWS_KEY set
Created a fullvideotask in the admin site: /admin/mturk/fullvideotask/

I can see that the Task class in mturk/models.py actually calls publish and creates the HIT with AWS, but I can't see where the publish method is invoked.
Can you please provide some more instuctions?
Thanks for your help.

As of right now the task of publishing HIT's is done manually via Django shell. To run django shell, run python manage.py shell, then import fullvideotask(fvt), query the fvt model that you want, and run its publish method. Similarly you can also use the class function, bath_create_and_publish() to create multiple task objects and publish them all in one go. Admin dashboard HIT status/management functionality is in the works but feel free to build one!

Thanks for this - that worked. For those people who aren't that familiar with Django shell (like me) you'll need to do this:
python manage.py shell < script.py
where script.py is a script something along the lines of:
`from mturk.models import FullVideoTask
tasks = [x for x in FullVideoTask.objects.all() if x.hit_id != ""]
for task in tasks :
task.publish()
`

@alex-birch take a look at the new branch, mturk. Its a dashboard like functionality for users to publish tasks as HITs

Hi @clayshieh , where can I find the dashboard to publish HIT tasks?

There's a dropdown in /admin under Video objects to create the task and publish it in one step. I forget if there's another dashboard that just publishes HITs

@hocherie to expand on that, go to /admin, click Videos, select the check boxes next to the videos that you want to publish, and in the action menu drop down select "Publish to Turk" and then press the Go button.

@clayshieh @antingshen when selecting "Publish to Turk", I get the error:

Request Method: | POST
-- | --
http://ec2-35-182-160-204.ca-central-1.compute.amazonaws.com:5000/admin/annotator/video/
1.10
CommunicationError
XML malformed
./mturk/mturk_api.py in validate, line 315
/home/ubuntu/beaverdam/venv/bin/uwsgi
3.5.2
['.',  '',  '/home/ubuntu/beaverdam/venv/lib/python35.zip',  '/home/ubuntu/beaverdam/venv/lib/python3.5',  '/home/ubuntu/beaverdam/venv/lib/python3.5/plat-x86_64-linux-gnu',  '/home/ubuntu/beaverdam/venv/lib/python3.5/lib-dynload',  '/usr/lib/python3.5',  '/usr/lib/python3.5/plat-x86_64-linux-gnu',  '/home/ubuntu/beaverdam/venv/lib/python3.5/site-packages']
Thu, 1 Nov 2018 13:13:50 -0700

I have set the AWS key and ID via:
export AWS_KEY="key"
export AWS_ID="id"
and have tried this with both sandbox set to true and false
Thanks!

It's hard to say. There's many reasons why AWS could be rejecting your request and they don't give you a clear error. See https://github.com/cvondrick/vatic/issues/12

In particular, the error I am seeing is:
<CreateHITResponse><OperationRequest><RequestId>1817f6d5-518e-4acb-830e-7d4111e7f4b6</RequestId><Errors><Error>AWS.NotAuthorized<Message>The identity contained in the request is not authorized to use this AWSAccessKeyId (1541519274636)</Message></Error></Errors></OperationRequest></CreateHITResponse>'

I am exporting the correct key, ID, have all appropriate permissions for my IAM, and have tried placing the key and ID in my bashrc as well to no avail. At one point, I put in the wrong key and ID and changed it back to the correct one and it started being able to post videos to sandbox, but with sandbox set False, I am running into this issue.

Another note for future users, when BeaverDam was made, boto didn't support mturk in Python 3, so BeaverDam makes custom requests. Now, using boto3 might provide additional functionality and is the official way to connect to mturk.

@antingshen thanks for your note. Before I consider refactoring the code to use boto3 to connect, I have a follow up question:
As aforementioned, when trying to "Publish to Turk", when I have this config:
web client <-> nginx <-> socket <-> uWSGI <-> beaverdam

Along with the DEBUG=True error I showed above about XML malformed, I get the following error in my uwsgi logs when running sudo uwsgi --ini /etc/uwsgi/sites/beaverdam.ini:
'<?xml version="1.0"?>\n<CreateHITResponse><OperationRequest><RequestId>b17faffb-544c-44d4-a946-c0278f627623</RequestId><Errors><Error>AWS.NotAuthorized<Message>The identity contained in the request is not authorized to use this AWSAccessKeyId (1542143607840)</Message></Error></Errors></OperationRequest></CreateHITResponse>'
HTTP Response = b'<?xml version="1.0"?>\n<CreateHITResponse><OperationRequest><RequestId>b17faffb-544c-44d4-a946-c0278f627623</RequestId><Errors><Error>AWS.NotAuthorized<Message>The identity contained in the request is not authorized to use this AWSAccessKeyId (1542143607840)</Message></Error></Errors></OperationRequest></CreateHITResponse>'

However, publishing works fine and the AWS key and ID are accepted when going directly from browser to uwsgi, that is:
web client <-> uWSGI <-> beaverdam

Any help would be great. Thanks.

Hmm I haven't seen that error before. Did you check what AWS says that error means?

authors: alex-birch antingshen clayshieh hocherie uzbhutta

repo: antingshen/BeaverDam

license_source: bigquery


text:
added dragging shortcuts #44

Sean and I added the new shortcuts for dragging. I also had to reassign some shortcuts for rewinding, forwarding, and deleting. Wasn't too sure what the new assignments should be though.

Good initial choices!

I would avoid using arrow keys actually

People use arrow keys to scroll the page (even though our page ideally doesn't need to scroll, people will get it confused either when they either use our app or another app where they need to scroll.
The arrow keys are used to edit text in the "time" box.

I think we should use the comma and period keys for forward and rewind because the keys have < and > on them, which look like forward and rewind. We can have / be the normal play/pause button bc it's right next to them.
Now that Chrome is not longer using Backspace for "back", we can use the backspace key for deleting keyframes.

Well, my rational for arrow keys comes from most video sites (i.e. youtube and etc.), so I don't really think that should be confusing. Not quite sure what you mean by editing text in the time box since if you edit it, it'd change the time.

I do like the idea of < and > though, Gabe an I were actually discussing that earlier.

I guess backspace is fine then if we only the more recent version of Chrome. However, one thing that I was looking to ask was should we remove ECMA6 destructuring in place of normal extraction? (i.e. getting rid of {...} = ...). The reason I say this is because that would open up Beaverdam to more people with older browsers (ones that support ECMA5). (In fact even my work laptop didn't have chrome updated in a while which is where I first came across the browser necessity).

Arrow keys are used for most video sites (i.e. youtube and etc.), so I don't really think that should be confusing (or at least instead of play/ rewind, we skip a few frames in either direction).

This is true. However, I actually dislike the YouTube shortcut because sometimes when I'm trying to scroll the page or do something else I accidentally move the playhead for this exact reason.

Not quite sure what you mean by editing text in the time box since if you edit it, it'd change the time.

I meant if you are trying to type in the text box and realize you need to move the cursor one char to the left, you press the left key and oops, you accidentally rewound your video.

I guess backspace is fine then if we only the more recent version of Chrome. However, one thing that I was looking to ask was should we remove ECMA6 destructuring in place of normal extraction? (i.e. getting rid of {...} = ...). The reason I say this is because that would open up Beaverdam to more people with older browsers (ones that support ECMA5). (In fact even my work laptop didn't have chrome updated in a while which is where I first came across the browser necessity).

That's an easy one. When we want to support older browsers, we'll use Babel to convert it to ES5 automatically. We use jQuery and Raphael instead of pure JS for similar reasons – they also act as compatibility layers.

authors: GabeArr allenywang szhu


text:
Cannot paste multiple lines into the example program.

Highlight and copy the 10 lines below:
This is line 1.
This is line 2.
This is line 3.
This is line 4.
This is line 5.
This is line 6.
This is line 7.
This is line 8.
This is line 9.
This is line 10.
Now run the example program and paste these10  lines into it.  This is what you get:
$ ./linenoise_example
hello> This is line 1.
echo: 'This is line 1.'
hello>
Only the first line is received.  Lines 2 - 9 are ignored.  Each line should be printed with a prompt and a response.  Pasting short code snippets into a REPL is a very common thing to do.

I think I found the explanation for this problem in the linux termios man page where it explains the TCSAFLUSH flag to tcsetattr:

      the change occurs after all output written to the object

          referred by fd has been transmitted, and all input that
          has been received but not read will be discarded before
          the change is made.

That flag is used in disableRawMode.  So everything in the input buffer after the first newline gets discarded when disableRawMode is called after processing the first line.
The fix for this appears to be to use TCSADRAIN instead of TCSAFLUSH.  That drains the output but does not discard the input.

repo: antirez/linenoise


text:
Labels in argument lists

When formatting a label in an argument list, the label is attached to the comma following the previous argument or open brace, e.g.,
#set heading(numbering: "1.")

= Heading <label>

#ref( <label>
  ,
)

Instead, I would expect labels to be treated like other values in argument lists:
#set heading(numbering: "1.")

= Heading <label>

#ref(
  <label>,
)

With b8870c069ee1c5b579a08e40ac4dac1c47915b4b the special handling for labels is only used in markup mode. If some issues persist, please reopen the issue.

authors: antonWetzel m-haug

repo: antonWetzel/prettypst


text:
Analyse --mode reads error- missing script

Hello,
I am able to run haystac analyse --mode abundances, but haystac analyse --mode reads fails. I've included the snakemake log. My goal is to recover the reads that are being assigned to each category, i.e. dark and grey matter, but it looks like this code may have been refactored.
haystaclog.txt
I installed haystac through conda. Thank you for your help and please let me know if I can provide more information.
Best,
Peter

Hello Peter,
I hope you are doing great and apologies for not getting back to you earlier !
Thank you for using haystac and for raising this issue.
Indeed the code for haystac analyse --mode reads has been recently refactored and optimised as a new version will be released soon on conda.
I went through the log file and I was wondering if you could upload the likelihood matrix that haystac analyse --mode abundances has produced. From your log file it should be here /local/workdir/pk445/haystac/both_notrim_reads_.05/probabilities/both_sample/both_sample_likelihood_ts_tv_matrix.csv.
I will make sure I get back to you ASAP.
Thank you for your help and patience !
Best,
Antony

Hello Antony,
Thanks for the response. In the meantime, is there a workaround for using this feature?

repo: antonisdim/haystac


text:
Creative Development

Quiz using app lab in code.org: https://studio.code.org/projects/applab/jau_YPo5u3K2kjagwJ9Ow50YORprzrWIiwIWRQOfrSw
Quiz plan post on my fast page: https://antonyrc6.github.io/Antony-s-fast-page/markdown/2022/09/11/quiz-plan.html

I bet Anthony deserves a 2.8. His quiz is completely organized. He even put up notifications to figure out if an answer is right or wrong. When the quiz was finished, the message said that all answers were correct. He did really well on the College Board Task as well.


text:
Add logs for the kind tests in jenkins

Closes: #6538

To prevent log files from piling up and consuming too much storage, you can implement a cleanup strategy. This can be done by periodically deleting old log files or by keeping only a limited number of the most recent log files.

To prevent log files from piling up and consuming too much storage, you can implement a cleanup strategy. This can be done by periodically deleting old log files or by keeping only a limited number of the most recent log files.

Files in the job's workspace will be cleaned at the beginning of the job. The retention period for job history (including saved log files) is configured by Jenkins, so there is no need to build another cleanup script.

@XinShuYang could you take another look and let me know if I can merge this?

@XinShuYang could you take another look and let me know if I can merge this?

Yes we can merge it.

/test-kind-all

/test-kind-all

it will trigger only ipv4 jobs, lemme confirm all other ipv6 and dual jobs once.

/test-kind-ipv6-conformance

/test-kind-ipv6-conformance

/test-kind-ipv6-all

/test-kind-ipv6-all

authors: KMAnju-2021 XinShuYang antoninbas rajnkamr

repo: antrea-io/antrea


text:
Obj file parser

Do we actually even really need our own file format?
Is there any additional information we put inside the *.csl files, or could we theoretically just write an *.obj parser and not have to use a custom blender plugin?

Yes we really do need our own file format.

.obj is a generic format that supports many things we do not support (for example quads instead of only triangles like .csl. Quads and NGons are made into triangles inside the Blender exporter automatically) Do you want to handle all of that inside the Interpreter with much higher performance requirements as well?

We would not have flexibility in how we store scenes, materials, shaders and textures.

How would we store animations  and rigging.., or properties for the physics engine..., or .obj files linking to other .obj files to make open-world loading straightforward for example? I could list more but this is just off the top of my head. Or even packing textures inside the .csl. The possibilities are endless with our own format.

One would have to translate all the different .obj types to our own engine structure anyways, so why not do it through Blender and let Blender handle all the complexity of different file formats and have the exporter work for all of them? I have successfully imported a random .obj model from the internet into Blender and after a few adjustments of materials exported it to .csl and displayed it correctly in the renderer. Its name is some\_car.csl and can be found in the assets.

Every game engine has its own format and for good reason, why not we? :) The closest to a generic 3D game engine format would be COLLADA, but we would still lose flexibility and functionality and we cannot expect to support everything in COLLADA. Blender can still import COLLADA.

Assuming we adopted for example COLLADA, which would in theory be a better fit. What would we do with .obj files? Import them into Blender, export them into COLLADA and interpret them into our own structure? Adopt the COLLADA structure? Write yet another interpreter for .obj? For .fbx as well?

Blender supports custom object properties. We could use those to store various parameters (for example physics, storage or open-world related)  and have them picked up by the exporter.

You could still make a limited direct .obj to .csl translator if you like, but i personally would not consider that high priority.

Compressed binary format when/how?

Summary: It is IMHO much simpler to write ONE exporter and ONE fast parser and interpreter for a flexible file format suited to our engine capabilities than limiting ourselves to any specific generic asset format. We would need to convert to our internal format anyways unless we completely rewrite everything yet again.
Do you need any more arguments or are these enough?

Ok ok, I get it. No obj parser :D

authors: antsouchlos philsegeler


text:
ScrollBar  横向滚动到最后

[ ] I have searched the issues of this repository and believe that this is not a duplicate.

What problem does this feature solve?
ScrollBar  横向滚动无法初始化的时候滚动到最后。
What does the proposed API look like?
boolean  开启是否滚动到最后

可以用 4.x， 设置下 range 就可以了
https://f2.antv.vision/zh/examples/line/line#pan
<ScrollBar mode="x" range={[0.8, 1]} />

感谢，可以啦

authors: xienas zengyue


text:
「wip」feat(transform): add exponential smoothing data transform methods

Checklist

[ ] npm test passes
[ ] benchmarks are included
[ ] commit message follows commit guidelines
[ ] documents are updated

Description of change
add exponential smoothing data transform methods

这个算法有来源出处吗？以及使用场景？

这个算法有来源出处吗？以及使用场景？

好的，晚点补一下

authors: hustcc lulusir


text:
当渲染的shape为图片的时候，一直在请求拿图片数据。特别消耗浏览器性能

G6 Version: 3.2.3

Platform: Chrome86.0.4240.111（正式版本） (x86_64)

Mini Showcase(like screenshots):

CodePen Link: https://codepen.io/fred_zhao/pen/abZExEx

做Promise处理是为了可以处理图片异常然后给到图片的默认值

应该是异常图片在渲染引擎内部没有处理。目前在给到 G6 图片之前，先检查一下异常图片吧

应该是异常图片在渲染引擎内部没有处理。目前在给到 G6 图片之前，先检查一下异常图片吧

。。。。我做了处理了，特意用promise处理了。。但是无限请求太可怕了，这个有解决方案么

应该是异常图片在渲染引擎内部没有处理。目前在给到 G6 图片之前，先检查一下异常图片吧

能处理下你们钉钉交流群的加群请求么？好久了都没有人通过

解决方案是在图片加载的时候处理一层，将图片转换成base64。然后再给到graph渲染。

应该是异常图片在渲染引擎内部没有处理。目前在给到 G6 图片之前，先检查一下异常图片吧

能处理下你们钉钉交流群的加群请求么？好久了都没有人通过

两个钉钉群都满了。。。。。有问题的话，建议还是在 issue 里提，我们看到的话会及时处理

解决方案是在图片加载的时候处理一层，将图片转换成base64。然后再给到graph渲染。

下个小版本我们内置处理掉

authors: Yanyan-Wang wustzhaohui


text:
Smooth animation for data updates

[x] I have searched the issues of this repository and believe that this is not a duplicate.

What problem does this feature solve?
Is there a possibility to make a smooth transition between data updates in graph so that there is no "hard cut" but instead an animation? This would make it much easier for users to follow and understand updates to a graph (e.g. when filtering the data)
What does the proposed API look like?
A new option in GraphOptions

It is already possible, animate: True and change the data using changeData


text:
增加 非鼠标事件触发节点或边 tooltip 的api ,

[ ] I have searched the issues of this repository and believe that this is not a duplicate.

What problem does this feature solve?
例如我在图之外给某个节点/边添加属性后， 在没有鼠标事件，需要用户在图上以tooltip 的形式感知一下，
What does the proposed API look like?
例如 graph.tooltipFoucs(node/nodeId, offset, )

这个内置 tooltip 不会考虑，可以自己加 DOM 来实现。或者模拟鼠标触发事件 graph.emit('node:mouseenter', {item, target});


text:
G6 5.0.0-beta.27版本tooltip使用预定义DOM元素无法正常显示

问题描述
G6 5.0.0-beta.27版本tooltip的getContent回调函数使用预定义的DOM元素作为返回值，导致tooltip无法正常显示，tooltip直接把图的画布给挤占了，而G6 4.8.23版本tooltip可以在正确的节点/边的位置上显示预定义的DOM元素，下面的图是不同版本下鼠标悬浮在节点上tooltip的显示情况：
5.0.0-beta.27版：

4.8.23版：

重现链接
5.0.0-beta.27版复现链接： https://codesandbox.io/p/sandbox/stupefied-feather-354gpk?file=%2Findex.ts%3A23%2C4 ### 4.8.23版复现链接： https://codesandbox.io/p/sandbox/vibrant-matan-5skrtw?file=%2Findex.js%3A74%2C20
重现步骤
如复现链接所示
预期行为
期望5.0.0-beta.27版tooltip可以使用预定义的DOM元素作为显示的元素
平台

操作系统: [Windows]
网页浏览器: [edge]
G6 版本: [ 5.0.0-beta.27]

屏幕截图或视频（可选）
No response
补充说明（可选）
No response

v5版本需要return outDiv.outerHTML

authors: l-besiege-l uhobnil


text:
为什么我使用g6的元素实例方法item.hide()就正常，item.changeVisibility()就报错呢？？

问题描述
为什么我使用g6的元素实例方法item.hide()就正常，item.changeVisibility()就报错呢？？
重现链接
.
重现步骤
.
预期行为
.
平台

操作系统: [macOS, Windows, Linux, React Native ...]
网页浏览器: [Google Chrome, Safari, Firefox]
G6 版本: [4.5.1 ... ]

屏幕截图或视频（可选）

补充说明（可选）
No response

从你的代码看，buttons和buttonParents似乎是图形（Shape），而不是元素（item，比如node，combo、edge）
操作图形显示/隐藏只能通过shape.hide(), shape.show()达成
参考图形（Shape）的Typescript type: IShape extends IElement, IElement存在hide/show定义，但没有changeVisibility


text:
🐛 大数据性能表现栈溢出 Maximum call stack size exceeded

🏷 Version

Package
Version

@antv/s2
@antv/s2-v2.0.0-next.27

@antv/s2-react

@antv/s2-vue

Sheet Type

[ ] PivotSheet
[x] TableSheet
[ ] GridAnalysisSheet
[ ] StrategySheet
[ ] EditableSheet

🖋 Description

官网“100w条数据性能表现-明细表” 示例，下滑过程中栈溢出，不到 1w条数据稳定复现  https://s2.antv.antgroup.com/zh/examples/case/performance-compare#table

⌨ Code Snapshots

🔗 Reproduce Link
https://s2.antv.antgroup.com/zh/examples/case/performance-compare#table

🤔 Steps to Reproduce

😊 Expected Behavior

😅 Current Behavior

💻 System information

Environment
Info

System
macos 14.3

Browser
chrome  129.0.6668.101（正式版本） (arm64)

已知问题, 等待底层渲染引擎解决中 https://github.com/antvis/G/issues/1712

https://github.com/antvis/S2/issues/2771


text:
Notification page added

#3861
New Notification Page added
Fixes: #3861
Description
New Notification page is added. That page will make very easy to navigate notifications , the page sort the notifications into 4 categories that is All, Unread, Read, Archived. that interface easy to use as well as to understand.
#3861
Type of PR

[x] Feature enhancement

Screenshots / videos (if applicable)

Checklist:

[x] I have made this change from my own.
[x] I have taken help from some online resources.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generate no new warnings.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers and screenshots after making the changes.

please record a video from home page like end to end @Pratikpawar13

are u working on this issue? gssoc ext is about to end.

work with updated branch and raise new PR @Pratikpawar13


text:
Managed::with operator closes resource to early

Need to pass both the current instance and the transforming function down the chain rather than calling map.
To replicate assert the session is still open in HibernateTest (in 10.0.3 this will fail)
@Test
    public void hibernate(){

        Try<String, Throwable> res = Managed.of(factory::openSession)
            .with(Session::beginTransaction)
            .map((session, tx) ->{
                try {
                    verify(session,never()).close();
                }catch(Exception e) {
                    e.printStackTrace();
                }

                return deleteFromMyTable(session)
                    .bipeek(success -> tx.commit(),error -> tx.rollback());

            } ).foldRun(Try::flatten);

        assertThat(res,equalTo(Try.success("deleted")));

    }

Merged

repo: aol/cyclops


text:
fixing scrolling issue mentioned in Issue #142

I approached the safer solution:

Made a border around the window screen to make it obvious that it's a screen

There were many solutions I could think of, but I approached the safest solution that taking into consideration the future development of that feature, I simply drew a border line around that screen to make it obvious to users that it's a screen, not a normal white space. Solutions I thought of doing:

I could have set the Height of the screen to auto and that should have solved the problem FOR NOW as the feature isn't already in use (from my understanding)

I could have made a simple code checking if the number of elements is 0, and if so it should remove that white space (screen) till the number of elements becomes at least 1, then shows the screen. But this solution will edit the code structure a bit and could be against the future plans as this screen functionality is still not 100% clear to me.

How it looks after putting the borderline:

repo: apache/age-viewer


text:
AMBARI-24316 Inconsistent Ambari warnings

How was this patch tested?
21825 passing (31s)
48 pending

retest this please

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/3190/
Test PASSed.

authors: asfgit atkach

repo: apache/ambari


text:
proposal: need an interface or function to translate old resource objects

backgroup
At present, for ApisixRoute Ingress ... resource update, we use the TranslateUpstream method to translate old objects. This method requires parsing the latest services to build objects, resulting in data inconsistency.
Due to the wrong construction of upstream objects, upstream cannot be deleted, resulting in object redundancy.
proposal
Implement an interface or function for related resources

Get route object from cache.
Using route to construct Upstream and PluginConfig.
Compare and write APISIX.

dependencies

[ ] ApisixRoute #1177
[ ] Ingress
[ ] Gateway

[ ] HTTPRoute
[ ] ...

#1050 Or we can realize the function of regularly cleaning up redundant objects in upstream.

Using route to construct Upstream and PluginConfig.

why we need this one?

Using route to construct Upstream and PluginConfig.

why we need this one?

Because the route object contains upstream_ id and plugin_id. For delete event, we only need these fields

repo: apache/apisix-ingress-controller


text:
docs: fix APISIX helm installation

This change was introduced from
https://github.com/apache/apisix-helm-chart/pull/74
Signed-off-by: Jintao Zhang zhangjintao9020@gmail.com
Please answer these questions before submitting a pull request

Why submit this pull request?

[x] Bugfix

[ ] New feature provided

[ ] Improve performance

[ ] Backport patches

Related issues

Bugfix

Description

How to fix?

New feature or improvement

Describe the details and related test reports.

Backport patches

Why need to backport?

Source branch

Related commits and pull requests

Target branch

Codecov Report

Merging #459 (56aefa5) into master (5d479ae) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #459      +/-   ##
==========================================
- Coverage   37.04%   37.03%   -0.02%
==========================================
  Files          47       46       -1
  Lines        3841     3840       -1
==========================================
- Hits         1423     1422       -1
  Misses       2233     2233
  Partials      185      185

Impacted Files
Coverage Δ

test/e2e/e2e.go

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4a55307...56aefa5. Read the comment docs.


text:
chore: Fix incorrect lilnks

Signed-off-by: imjoey majunjiev@gmail.com
Changes:

Fix the broken the links for Contributor Guide and Committer Guide in team.md
Add missing link in committer-guide.md

ping @juzhiyuan @liuxiran @nic-chen for reviewing. Thanks.

ping @juzhiyuan @liuxiran @nic-chen for reviewing. Thanks.

repo: apache/apisix-website


text:
dev/release: publish adbc_core crate on https://crates.io

What would you like help with?
Would it please be possible to publish the latest release of adbc_core on https://crates.io?

[ ] Update release docs
[ ] Update release scripts
[ ] Check if we can do a prerelease
[ ] Update verification script

CC: @alexandreyc

I'm planning a release this week. I'll give it a shot.

Many thanks @lidavidm!

Thanks @lidavidm
If you need some help don't hesitate to reach out!

Thanks!

authors: alexandreyc lidavidm rz-vastdata

repo: apache/arrow-adbc


text:
Cannot find the fields when making an aggregation on parquet file.

Describe the bug
Hey there. It seems that datafusion cannot recognize the field name existence when making an aggregation on a parquet file.  The code fail to run with following error:
Error: SchemaError(FieldNotFound { field: Column { relation: None, name: "fl_date" }, valid_fields: [Column { relation: Some(Bare { table: "?table?" }), name: "FL_DATE" }, Column { relation: Some(Bare { table: "?table?" }), name: "DEP_DELAY" }, Column { relation: Some(Bare { table: "?table?" }), name: "FL_DATE" }, Column { relation: Some(Bare { table: "?table?" }), name: "DEP_DELAY" }] })

Maybe I was making some mistakes?
To Reproduce

Download the flights 1m data:
https://www.tablab.app/datasets/sample/parquet

Run the code below:

use datafusion::{
    arrow::datatypes::{DataType, Field, Schema},
    prelude::*,
};

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx: SessionContext = SessionContext::new();
    let schema = Schema::new(vec![
        Field::new("FL_DATE", DataType::Utf8, true),
        Field::new("DEP_DELAY", DataType::Int32, true),
    ]);
    let df = ctx
        .read_parquet(
            "../../dataset/flights.parquet",
            ParquetReadOptions::default().schema(&schema),
        )
        .await?;
    let df = df
        .select_columns(&["FL_DATE", "DEP_DELAY"])?
        .aggregate(vec![col("FL_DATE")], vec![sum(col("DEP_DELAY"))])?;
    df.show().await?;
    Ok(())
}

Expected behavior
The aggregated data is displayed.
Additional context
cargo.toml
[package]
name = "data_engines"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
datafusion = "34"
tokio = { version = "1.35.1", features = ["full"] }

@VungleTienan
You simply need to use raw strings in order to prevent the normalizing of the column names.
let df = df
        .select_columns(&["FL_DATE", "DEP_DELAY"])?
        .aggregate(vec![col(r#""FL_DATE""#)], vec![sum(col(r#""DEP_DELAY""#))])?;

Thanks so much @marvinlanhenke

@VungleTienan I think you can close the issue now? Best regards

authors: VungleTienan marvinlanhenke

repo: apache/arrow-datafusion


text:
Update custom-table-providers.md

Fixes a typo in the documentation.

Thanks @nickpoorman

authors: Dandandan nickpoorman


text:
Array sum result is wrong with remainder fields when simd is on

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11051
Minimal example
{quote}use arrow::{array::PrimitiveArray, datatypes::Int64Type};
fn main() {
    let mut s = vec![];
    for _ in 0..32

     {
        s.push(Some(1i64));         s.push(None);
    }
    let v: PrimitiveArray = s.into();
    dbg!(arrow::compute::sum(&v));
}

{quote}
 dependency
{quote}arrow = {version = "2", features = ["simd"]}
{quote}
The following code in compute::sum is wrong. The bit mask is checked reversed.
{quote}            remainder.iter().enumerate().for_each(|(i, value)| {
                if remainder_bits & (1 << i) != 0  {
remainder_sum = remainder_sum + *value;
}
});

{quote}

Comment from Jörn Horstmann(jhorstmann) @ 2021-01-01T10:11:26.248+0000:
Hi [~niuzr], could you try the same with the latest master branch? There were some changes and also a bugfix how the vector masking is calculated after the 2.0 release in ARROW-10216.

repo: apache/arrow-rs


text:
Add a method to return the number of skipped rows in a RowSelection

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
RowSelection has a row_count method that returns the number of selected rows, but is missing a way to count the number of de-selected rows without iterating on the selectors
Describe the solution you'd like
Implement it as RowSelection::skipped_row_count
Describe alternatives you've considered
current state, which is that users have to reimplement it themselves
Additional context
Datafusion had to implement it here:
https://github.com/apache/datafusion/blob/f2159e6cae658a0a3f561ec2d15ea948213fd0f8/datafusion/core/src/datasource/physical_plan/parquet/page_filter.rs#L271-L277
Suggested by @alamb here: https://github.com/apache/datafusion/pull/12545#discussion_r1768748882

label_issue.py automatically added labels {'parquet'} from #6429

authors: alamb progval


text:
fix: Correctly handle take on dense union of a single selected type

Which issue does this PR close?
Closes #6206.
What changes are included in this PR?
At #5873, I naively called filter_primitive instead of filter to avoid arcing and downcasting, but this bypass the check for when all values match the predicate, which filter_primitive expects to happen, leading unreachable!() to be hit
This PR calls filter instead of filter_primitive, and removes the added pub(crate) from filter_primitive to avoid future misuse.
Are there any user-facing changes?
No

Thank you @gstvg and @tustvold

authors: alamb gstvg


text:
write_dataset freezes

Describe the bug, including details regarding any error messages, version, and platform.
An otherwise perfectly functioning arrow dataset does not finish the command write_dataset when passing on a hive structure, and I have to interrupt R. Looking at the folder structure, it seems to be writing files perfectly well until some point after which no new files are written -- but the job isn't finished.
The dataset also writes well into a single file (write_dataset without partitioning or grouping). It also writes well when I create less groups that I would like to. I haven't seen anyone complain about this, so I suspect that I am doing something so silly that no one has attempted before. Am I creating too many groups?
Grouping that works: A, B, C, D, E, where all groups are binary.
Grouping that doesn't work: A, B, C, D, X, where X has 90+ values (and not all values exist for each level of other variable. So, say, a combination A=1, B=1, C=1, D=1 might not have X=67.
Grouping that crashes: X, A, B, C, D.
In fact, just splitting into X crashes (not freezes).
I am on Garuda Linux (Arch-based) with R version 4.2.3.
Component(s)
R

Are you able to capture a core dump or create a small script that reproduces this?  Which version of Arrow are you using?

I can confirm similar behaviour with Python using pyarrow==12.0.1 with both write_dataset and the older write_to_dataset with a large number of partitions (over 5000 in my case). I'll post more details and try to dig in a bit deeper, but for now, this is mostly just to say "you're not alone" :)

Another thing to check is to monitor memory.  write_dataset, if it runs long enough, will fill up the OS's disk cache.  This can often lead to swapping / etc which can cause the entire system to freeze and run slowly.

Also, if you can create any kind of reproducible example we can take a look further.

authors: joshbode sometimesabird westonpace

repo: apache/arrow


text:
Pyarrow.Table.join() breaks on large tables v.18.0.0.dev486

Describe the bug, including details regarding any error messages, version, and platform.
Hi,
In my task I need to join two tables. One of 18m rows and another of 487m rows.
t_18m.join(t_487m, keys=[''col1, 'col2', 'col3'],  join_type="left outer")
I was using the most actual pyarrow version which is 17 at the moment. While performing a join it breaks with Segmentation fault (core dumped)
I tried to investigate and found the most recent version w/o such behaviour is v13. But I instead of Segmentation fault it:
either silently produces wrong result
or breaks with ! Invalid: There are more than 2^32 bytes of key data.  Acero cannot process a join of this magnitude
Next I searched on the github issues and found there are many similar user cases around. That's why I didn't include to many details in this report. You probably know these issues well.
There was #43495 the enchancement request, which as far as I understand has been included to the v18. I installed the v.18.0.0.dev486 package in order to test, but unfortunately It still throws the segmentation fault error on my case.
So if the enchancement is already merged into v.18.00 it still does not fix the problem.
Component(s)
Python

I also tryied to reduce the size of the right table and the working limit actually varies for me. Not able to find the exact number. I'm getting either the seg fault or the join result is incorrect.
My system has 4Tb memory in total, so it's not connected to the out-of-memory issue.
Here is the other system specs:

Oracle Linux Server 7.8

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 16511255
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Python 3.10.15

import pyarrow as pa
pa.version
'18.0.0.dev486'

Hi @kolfild26 , thanks for reporting this.
There are lots of solved issues from v13 to v18 that may cause silent wrong answer or segfault in hash join, and possibly more unrevealed ones as well. So it is not too surprising that different versions behave differently.
Could you please provide us the complete schemas and the estimated sizes of both tables? And better yet, could you give a more-or-less working limit of your case? These are essential informations to investigate this issue.
Also, there might be a workaround that worth a try, change t_18m.join(t_487m, keys=[''col1, 'col2', 'col3'],  join_type="left outer") to t_487m.join(t_18m, keys=[''col1, 'col2', 'col3'],  join_type="right outer"). (I assume t_18m is much smaller than t_487m and this will make our hash join to use the small table to build the hash table.)
Thanks.

Hi @kolfild26 , thank you for the feedback and further information. I'll try to reproduce the issue. However it will be helpful if you can supply the following information as well:

Any stacktrace of the segfault;
The join cardinality, or equally, the number of rows of the (left/right) join result.

Hi @kolfild26 , thank you for the feedback and further information. I'll try to reproduce the issue. However it will be helpful if you can supply the following information as well:

Any stacktrace of the segfault;
The join cardinality, or equally, the number of rows of the (left/right) join result.

My first attempt to reproduce the issue using non-null arbitrarily random distributed columns at the same schema and scale, failed (that is, my test passed w/o segfault). So I also need information about the distributions of each key column: null probability, min/max, any high cardinality value. Thank you.

@zanmato1984
Stacktrace:
Dec 16 01:07:44 kernel: python[37938]: segfault at 7f3004626050 ip 00007f3fc25441cd sp 00007f3f10b09018 error 4 in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: python[37971]: segfault at 7f3004626050 ip 00007f3fc25441db sp 00007f3f002b0018 error 4
Dec 16 01:07:44 kernel: python[37961]: segfault at 7f3004626050 ip 00007f3fc25441cd sp 00007f3f052d0018 error 4 in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: python[37957]: segfault at 7f3004626050 ip 00007f3fc25441db sp 00007f3f072d8018 error 4
Dec 16 01:07:44 kernel: python[37940]: segfault at 7f3004626050 ip 00007f3fc25441cd sp 00007f3f0fb07018 error 4
Dec 16 01:07:44 kernel: in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: python[37974]: segfault at 7f3004626050 ip 00007f3fc25441cd sp 00007f3d18f6d018 error 4 in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: python[37966]: segfault at 7f3004626050 ip 00007f3fc25441db sp 00007f3f02abf018 error 4
Dec 16 01:07:44 kernel: python[37951]: segfault at 7f3004626050 ip 00007f3fc25441db sp 00007f3f0a2ec018 error 4
Dec 16 01:07:44 kernel: python[37973]: segfault at 7f3004626050 ip 00007f3fc25441cd sp 00007f3efb7fe018 error 4
Dec 16 01:07:44 kernel: in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: python[37953]: segfault at 7f3004626050 ip 00007f3fc25441db sp 00007f3f092e6018 error 4
Dec 16 01:07:44 kernel: in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 kernel: in libarrow.so.1801[7f3fc1670000+2269000]
Dec 16 01:07:44 abrt-hook-ccpp: Process 35963 (python3.10) of user 1000 killed by SIGSEGV - dumping core

Here is the tables's statistics:

Script to get stats
import pyarrow as pa
import pyarrow.compute as pc
import pandas as pd
import pyarrow.types as patypes

def get_column_distributions(table):
    distributions = {}
    total_rows = table.num_rows

    for column in table.schema.names:
        col_data = table[column]
        null_count = pc.sum(pc.is_null(col_data)).as_py()
        null_percentage = (null_count / total_rows) * 100 if total_rows > 0 else 0

        # Compute the cardinality (unique count / total count)
        unique_count = pc.count_distinct(col_data.filter(pc.is_valid(col_data))).as_py()
        cardinality_percentage = round((unique_count / total_rows)*100,3) if total_rows > 0 else 0

        if patypes.is_integer(col_data.type) or patypes.is_floating(col_data.type):
            stats = {
                "count": pc.count(col_data).as_py(),
                "nulls": null_count,
                "null_percentage": null_percentage,
                "cardinality_percentage": cardinality_percentage,
                "min": pc.min(col_data).as_py(),
                "max": pc.max(col_data).as_py(),
            }
        elif patypes.is_string(col_data.type) or patypes.is_binary(col_data.type):
            value_counts = pc.value_counts(col_data.filter(pc.is_valid(col_data)))
            stats = {
                "nulls": null_count,
                "null_percentage": null_percentage,
                "cardinality_percentage": cardinality_percentage,
                "value_counts": value_counts.to_pandas().to_dict("records"),
            }
        else:
            stats = {
                "nulls": null_count,
                "null_percentage": null_percentage,
                "cardinality_percentage": cardinality_percentage,
                "message": f"Statistics not supported for type: {col_data.type}"
            }

        distributions[column] = stats

    return distributions

small

large

Would it be easier if I attached the tables here?

Would it be easier if I attached the tables here?

@kolfild26 yeah please, that's even more useful.

The join cardinality, or equally, the number of rows of the (left/right) join result.

And also, do you have this one?

Cardinality can refer to different things. In a database context, cardinality usually refers to the number of unique values in a relational table column relative to the total number of rows in the table. So, if are both talking about the same, cardinality is presented in the report above, cardinality_percentage = (unique_count / total_rows)*100

But "cardinality" can also represent the size of the join result which is what I originally asked about. Do you have that? (You can just run the right join and count the number of rows).
And thank you for the source files. I'll try to reproduce the issue using these files in my local.

Hi @kolfild26 , I've successfully run the case in my local (M1 MBP with 32GB memory, arrow 18.1.0) but didn't reproduce the issue.
My python script:
import pandas
import pickle
import pyarrow

def main():
    print("pandas: {0}, pyarrow: {1}".format(pandas.__version__, pyarrow.__version__))
    with open('small.pkl', 'rb') as f: small = pickle.load(f)
    with open('large.pkl', 'rb') as f: large = pickle.load(f)
    print("small size: {0}, large size: {1}".format(small.num_rows, large.num_rows))
    join = small.join(large, keys=['ID_DEV_STYLECOLOR_SIZE', 'ID_DEPARTMENT', 'ID_COLLECTION'], join_type='left outer')
    print("join size: {0}".format(join.num_rows))

if __name__ == "__main__":
    main()

Result:
python test.py
pandas: 2.2.3, pyarrow: 18.1.0
small size: 18201475, large size: 360449051
join size: 18201475

Did I miss something?

The resulted join size looks correct.
Could you please check:

apply filter ID_DEV_STYLECOLOR_SIZE = 88506230299 and ID_DEPARTMENT = 16556030299. It should return 2 in PL_VALUE column.
Apply sum(PL_VALUE) and it should return 58360744

That's just to eliminate 'false positive'. I mentioned that I tested on different versions and it sometimes caused a silent wrong answer even though there were no seg.fault.
If all above is correct, might the segfault error be caused by any system/os settings?

my setup
    Oracle Linux Server 7.8

    ulimit -a
    core file size (blocks, -c) 0
    data seg size (kbytes, -d) unlimited
    scheduling priority (-e) 0
    file size (blocks, -f) unlimited
    pending signals (-i) 16511255
    max locked memory (kbytes, -l) unlimited
    max memory size (kbytes, -m) unlimited
    open files (-n) 4096
    pipe size (512 bytes, -p) 8
    POSIX message queues (bytes, -q) 819200
    real-time priority (-r) 0
    stack size (kbytes, -s) unlimited
    cpu time (seconds, -t) unlimited
    max user processes (-u) 4096
    virtual memory (kbytes, -v) unlimited
    file locks (-x) unlimited

    Python 3.10.15

    import pyarrow as pa
    pa.version
    '18.1.0'

apply filter ID_DEV_STYLECOLOR_SIZE = 88506230299 and ID_DEPARTMENT = 16556030299. It should return 2 in PL_VALUE column.

Correct:
>>> cond = pc.and_(pc.equal(large['ID_DEV_STYLECOLOR_SIZE'], 88506230299), pc.equal(large['ID_DEPARTMENT'], 16556030299))
>>> filtered = large.filter(cond)
>>> print(filtered)
pyarrow.Table
ID_DEV_STYLECOLOR_SIZE: int64
ID_DEPARTMENT: int64
ID_COLLECTION: int64
PL_VALUE: int64
----
ID_DEV_STYLECOLOR_SIZE: [[88506230299]]
ID_DEPARTMENT: [[16556030299]]
ID_COLLECTION: [[11240299]]
PL_VALUE: [[2]]
>

Apply sum(PL_VALUE) and it should return 58360744

No:
>>> sum = pc.sum(large['PL_VALUE'])
>>> print(sum)
461379027

That's just to eliminate 'false positive'. I mentioned that I tested on different versions and it sometimes caused a silent wrong answer even though there were no seg.fault.

Hmm, I think we should only focus on v18.1.0. As I mentioned, there are a lot of fixes ever since, so the behavior in prior versions will vary for sure, and I think most of the issues (if not all) are already addressed.

If all above is correct, might the segfault error be caused by any system/os settings?

I also verified on my Intel MBP (I just realized that we have x86-specialized SIMD code path for hash join so I wanted to see if the issue was there), but still unable to reproduce. And your setup doesn't seem to have any particular thing to do with this issue.
To proceed with the debugging:

Did you run my python script on your env to see if it runs into segfault? (And in case it doesn't, would you kindly help to fix it to make the segfault happen?) I think this is quite essential, because we need to agree on a minimal reproducible case (at least on either env of us). Then I can ask some other people to help verifying on broader environments.
Would you help to confirm the difference of sum(PL_VALUE) in my run (461379027) against yours (58360744)?
What is your CPU model?
In your original run of segfault (again, on v18.1.0), is it always reproducible or by chance?

Debugging this kind of issue is tricky and takes time and communication. I really appreciate your patience @kolfild26 , thank you!

2⃣ I meant filter() and sum() to be applied to the resulted table, i.e. join while you have applied to large.
3⃣ Intel(R) Xeon(R) Gold 6246 CPU @ 3.30GHz. 4 sockets * 12 cores = 48 logical cpus
1⃣ 4⃣  Yes, segfault occures always, having the fixed size of the input tables. All recent tests I refer to, are on v18.1.0

I can now confirm that the problem does exist.
By applying filter and sum on the join result, I found my previous non-segfault runs were false positive:
    join = small.join(large, keys=['ID_DEV_STYLECOLOR_SIZE', 'ID_DEPARTMENT', 'ID_COLLECTION'], join_type='left outer')
    print("join size: {0}".format(join.num_rows))
    cond = pc.and_(pc.equal(join['ID_DEV_STYLECOLOR_SIZE'], 88506230299), pc.equal(join['ID_DEPARTMENT'], 16556030299))
    filtered = join.filter(cond)
    print("filtered")
    print(filtered)
    sum = pc.sum(join['PL_VALUE'])
    print("sum")
    print(sum)

Result:
filtered: PL_VALUE: [[null]]
...
sum: 33609597 # Another run emits 33609997

And I also happen to have access to a x86 Ubuntu desktop, on which I reproduced the segfault.
I'm now digging into it.
Also, considering the silent wrong answer on some platforms, I'm marking this issue critical.
Thanks alot @kolfild26 for helping me to reproduce the issue!


text:
GH-38699: [C++][FS][Azure] Implement CreateDir()

Rationale for this change
It seems that we can't create a directory explicitly without hierarchical namespace support.
It seems that Azure Blob Storage supports only virtual directory. There is no directory. If a file (blob) name has "/", it's treated that the file (blob) exists under a virtual directory.
It seems that Azure Data Lake Storage Gen2 supports a real directory.
See also:
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction
What changes are included in this PR?
This change chooses the following behavior:

Container can be created with/without hierarchical namespace support.
Directory can be created with hierarchical namespace support.
Directory can't be created without hierarchical namespace support. So do nothing without hierachical namespace support. (arrow::Status::OK() is just returned.)

Are these changes tested?
Azurite doesn't support hierarchical namespace yet. So I can't test the implementation for hierarchical namespace yet. Sorry.
Are there any user-facing changes?
Yes.

Closes: #38699

@Tom-Newton What do you think about this behavior?

This change chooses the following behavior:

Container can be created with/without hierarchical namespace support.
Directory can be created with hierarchical namespace support.
Directory can't be created without hierarchical namespace support. (arrow::Status::NotImplemented is returned for this case.)

Do you have any (simple) document how to setup an account for AzureHierarchicalNamespaceFileSystemTest?

What do you think about this behavior?
The only part I'm questioning is whether to return arrow::Status::NotImplemented on flat blob storage accounts. Possibly it would be better just to return a success status without doing anything.

I think the right choice will depend on how much CreateDir() is used. For example if CreateDir() is used every time arrow writes a partitioned parquet table, then returning an error status could be a bit of a problem.

Do you have any (simple) document how to setup an account for AzureHierarchicalNamespaceFileSystemTest?

You will need an azure account. You should be able to create a free account at https://azure.microsoft.com/en-gb/free/. You should the. Be able to create a storage account through the portal web UI.
I can probably write a more specific doc if needed but this is Azure's doc https://learn.microsoft.com/en-us/azure/storage/blobs/create-data-lake-storage-account
A few suggestions on configuration:

Use Standard general-purpose v2 not premium
Use LRS redundancy
Obviously you will want to enable hierarchical namespace.
Set the default access tier to hot
SFTP, NFS and file shares are not required.

The only part I'm questioning is whether to return arrow::Status::NotImplemented on flat blob storage accounts. Possibly it would be better just to return a success status without doing anything.
I think the right choice will depend on how much CreateDir() is used. For example if CreateDir() is used every time arrow writes a partitioned parquet table, then returning an error status could be a bit of a problem.

Good point! I'll change the behavior to just return arrow::Status::OK().

Do you have any (simple) document how to setup an account for AzureHierarchicalNamespaceFileSystemTest?

You will need an azure account. You should be able to create a free account at https://azure.microsoft.com/en-gb/free/. You should the. Be able to create a storage account through the portal web UI.
I can probably write a more specific doc if needed but this is Azure's doc https://learn.microsoft.com/en-us/azure/storage/blobs/create-data-lake-storage-account
A few suggestions on configuration:
...

Thanks! I could create an account and confirm that the implementation passes the added tests.
I'll add the provided information as a comment for other developers.

I've updated.

I've updated.

I'll merge this.

authors: Tom-Newton kou


text:
GH-39537: [Packaging][Python] Add a numpy<2 pin to the install requirements for the 15.x release branch

Rationale for this change
PyArrow wheels for the 15.0.0 release will not be compatible with future numpy 2.0 packages, therefore it is recommended to add this upper pin now for releases. We will keep the more flexible pin on the development branch (by reverting this commit on main, but so it can be cherry-picked in the release branch)

Closes: #39537

@github-actions crossbow submit -g wheel -g python

I think the macOS wheel failures are unrelated and started happening yesterday, probably due to: https://github.com/apache/arrow/pull/39065 being merged. I'll investigate further and open an issue.

We can remove this pin once we build our wheels with NumPy 2, right?

Yes, indeed.

authors: jorisvandenbossche raulcd


text:
What is the version of beam-runners-flink-1.11 which support JDK 17

What needs to happen?
Need to know version of beam-runners-flink-1.11 which support JDK 17
Issue Priority
Priority: 1
Issue Component
Component: beam-community

Like #22041, please ask on user@beam.apache.org.

authors: chiransiriwardhana manuzhang

repo: apache/beam


text:
[Failing Test]: PostCommit Java SingleStoreIO IT failing

What happened?
Since Jan 31, 2024
Fails Install Singlestore cluster
Run kubectl apply -f /runner/_work/beam/beam/.test-infra/kubernetes/singlestore/sdb-cluster.yaml
memsqlcluster.memsql.com/sdb-cluster created
error: timed out waiting for the condition on memsqlclusters/sdb-cluster
Error: Process completed with exit code 1.

Performance test also failing
Issue Failure
Failure: Test is continually failing
Issue Priority
Priority: 2 (backlog / disabled test but we think the product is healthy)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

workload logs:
Status: Pods have warnings. node-sdb-cluster-master

2024-03-07 15:19:31.496 EST
✓ Created node with node ID A40217C2599E6693E3D37C2BCB195DA378E230AA
2024-03-07 15:19:31.908 EST
memsqlctl will perform the following actions:
2024-03-07 15:19:31.908 EST
  · Update configuration setting on node with node ID A40217C2599E6693E3D37C2BCB195DA378E230AA on port 3306
2024-03-07 15:19:31.908 EST
    - Update node config file with setting minimum_core_count=0
2024-03-07 15:19:31.908 EST
{}
2024-03-07 15:19:31.908 EST
Would you like to continue? [Y/n]:
2024-03-07 15:19:31.908 EST
Automatically selected yes, non-interactive mode enabled

...

2024-03-07 15:19:38.746 EST
2024-03-07 20:19:38.746   INFO: Thread 115121 (ntid 225, conn id -1): memsqld_main: Flavor: 'production'
2024-03-07 15:19:38.756 EST
2024-03-07 20:19:38.756  ERROR: Thread 115104 (ntid 361, conn id -1): Run: Error getting cluster database
2024-03-07 15:19:38.756 EST
2024-03-07 20:19:38.756  ERROR: Thread 115104 (ntid 361, conn id -1): Run: Error getting cluster database
2024-03-07 15:19:38.756 EST
2024-03-07 20:19:38.756  ERROR: Thread 115104 (ntid 361, conn id -1): Run: Error getting cluster database
2024-03-07 15:19:38.756 EST
2024-03-07 20:19:38.756  ERROR: Thread 115104 (ntid 361, conn id -1): Run: Error getting cluster database
2024-03-07 15:19:38.757 EST
2024-03-07 20:19:38.757   INFO: Thread 115121 (ntid 225, conn id -1): CreateDatabase: CREATE DATABASE `memsql` with sync durability / sync input durability, 0 partitions, 0 sub partitions, 0 logical partitions, log file size 16777216.

...

2024-03-07 15:19:40.691 EST
Started singlestore (199)
2024-03-07 15:19:40.694 EST
Ensuring the root password is setup
2024-03-07 15:19:40.787 EST
Error 2277: This node is not part of the cluster.
2024-03-07 15:19:40.845 EST
2024-03-07 20:19:40.845   INFO: Thread 115120 (ntid 344, conn id -1): OnAsyncCompileCompleted: Query information_schema.'SELECT 1' submitted 177 milliseconds ago, queued for 17 milliseconds, compiled asynchronously in 160 milliseconds
2024-03-07 15:19:40.847 EST
2024-03-07 20:19:40.847  ERROR: [0 messages suppressed] ProcessHandshakeResponsePacket() failed. Sending back 1045: Access denied for user 'root'@'localhost' (using password: NO)

...

2024-03-07 15:19:41.181 EST
2024-03-07 20:19:41.181   INFO: Thread 115120 (ntid 344, conn id -1): OnAsyncCompileCompleted: Query (null).'SELECT @@MEMSQL_VERSION' submitted 133 milliseconds ago, queued for 17 milliseconds, compiled asynchronously in 116 milliseconds
2024-03-07 15:19:50.497 EST
Error 2277: This node is not part of the cluster.
2024-03-07 15:20:00.494 EST
Error 2277: This node is not part of the cluster.
2024-03-07 15:20:10.569 EST
Error 2277: This node is not part of the cluster.
2024-03-07 15:20:20.496 EST
Error 2277: This node is not part of the cluster.
2024-03-07 15:20:30.496 EST
Error 2277: This node is not part of the cluster.

Status: Pods have warnings. node-sdb-cluster-leaf-ag1

2024-03-07 15:19:49.683 EST
Initializing OpenSSL 1.0.2u-fips  20 Dec 2019
2024-03-07 15:19:49.688 EST
ERROR 2277 (HY000) at line 1: This node is not part of the cluster.
2024-03-07 15:19:49.704 EST
[2024-03-07 20:19:49 startup-probe] Aborting due to query failure
2024-03-07 15:19:49.808 EST
2024-03-07 20:19:49.808   INFO: Thread 115120 (ntid 388, conn id -1): OnAsyncCompileCompleted: Query (null).'select @@version_comment limit 1' submitted 142 milliseconds ago, queued for 17 milliseconds, compiled asynchronously in 125 milliseconds
2024-03-07 15:19:54.664 EST
ERROR 2277 (HY000) at line 1: This node is not part of the cluster.
2024-03-07 15:19:54.669 EST
[2024-03-07 20:19:54 startup-probe] Aborting due to query failure

...

2024-03-07 15:24:34.665 EST
ERROR 2277 (HY000) at line 1: This node is not part of the cluster.
2024-03-07 15:24:34.671 EST
[2024-03-07 20:24:34 startup-probe] Aborting due to query failure
2024-03-07 15:24:39.661 EST
ERROR 2277 (HY000) at line 1: This node is not part of the cluster.
2024-03-07 15:24:39.667 EST
[2024-03-07 20:24:39 startup-probe] Aborting due to query failure
2024-03-07 15:24:42.829 EST
2024-03-07 20:24:42.829  ERROR: Thread 115101 (ntid 408, conn id -1): Run: Error getting cluster database
2024-03-07 15:24:42.829 EST
2024-03-07 20:24:42.829  ERROR: Thread 115103 (ntid 406, conn id -1): Run: Error getting cluster database
2024-03-07 15:24:42.829 EST
2024-03-07 20:24:42.829  ERROR: Thread 115104 (ntid 405, conn id -1): Run: Error getting cluster database

The k8s configurations has been changed for months but the cluster failing to create suddenly since Jan 31. CC: @AdalbertMemSQL the author

Hey @Abacn
Is it possible to somehow retrieve full workload logs?

Fixed by #30725

authors: Abacn AdalbertMemSQL


text:
[Task]: Improve Enrichment docs

What needs to happen?
There are a few targeted fixes needed for the Enrichment docs:

https://beam.apache.org/documentation/transforms/python/elementwise/enrichment/ should mention BigQuery
https://beam.apache.org/documentation/transforms/python/elementwise/enrichment/ should describe how we do batching and how we do caching (with_redis_cache)
https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb and https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/vertex_ai_feature_store_enrichment.ipynb should elaborate a little more on what a cross-join means in this context (maybe a picture would be nice?)
In https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/bigtable_enrichment_transform.ipynb the example handler with composite row key support link is dead

Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components

[X] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[X] Component: Website
[ ] Component: Infrastructure
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

@claudevdm this would be good to pick up at some point when you have space (don't drop other things, just when this fits in nicely)


text:
[BEAM-9641] Support ZetaSQL DATE type as a Beam LogicalType

This PR adds support of all ZetaSQL (BigQuery Standard SQL) DATE functions to BeamSQL:

CURRENT_DATE
EXTRACT
DATE (constructing DATE from DATETIME not supported)
DATE_ADD
DATE_SUB
DATE_DIFF
DATE_TRUNC
FORMAT_DATE
PARSE_DATE
UNIX_DATE
DATE_FROM_UNIX_DATE

WEEK part not supported

r: @apilloud
cc: @TheNeuralBit @kennknowles

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

What do you think about going ahead and defining the date logical type in org.apache.beam.sdk.schemas.logicaltypes? It would be useful in other contexts - for example it would give us something to map Avro's logical date type to (currently it is just overloaded with millis-instant onto DATETIME)
cc: @reuvenlax

What do you think about going ahead and defining the date logical type in org.apache.beam.sdk.schemas.logicaltypes? It would be useful in other contexts - for example it would give us something to map Avro's logical date type to (currently it is just overloaded with millis-instant onto DATETIME)

Done. Thanks for the suggestion. I made the Date type a public logical type in org.apache.beam.sdk.schemas.logicaltypes and added a layer of indirection by letting SqlTypes.DATE reference it.

Oops, forgot to include in my comments: ZetaSQL's range is much smaller than the underlying type, can you add a test or two for that?  How do out of range values fail? (Also worth asking, do we need any special treatment for boundary conditions (LocalDate.MIN, LocalDate.MAX)? Probably not for now.)

Ah, just realized that the previous comments were not sent out.

Could you help trigger the tests again?
For the comment on range: Thanks for pointing it out. I overlooked this problem. I would like to create a separate PR to address it, along with range testing for other types as well.

retest this please

The failing test SparkPortableExecutionTest.testExecution should be unrelated to this change.

Run Java PreCommit

Run SQL Postcommit

Rebased against master. Please run precommit tests again.

retest this please

retest this please

Java PreCommit failed due to a build failure. Please help run again.

Run Java PreCommit

Run Java PreCommit

Run Java PreCommit

Something just occurred to me - are there any tests that use the DATE Type in an aggregation (e.g. MAX)?
I'd think that would run into the same issue I have in #11456

Interesting question. You should probably add a test for JOIN as well, which will have a similar class of problems.

are there any tests that use the DATE Type in an aggregation (e.g. MAX)?

No. Thanks for bringing this up. I think it is likely to run into the problem.

authors: TheNeuralBit apilloud robinyqiu


text:
[BEAM-10572] Eliminate nullability errors from :sdks:java:extensions:sql:datacatalog

Fixing the nullability issues for sub-module sdks:java:extensions:sql:datacatalog

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

Run SQL_Java11 PreCommit

Run SQL_Java11 PreCommit

Hi Jayendra, thanks for volunteering to fix these issues.
The SQL_Java11 precommit failure doesn't look like a flake, it seems like this change introduced an error. Please take a look.
05:09:22 > Task :sdks:java:extensions:sql:datacatalog:compileTestJava FAILED
05:09:22 error: option -Xbootclasspath/p: cannot be used together with --release
05:09:22 Usage: javac <options> <source files>
05:09:22 use --help for a list of possible options

Hi Jayendra, thanks for volunteering to fix these issues.
The SQL_Java11 precommit failure doesn't look like a flake, it seems like this change introduced an error. Please take a look.
05:09:22 > Task :sdks:java:extensions:sql:datacatalog:compileTestJava FAILED
05:09:22 error: option -Xbootclasspath/p: cannot be used together with --release
05:09:22 Usage: javac <options> <source files>
05:09:22 use --help for a list of possible options

When I run that task locally I don't get any error.
jayendra@alienware:~/beam$   ./gradlew :sdks:java:extensions:sql:datacatalog:compileTestJava
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon: https://docs.gradle.org/5.2.1/userguide/gradle_daemon.html.
Daemon will be stopped at the end of the build stopping after processing
Configuration on demand is an incubating feature.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 42s
73 actionable tasks: 37 executed, 1 from cache, 35 up-to-date

is it something specific to java version ?

Ah, yes. To run the same commands as the Java11 builds you must set some properties. For example on mac:
git checkout github/pr/12366 # I have fetch spec set up like this

./gradlew \
    -PcompileAndRunTestsWithJava11 \
    -Pjava11home=/Library/Java/JavaVirtualMachines/jdk-11-latest/Contents/Home \
    :sdks:java:extensions:sql:datacatalog:compileTestJava

I actually got a different failure: https://gradle.com/s/gegzdydsbu2si
Execution failed for task ':sdks:java:extensions:sql:datacatalog:compileTestJava'.
> release version 11 not supported

That seems a bit odd. I think I did not reproduce the problem properly or there are differences between my JDK11 and the one on Jenkins.
CC @tysonjh as we talked the other day about whether it mattered to make these unique Gradle targets versus just configuring these properties in the Jenkins job.

I expect this is a conflict between checkerframework and Java 11. Java 9+ are supported since 3.x but it seems it might rely on -Xbootclasspath/p option which has been removed. It may be that there is something specific to Gradle and how it uses a forked JVM in order to compile and run with Java 11.
Since checker is only needed for static analysis of the source, it can be disabled for Java 11 builds.

According to https://github.com/kelloggm/checkerframework-gradle-plugin/issues/43#issuecomment-551104939 it should be OK. I will submit a report or ask on the user list.

Ah, yes. To run the same commands as the Java11 builds you must set some properties. For example on mac:
git checkout github/pr/12366 # I have fetch spec set up like this

./gradlew \
    -PcompileAndRunTestsWithJava11 \
    -Pjava11home=/Library/Java/JavaVirtualMachines/jdk-11-latest/Contents/Home \
    :sdks:java:extensions:sql:datacatalog:compileTestJava

I actually got a different failure: https://gradle.com/s/gegzdydsbu2si
Execution failed for task ':sdks:java:extensions:sql:datacatalog:compileTestJava'.
> release version 11 not supported

That seems a bit odd. I think I did not reproduce the problem properly or there are differences between my JDK11 and the one on Jenkins.
CC @tysonjh as we talked the other day about whether it mattered to make these unique Gradle targets versus just configuring these properties in the Jenkins job.

When I run with Java 11 I use the following (example from different task):
./gradlew clean
./gradlew -Dorg.gradle.java.home=/usr/local/buildtools/java/jdk11 :runners:direct-java:validatesRunner --scan

Tyson's command will use JDK11 also for the main gradle task, while still having source and target Java version 8. Since checker does support Java 11 and is aware of the removal of -Xbootclasspath/p perhaps that approach causes the plugin to configure flags appropriately.

Confirmed that Tyson's command worked.

Yup: https://github.com/kelloggm/checkerframework-gradle-plugin/blob/6739a86cf030ab35634a2b0ab6ac8859fe835473/src/main/groovy/org/checkerframework/gradle/plugin/CheckerFrameworkPlugin.groovy#L373

Filed kelloggm/checkerframework-gradle-plugin#117. We don't have to wait for a fix, though. We can just disable checker for Java 11 for now, and we will probably at some point switch to Tyson's invocation for most tests - compiling and running on JRE11 but with Java 8 settings.

Filed kelloggm/checkerframework-gradle-plugin#117. We don't have to wait for a fix, though. We can just disable checker for Java 11 for now, and we will probably at some point switch to Tyson's invocation for most tests - compiling and running on JRE11 but with Java 8 settings.

Disable checker(for java 11) for whole project or for just this package ?

Filed kelloggm/checkerframework-gradle-plugin#117. We don't have to wait for a fix, though. We can just disable checker for Java 11 for now, and we will probably at some point switch to Tyson's invocation for most tests - compiling and running on JRE11 but with Java 8 settings.

Disable checker(for java 11) for whole project or for just this package ?

Disable the checker for the whole project.

Can some one point me how to disable that. I know we have to modify some groovy script under .test-infra/jenkins, but don't know exactly where and how ?

Here: https://github.com/apache/beam/blob/de8ff705145cbbc41bea7750a0a5d3553924ab3a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L763
This block should be skipped if compileAndRunTestsWithJava11 is set and Gradle's configuration phase runs using JDK8. I think it would be fine to just skip that block whenever compileAndRunTestsWithJava11 is set.

It also looks like -PskipCheckerFramework is supported by kelloggm/checkerframework-gradle-plugin so you can just add that flag to the Jenkins job. That is probably best.

Run SQL_Java11 PreCommit

It also looks like -PskipCheckerFramework is supported by kelloggm/checkerframework-gradle-plugin so you can just add that flag to the Jenkins job. That is probably best.

It's was there so just triggered the job.

authors: aromanenko-dev ibzib jayendra13 kennknowles tysonjh


text:
Remove redundant or inappropriate pieces of the capability matrix

Each commit is an independent change that removes something that should not be in the capability matrix:

runners that are on branches and not released
features that are not designed and may even require inventing new things, or may be impossible
redundant columns

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Choose reviewer(s) and mention them in a comment (R: @username).
[x] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[x] Update CHANGES.md with noteworthy changes.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Dataflow
Flink
Samza
Spark
Twister2

Go

---

---

---

Java

Python

---

---

XLang

---

---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website
Whitespace
Typescript

Non-portable

Portable
---

---
---
---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

http://apache-beam-website-pull-requests.storage.googleapis.com/13694/documentation/runners/capability-matrix/index.html

http://apache-beam-website-pull-requests.storage.googleapis.com/13694/documentation/runners/capability-matrix/index.html

@griscz wdyt?

@griscz wdyt?

Let me know when it is a good time to rebase and fix this one up.

the website revamp is done, you should be fine to rebase now.

Done. PTAL. Incidentally, the hack to add /index.html to all the URLs seems to be gone (needed to browse seemlessly on GCS staging).

Incidentally, the hack to add /index.html to all the URLs seems to be gone (needed to browse seemlessly on GCS staging).

I think this only happened on the runner detail links for some reason. I filed BEAM-11860 for it

authors: TheNeuralBit kennknowles


text:
Add documentation link to the interactive environment

Part of a documentation audit to add relevant gcloud docs links.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @KevinGG


text:
[Playground] Examples CD

Examples CD optimization
Main feature comparing to the old workflow is running the examples on production backend. This ensures that examples really work after deployment, and saves resources on building a temporary runner

Introduce BEAM_USE_WEBGRPC variable in grpc_client.py. It enables WEBGRPC protocol instead of GRPC, to access public production endpoints.
Reusable workflow to optimize GH workflows

addresses #23463
addresses #23464
addresses #23465
See example CD run of this job with necessary amendments (private GCP credentials, push GH triggers)
python passes and go doesn't: known issue #23600

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

lgtm

authors: eantyshev olehborysevych


text:
Build and publish multi-arch wheels separately from main wheels

Right now, building wheels is failing because it tries to upload an artifact for the multi-arch wheels with the same name as the regular wheels. Previously this worked silently and one was dropped. This fixes that issue by appending -aarch64 to the multi-arch wheels and makes sure that it gets uploaded to gcs as well.
Example succeeding now - https://github.com/apache/beam/actions/runs/7266076148

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @jrmccluskey