text:
Questions about training NER models

So, I'm glad I found this gem. Most NLP gems for Ruby seem to be many years out of date.
I got it up and running rather easily, but I am having some trouble that you may be ale to point em int he right direction about:
rake task to train the model:
trainer = Mitie::NERTrainer.new("#{Rails.root}/bin/total_word_feature_extractor.dat")

    # Give NER Model ingredient names to train it
    ingredient_data = CSV.read("#{Rails.root}/db/seeds/Recipes-All Recipes.csv", headers: true)
    ingredient_names = ingredient_data.map { |row| row.entries[0][1] }

    # Give NER Model measurement units to train it
    units =  %w(teaspoon tsp teaspoons tsps tablespoon tbsp tablespoons tbsps cup c cups ounce oz ounces gram gr g grams milligram mg miligrams calorie cals calories clove cloves')

    tokens = ingredient_names + units

    instance = Mitie::NERTrainingInstance.new(tokens)
    instance.add_entity(0..144, "Ingredient")
    instance.add_entity(145..170, "Measurement Unit")

    trainer.add(instance)

    model = trainer.train

    model.save_to_disk("#{Rails.root}/bin/eddi_model.dat")

"Soy Mustard Salmon" is actually the string name of of the ingredients in the CSV -- I expected it to say it was an Ingredient...but the generated model seems to score everything as a Measurement Unit -- despite it being a much smaller dataset in the instance.
I'm also noticing some portions where seems to match correctly, but it gives me the WHOLE string in the doc.entities data instead of the matching portion.
Example:

I will say that the ease of use was great -- I'm wondering if there's anything I can do to better train the model. More data?

Hey @hjoseph96, I'm not sure how well it'll work on short phrases (rather than complete sentences) or this specific use case, but for training, you'll want a training instance for each segment.
segment = "- 1 tablespoon minced garlic, (10g)"

# clean up data
segment = segment.delete_prefix("- ")

# tokenize
tokens = Mitie.tokenize(segment) # ["1", "tablespoon", "minced", "garlic", ",", "(", "10g", ")"]

# add entities
instance = Mitie::NERTrainingInstance.new(tokens)
instance.add_entity(1..1, "Measurement Unit") # tablespoon
instance.add_entity(2..3, "Ingredient")       # minced garlic
trainer.add(instance)

repo: ankane/mitie-ruby


text:
Align text on buttons

I think you should try to align the arrows on the frameworks button to the right

-Kindly I suggest to use descriptive class.
-Kindly I suggest to use border radius in the Grid section

I think using a more descriptive class names is better

I think using class names like Section 1, Section 2 etc. As you used it here is not much descriptive

I suggest you use descriptive class names (e.g. Header-section, Toolbar etc.) in your HTML codes

source: gharchive/issue

authors: Yakub-Ojo Yonasle mosidrum

license: MIT

license_type: permissive

license_source: github-api


text:
官网demo中完整版在QQ浏览器（Chromium 70.0.3538.25）中不能进入登录页。显示“globalThis is not defined”

⚠ 重要 ⚠ 在进一步操作之前，请检查下列选项。如果您忽视此模板或者没有提供关键信息，您的 Issue 将直接被关闭

[x] 已阅读 文档.
[x] 确保您的代码已是最新或者所报告的 Bug 在最新版本中可以重现. (部分 Bug 可能已经在最近的代码中修复)
[x] 已在 Issues 中搜索了相关的关键词
[ ] 不是 ant design vue 组件库的 Bug

描述 Bug
官网demo中完整版在QQ浏览器（Chromium 70.0.3538.25）中不能进入登录页。显示“globalThis is not defined”
复现 Bug
1、官网demo中完整版打开不正常，显示Logo不进入登录页。console中显示“Uncaught ReferenceError: globalThis is not defined at vendor.e7c0f2cb.js:14”。但是精简版是正常的。
2、分别下载完整版源码的 master分支 和 v2.7.2。不做任何更改，进行编译直接生成dist进行浏览。master分支表现和上面一样都无法进入登录页，提示“globalThis is not defined”，但是v2.7.2正常。
3、以上所有在谷歌浏览器（Chromium 95）以及360极速浏览器（Chromium 86）中均正常。
4、以上浏览器均为最新版。
系统信息

操作系统: WIN10
浏览器：QQ浏览器（Chromium 70.0.3538.25）
Node 版本:v14.18.1
包管理器 (npm/yarn/pnpm) 及其版本:yarn v1.22.17 vite v2.6.11(master分支)/vite v2.5.7（vben v2.7.2）

尽量使用高版本浏览器，这个项目不对历史老版本浏览器做太多的兼容，最好是chrome80+以上

authors: anncwb zhaohuiyingxue

repo: anncwb/vue-vben-admin


text:
🛑 Pedproject is down

In 832ce9a, Pedproject (https://педпроект.рф) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Pedproject is back up in 4ca6b49 after 34 minutes.

repo: ano-pedproject/repo


text:
🛑 Pedacademy is down

In eb8f59c, Pedacademy (https://педакадемия.рф) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Pedacademy is back up in 10845a6 after 32 minutes.


text:
duplicate image

hi
i connect with 2 device but i see 3 image on client

I think this is happening because the server is relying on the device itself when displaying the streams. To solve this, I've added a control to peer_connection.ontrack so it only runs once. Just create a variable x with a value of 0, create a condition of x == 0 and, if the condition is true, x += 1 and execute the content of the function.

authors: Diegocndd parsibox

repo: anoek/webrtc-group-chat-example

license: unlicense

license_source: bigquery


text:
🛑 APRO IT Solutions is down

In 0335f53, APRO IT Solutions (https://aproitsolutions.in/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: APRO IT Solutions is back up in 1cfa6f3.

repo: anonxrider/uptime


text:
changed the style of the cards on homepage

Changed the style of the cards on homepage, made them rounded in shape and added a title below

I deleted the login page code and if needed I will make it on time

source: gharchive/pull-request


text:
Share generation script/code

Thanks a lot for the theme, I like it a lot! Are you generating the theme with a script or so? If so, could you share that code? I'd like to modify it for myself just slightly to make the foreground shades a bit more bright.

Unfortunately, there isn't a generator script for monochrome, it was built long before the release with tweaks on-the-go.

Alright, thanks anyway!

I'm currently working on it.

It's on the master.

authors: anotherglitchinthematrix marcelbeumer

repo: anotherglitchinthematrix/monochrome


text:
[processBridge]: Error: Cannot find module 'ansi'

after installing the extension on Visual Studio Code on linux mint 18.1, the first launch pops the error mesage :
[processBridge]: Error: Cannot find module 'ansi'
node -v : v4.2.6
npm -v : 3.5.2
nod and npm were installed using synaptics (deb depots).
node-ansi installed (0.3.0-2)
node-ansi-color-table installed to (1.0.0-1)

Hi @xbeaudet, thank you for the report.
Could you please give me more information?
https://github.com/anseki/vscode-color/blob/master/.github/ISSUE_TEMPLATE.md

solved by : https://github.com/anseki/vscode-color/issues/10#issuecomment-270654744

Could you please report the solution?

sudo apt-get remove nodejs npm
curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs

Hi @somoza, thank you for the report. :smile:

Thank you, worked for me.

authors: PMassicotte anseki somoza xbeaudet

repo: anseki/vscode-color

license: mit


text:
catalog_collection script doesn't add plugins to cache

If the CPU count is 1

https://github.com/ansible/ansible-navigator/blob/main/share/ansible_navigator/utils/catalog_collections.py#L35

repo: ansible/ansible-navigator


text:
Make slides font, content and images more adaptive to the display resolution and content

There are many instance at lower display resolutions (like the MacBook Pro default) where the content disappears off the screen or disappears in a code box because the default font is to large. Conversely, at high resolutions there are screen where there is copious amounts of whitespace and the content is hard to see. Being HTML and in a browser the +/- controls help to a degree but even that is limited.
The content area should be more adaptive and should stretch/contract and scale to the what space is  available.
Mentions of adaptive sizing with reveal.js that I came across:

https://github.com/thoughtram/slide-deck-template
https://github.com/hakimel/reveal.js/issues/244

Thanks for your help on this @jvanpelt. Here is what feedback I have on this at the time:

Type face is too large overall at lower resolutions (See #167)
Code examples need help to make them more readable by forcing less scrolling. I realize this is a tricky painful thing to resolve, alas it important to communicating the content.

Slide 1: Title Screen

Use Title case like other decks

Slide 2: The Ansible Way

Remove footer Ansible footer
Ansible logo too large

Slide 3: Workflow

"Pull Quote" (here "Treat your Ansible content like code") is not differentiated from the slide title. This is carries thru the rest of the deck so I won't repeat it.

Slide 10: Inventory

These quasi-code examples (inventory host names) should not have bullet points and be single height lines. This carries throughout to other parts of the deck (like Slide 11) so I won't repeat it. Even at a high-resolution the font size seems a bit too large.

Slide 11: Inventory

The issues in Slide 10 are even more prevailent on this one even at a high-resolutions creen display.

Slide 15: Seperate Logic from Variables

This is a tricky painful thing, but this needs to optimize code examples to display as much as possible without scrolling. Scrolling makes it so much hard for viewers to take in examples especially when horizontal scrolling/wrapping happens

Slide 18-20: Use Native YAML Syntax

This code block should be marked as YAML for proper syntax highlighting. Pretty much any other playbook example for that matter.

Slide 33: Templates Jinja2

Split into two columns

Slide 34: Templates

Indent {{ }} example

Slide 38: Scaling your Ansible Workflow

Can we work in the Ansible Tower logo here?

repo: ansible/lightbulb


text:
Possibility for serialization within parallel processed groups

Proposal: Add possibility for serialization within parallel processed groups
Author: Your Name <@tuckson> IRC: handle (if different)
Date: 2019-07-12

Status: New
Proposal type: (core design)
Targeted release: <future release #, ex. 2.2>
Associated PR: <link to GH PR in ansible/proposals if PR was submitted>
Estimated time to implement: <X days, weeks, etc.>

Motivation
In ansible we can control how many servers are processed at the same time via the 'serial' parameter. However in a large, loadbalanced environment there is another architectural thing we need to worry about. These are the vips inside the loadbalancers. In short: many of may servers are grouped in pools in a loadbalancer. And while I do not want to process more then 2 memmbers of such a pool at the same time to avoind performance degradation for my customers, it makes perfectly sense to process servers in other pools at the same time (but also not more than 2 servers of such a pool). At this moment I see no way of achieving that inside ansible. Hence this feature request.
Problems
What problems exist that this proposal will solve?

unability to process parallel updates without bringing vips down
At current we have to either use multiple playbooks for the same task in different microservices or just accept processing is taking a long time. The requested feature would be a great help in reducing time to process certain (larger) automated processes while still keeping things safe for our customers.

Solution proposal
Suppose the top part of the playbook:
- name: Wedosomething
   hosts: allservers
   vars:
       srvdowntime: 120
   gather_facts: yes
   become: true
   serial: 2
   groups-parallel: yes

In the inventory:
[allservers:children]
Group1
Group2
Group3

[Group1]
Server[1:5]

[Group2]
Server[6:10]

[Group3]
Server[11:15]

If groups-parallel is set to 'yes' then 'serial: 2' scopes to childgroup level. In that case ansible starts and processes these servers at the same time:
Server1, Server2,Server6,Server7.Server11,Server12
(And it goes on per group until al servers are finished).
If groups-parallel is set to 'no' or is it (default) absent, than 'Serial: 2' scopes for to all hosts and ansible starts with processing
Server1, Server2
at the same time (and proceeds after that).
Without Serial groups-parallel is of no use. When you put a groupname without [:children] after 'hosts: ' neither. And the same goes for 'hosts: all'.
Dependencies (optional)
Explain any dependencies. This section is optional but could be helpful.

Dependency #1
Dependency #2

Testing (optional)
Does / should this require testing, and if so, how? Describe here. This section is optional but could be helpful.
Absolutely. Since this will require you to change the parts that process the inventory I suppose some serious testing would be nice.
Documentation (optional)
Does / should this require documentation? If so, describe here. This section is optional but could be helpful.
Yes, The documentation should describe the use and limits of the new parameter.
Anything else?
Feel free to ask for clarification

This assumes hosts are only part of one group, this is not true, they are ALWAYS part of at least 2 groups.

authors: bcoca tuckson

repo: ansible/proposals


text:
[EXPERIMENTAL] *** DO NOT MERGE *** Send Model ID Override to playbook gen/exp API

AAP-32080

We decided not to merge this PR to the main branch. I am going to close this.

repo: ansible/vscode-ansible


text:
fix: reformat the table styles

partially fixes #403
normal table

normal state
hover state
active state

data table

normal state
hover state
active state

long table -centered

normal state
hover state
active state

Table centered

normal state
hover state
active state

Data frame - sphinx gallery

normal state
hover state
active state

Dataframe -nbsphinx

normal state
hover state
active state

!

For our normal table, the size remains unchanged. For data frames, it is adjusted. However, as mentioned in this comment, there was a suggestion to change the size. I have now standardized the size accordingly. We can discuss and finalize the exact size , and i am pushing the  style adjustments.

I still can see the dataframe changing in size with the latest changes. Serve the docs with cd doc/_build/html && python -m http.server:
http://localhost:8000/examples/nbsphinx/jupyter-notebook.html#Render-a-data-frame

I still can see the dataframe changing in size with the latest changes. Serve the docs with cd doc/_build/html && python -m http.server:
http://localhost:8000/examples/nbsphinx/jupyter-notebook.html#Render-a-data-frame

ah, sorry, i didnt change the size, i changed the font family, changed the hovering size.

updated the screenshot.

Ah! Agreed. Everything working fine, now 👍🏼

repo: ansys/ansys-sphinx-theme


text:
Example proposal: For Solder ball modeling

📝 Description of the example
A script to create solderball model and also target solder for performing morphing
📁 Files needed for running the example
PyGeometry.zip
💻 Which operating system are you using?
Windows
📀 Which ANSYS version are you using?
2024r2
🐍 Which Python version are you using?
3.10
📦 Installed packages
ansys=geometry-core==0.6.6

Hi @agvarghe - what's the status here? Do you need any help?

Hi @RobPasMue
I had to lower the priorty in the last month, i plan to complete this in this week

Awesome, thanks @agvarghe!

authors: RobPasMue agvarghe

repo: ansys/pyansys-geometry


text:
Add sources in cfg format from txt file.

Description of the feature

Steps for implementing the feature
No response
Useful links and references
No response

Should this be added as a function to pyedb or not?

repo: ansys/pyedb


text:
Requesr for details on offline installation of pyfluent.

Hi all,
I'm trying to install pyfluent in a virtual desktop. The desktop is not connected to internet. could any one guide me on how to install pyfluent in offline mode?

Following this SO answer, I tested the following steps:

Download .whl files of ansys-fluent-core and its dependencies in a local directory

pip download ansys-fluent-core

Install pyfluent from the downloaded .whl files in a clean virtual environment

pip install --no-index --find-links /path/to/download/dir/ ansys-fluent-core

Then I can use pyfluent normally. I haven't tested on an offline machine, so let us know if this works.

repo: ansys/pyfluent


text:
chore(dependabot): ignore ethers for examples/packages

[中文版模板 / Chinese template]
💡 Background and solution

🔗 Related issue link

写法有点问题， 我后面再看看

参考 https://docs.github.com/zh/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#示例-3  示例三更改。

看了一下文档好像没找到不更新子包的方式，怀疑默认就是走了pnpm up --latest,要不然 pnpm add ethers@latest --workspace-root or pnpm add ethers@latest -w来手动更新根目录，然后那边就降低为minor，major的提示(减少提示频次)，patch就不去动他了。到时候我来手动更新一下根目录。
之所以不直接ignore了是为了留着提示我们这个依赖已经更新了。

ci has passed, please help me check it again. thx~ @jeasonstudio


text:
期望分享的时候能有复制图片到剪切板功能

右键可以复制图片啊

右键可以复制图片啊

哈哈是我傻了

右键可以复制图片

authors: MarvinXu linsizao

repo: antfu/handle


text:
Error when using concurrently

"scripts": {
    "renderer": "ts-node .tauri-vite/runner.js",
    "main": "tauri dev",
    "dev": "concurrently \"nr renderer\" \"nr main\""
  },

when use nr dev  main process is exit
"scripts": {
    "renderer": "ts-node .tauri-vite/runner.js",
    "main": "tauri dev",
    "dev": "concurrently \"yarn renderer\" \"yarn main\""
  },

nr dev  is ok

We temporarily close this due to the lack of enough information.
Please provide a minimal reproduction to reopen the issue.
Thanks.

Why reproduction is required

repo: antfu/ni


text:
为什么这个验签比bcprov-jdk慢至少2秒？

为什么这个验签比bcprov-jdk慢至少2秒？

是用javax.script.ScriptEngine调用的js版本的国密算法加解密，性能差点

为什么这个验签比bcprov-jdk慢至少2秒？

如上回答。

authors: Tyler-Rabbit geyiwei-hub usermtl

repo: antherd/sm-crypto


text:
To enable MM2S channel, need to know configurations

Cloned linux-readme branch and ran make verilog to generated verilog files. Able to see verilog file but by default only S2MM channel is enabled, tried configuring DMATop.scala to enable MM2S but some unknown error shown in below snapshot. Need to know how to enable MM2S channel. Below are the steps I followed:
--> git clone https://github.com/antmicro/fastvdma.git -b linux-readme
--> cd fastvdma
--> make testall (able to generate out.png)
--> make verilog

hmm, linux-readme branch is outdated. It has been merged to the main branch quite some time ago. You should not use it.
If you use the code from master branch you can set DMA config by setting env variable (as the CI script does e.g. here https://github.com/antmicro/fastvdma/blob/master/.github/scripts/test.sh#L7)
Here https://github.com/antmicro/fastvdma/blob/master/src/main/scala/DMAController/DMAConfig.scala#L53 is the list of available configs.

repo: antmicro/fastvdma


text:
feat(ci): Bump go version for action tests

Overview
Updates the Go version for the action tests job.

📚 $\text{Stack Overview}$
Pulls submitted in this stack:

#730 👈
main

This comment was automatically generated by st.

repo: anton-rs/kona


text:
feat: let custom filters be loaded from scripts

{
    fn: 'nrf-plugin-snyk'
}

:tada: This issue has been resolved in version 1.11.0 :tada:
The release is available on:

v1.11.0
npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

repo: antongolub/npm-registry-firewall


text:
Support multiple calendars per page

Support multiple calendars by appending options.id to the HTML tag id of calendar.

Thank you :)

repo: antoniovlx/leaflet-calendar


text:
. mjs configuration file error

pkg.json
  "scripts": {
    "test": "node scripts/test.mjs"
  },

scripts\test.mjs
import { lilconfig } from "lilconfig";

const res = await lilconfig("myapp").search(); // Promise<LilconfigResult>

console.log(res);

Create myapp.config.mjs in the project root directory
npm run test

> node scripts/test.mjs

node:internal/modules/esm/load:227
    throw new ERR_UNSUPPORTED_ESM_URL_SCHEME(parsed, schemes);
          ^

Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only URLs with a scheme in: file, data, and node are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
    at throwIfUnsupportedURLScheme (node:internal/modules/esm/load:227:11)
    at defaultLoad (node:internal/modules/esm/load:109:3)
    at ModuleLoader.load (node:internal/modules/esm/loader:666:12)
    at ModuleLoader.loadAndTranslate (node:internal/modules/esm/loader:479:43)
    at #createModuleJob (node:internal/modules/esm/loader:503:36)
    at #getJobFromResolveResult (node:internal/modules/esm/loader:274:34)
    at ModuleLoader.getModuleJobForImport (node:internal/modules/esm/loader:242:41)
    at async onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:542:25) {
  code: 'ERR_UNSUPPORTED_ESM_URL_SCHEME'
}

Node.js v22.11.0

Hi and thanks for the report. Unfortunately I am not able to repro this.
Seeing the backward slashes for the script in your message and the mention of c: in the error message I presume that you are using Windows OS. I do not have a windows machine to test this. My assumption is that the dynamic import gets a file path like c:\path\to\myapp.config.mjs and takes c: parts as a path scheme. Which is not the case for Windows OS. In your node modules do you mind adding console.log(id) here and re-running your script to see the full path that it outputs?
If this is correct, this should be fairly easy to fix. We can add const url = require('url') to the top of lilconfig/src/index.js and replace the dynamic import line with this
-		const mod = await import(/* webpackIgnore: true */ id);
+		const fileUrl = url.pathToFileURL(id).href;
+		const mod = await import(/* webpackIgnore: true */ fileUrl);

You can try this out by manually applying this change in your project's node_modules/lilconfig/src/index.js and rerunning your script.
Please let me know if this fixes the issue for you? In this case I am happy to release a new version that fixes this.

The fixed is published to npm in v3.1.3
Thanks for the report


text:
Save config on options window closing event

Now configuration should be saved when options window is closed.

List any issues that this PR fixes: None

This pull request proposes the following changes:

Changes code to save global configuration on options window of any device in device manager.

Known issues/To do:

None

I tested it and can confirm, that it is working.

authors: SnakePin gitmacer

repo: antonpup/Aurora


text:
Tabli popup window throws exception on open attempt

Error in response to storage.set: Error: Invalid value for argument 1. Property 'type': Value must be one of: [normal, popup, panel].
    at Object.callback (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/common.js:5:18916)
    at Object.u [as showPopout] (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/common.js:5:18870)
    at Object.I [as showPopout] (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/common.js:5:16830)
    at Object.handlePopoutClick (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/tabliPopup.bundle.js:19:16381)
    at Object.r (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/tabliPopup.bundle.js:1:20975)
    at a (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/tabliPopup.bundle.js:1:19316)
    at Object.s [as executeDispatchesInOrder] (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/tabliPopup.bundle.js:1:19531)
    at f (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/tabliPopup.bundle.js:1:15923)
    at v (chrome-extension://igeehkedfibbnhbfponhjjplpkeomghi/js/tabliPopup.bundle.js:1:16049)

Related error from Inspect Popup context-option. Unfortunately not super helpful due to bundler. Popup window does not display with either Ctrl-, or manually clicking the button in the pop-out.
Version 0.9.4

Yikes!  Thanks for letting me know.  Investigating.
What OS?

OK, I see the issue.
In pact.js when opening the popup window, Tabli calls chrome.windows.create() with a type of 'detached_panel'.  Detached panels were an experimental Chrome feature that apparently has been removed.  It's supposed to just revert to the supported value 'popup', but apparently doesn't depending on Chrome version / OS. (Works fine for me on Chrome 55, macOS Sierra).

Fix just committed.  Will roll a new release.

@miwu Thanks for confirming that the popout bug is fixed in 0.9.5.
You are 100% correct that the popout window doesn't have reasonable resize behavior (but should)!
This is a holdover from before the popup window was implemented; popups in Chrome extensions have a fixed maximum size.
I will open an issue for popout resizing, though I can't make any promises about when I'll get to it.
Sizing and layout issues in CSS are notoriously fiddly, and it's even more challenging in the context of extensions -- see, for example: issue 45

authors: antonycourtney miwu

repo: antonycourtney/tabli


text:
Pogostick option

I think a really useful tool to have under the utilities section would be PogoStick, I use it pretty often, & can PXE boot it locally, but am unsure if it can be made to work with netboot.xyz
When unpacked, the Pogostick ISO takes up about 16 MB

have tested & managed to get my normal PXE server to boot a local copy of the ISO with memdisk!
Now to try opening the iso over t'internet...
The main problem is that it's stored on the website as a ZIP, do you know if that can be unpacked on the fly?

I gave it a try and it appears that MEMDISK decompressed the zip on the fly (learned something new) so it got to the initial menu, but then promptly kernel crashed in Fusion.  It did fully load in Virtual Box though and got to the menu.  I'll mock up an option and you can give it a try via the test branch.

Great, thanks! :) Also CC'd you in on an intro email to the project owner, in the interest of keeping everyone in the loop

https://github.com/antonym/netboot.xyz/pull/34

successfully tested on a ~3 yr Alienware, will test more thoroughly tomorrow if needed :)

https://github.com/antonym/netboot.xyz/pull/34


text:
feat: add mark.liquid

add liquid

Pull Request Test Coverage Report for Build 6141572391

118 of 125 (94.4%)  changed or added relevant lines in 9 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.006%) to 88.564%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/mark/gauge.ts
4
6
66.67%

src/shape/liquid/liquid.ts
35
37
94.59%

src/shape/liquid/wave.ts
39
42
92.86%

Totals

Change from base Build 6070169865:
-0.006%

Covered Lines:
9934

Relevant Lines:
10853

💛  - Coveralls

数据格式支持两种：

直接数据（主要的写法，文档、demo 只透漏这一种）

.data(0.75)

只是为了兼容旧的

.data({ percentage: 0.75 })

authors: ai-qing-hai coveralls hustcc


text:
Behavior的drag-node的shouldEnd不起作用

问题描述
<template>

      默认模式
      编辑模式

</template>

shouldBegin有效，但是shouldEnd无法被触发
重现链接
无
重现步骤
正常拖拽节点
预期行为
期望shouldEnd能正常触发
平台

操作系统: [Windows]
网页浏览器: [Google Chrome]
G6 版本: [4.8.17 ]
vue版本[3.3.4]

屏幕截图或视频（可选）
No response
补充说明（可选）
No response

我在官网上是可以触发的呀，你要不写成在线复现的 demo


text:
官网的飞行图例子，如何给飞行线路中间加数据标注？

https://l7.antv.antgroup.com/zh/examples/line/animate/#plane_animate2
请教各位大佬，官网这个例子，如何给线段中间加数据标注？

计算中心点，单独标注

计算中心点，单独标注

好的，两个点的中心点，应该就是用经纬度计算一下吧，再请问一下，那这个是有贝塞尔曲线的，如何计算偏移量呢？有方法吗？

float bezier3(vec3 arr, float t) {
float ut = 1. - t;
return (arr.x * ut + arr.y * t) * ut + (arr.y * ut + arr.z * t) * t;
}
https://github.com/antvis/L7/blob/master/packages/layers/src/line/shaders/line_arc_vert.glsl#L60  参考这里

float bezier3(vec3 arr, float t) { float ut = 1. - t; return (arr.x * ut + arr.y * t) * ut + (arr.y * ut + arr.z * t) * t; }
https://github.com/antvis/L7/blob/master/packages/layers/src/line/shaders/line_arc_vert.glsl#L60 参考这里

谢谢，arr应该是坐标，参数t是什么？

authors: jiazehua lzxue


text:
Mapbox底图绘制”线路围墙“高度设置无效

L7 Version:  2.13.9

Platform: windows 11

Mini Showcase(like screenshots):

CodePen Link: 无

Mapbox底图绘制”线路围墙“出错，围墙的高度无法设置，永远是无限大。
将官网示例换成Mapbox底图即可复现：
import { Scene, LineLayer } from '@antv/l7';
import { Mapbox } from '@antv/l7-maps';

const scene = new Scene({
  id: 'map',
  map: new Mapbox({
    center: [ 121.461531, 31.096775 ],
    zoom: 8.64,
    rotation: 358.78,
    pitch: 45.42056074766357,
    style: 'https://thope.tianhengyun.com/api/styles/v1/admin/ZPUdKizRzR'
  })
});

scene.addImage(
  '02',
  'https://gw.alipayobjects.com/zos/bmw-prod/ce83fc30-701f-415b-9750-4b146f4b3dd6.svg'
);

scene.on('loaded', () => {
  fetch(
    'https://gw.alipayobjects.com/os/bmw-prod/0290a972-eedd-42f6-b69e-50a35e8a0824.json'
  )
    .then(res => res.json())
    .then(data => {
      const layer = new LineLayer({})
        .source(data.slice(0, 50), {
          parser: {
            type: 'json',
            coordinates: 'coordinates'
          }
        })
        .animate({
          interval: 1, // 间隔
          duration: 1, // 持续时间，延时
          trailLength: 2 // 流线长度
        })
        .size(20)
        .shape('wall')
        .texture('02')
        .style({
          opacity: 'testOpacity',
          lineTexture: true, // 开启线的贴图功能
          iconStep: 40, // 设置贴图纹理的间距
          iconStepCount: 4,
          sourceColor: '#00BCD2',
          targetColor: '#0074d0'
        });
      scene.addLayer(layer);
    });
});

值调整一下就好了，或者设置一下 heightfixed

这是一个明显的bug，size设置成0.01的实际意义是什么呢，一个一个试吗？而且也只能在当前级别有效，缩放后还是会出现问题。

发生了同样的问题，求解决

发生了同样的问题，求解决

size 是宽度，你具体需要的是什么效果，可以附个 在线demo

应该是高度单位没有统计，这边跟进一下，有兴趣的同学可以 PR

在使用mapbox地图，添加LineLayer图层时，在部分层级下坐标会发生偏移。可以看到在缩放时，黄色的LineLayer和红色的mapbox line图层有个很明显的偏移
https://github.com/antvis/L7/assets/59105862/d85afdd6-29d5-4819-ae01-ecc58cdb7835

在使用mapbox地图，添加LineLayer图层时，在部分层级下坐标会发生偏移。可以看到在缩放时，黄色的LineLayer和红色的mapbox line图层有个很明显的偏移
L7bug.mp4

应该是缩放等级 到了12 等级切换是出现的

authors: Azusa89757 lzxue wanyanyan wfJoanna yiiiiiiqianyao


text:
import { type 'xxx' } from './xxx' TS:1005: ',' expected

问题描述

@antv/l7: 2.17.3 版本，安装的依赖是commonjs 的包，<= 2.17.3 应该是esnext 的包， ts 会报错；使用 <= 2.17.3 的包 会出现这个问题 https://github.com/antvis/L7/issues/1782

重现链接
No response
重现步骤
No response
预期行为
No response
平台

操作系统: [macOS, Windows, Linux, React Native ...]
网页浏览器: [Google Chrome, Safari, Firefox]

屏幕截图或视频（可选）
No response
补充说明（可选）
No response

升级最新版本 2.22.0 或 2.21 吧 @zhangjuncheng1

我试试

升级 typescript 5 以上

项目中很多是基于typescript4的依赖，要是能基于typescript4发一个版本就好了

4 支持这样写吗 import type { 'xxx' } from './xxx'，可以的话可以来帮忙改下

我看从3.8 开始支持的，应该可以

我看从3.8 开始支持的，应该可以

可以来贡献一下，提交一个 PR


text:
当graph中没有元素时，不会触发render:done事件

Describe the bug
当graph中没有元素时，不会触发render:done事件
Your Example Website or App
none
Steps to Reproduce the Bug or Issue
none
Expected behavior
正常触发render:done事件
Screenshots or Videos
No response
Platform

OS: Windows
Browser: electron
Version: [e.g. 2.11.1]

Additional context
No response

没有元素为啥要触发 render:done 事件呢？

@NewByVector 会更统一
大多数人没想到这个，因此可能用了 render:done ，但没想到没有触发一些必要的初始化操作

只有元素渲染是异步的，才需要 render:done 事件。所以没有元素不触发是比较合理的。

render:done只是从字面意思就是渲染结束，并没有说是某个元素或者node的渲染，没有元素graph自然也就渲染结束了，用户可以统一做后续处理，就像html渲染结束并不意味着html中必须有任何元素
不过可能是牛角尖了，也可以close，或者记录为次要

render:done只是从字面意思就是渲染结束，并没有说是某个元素或者node的渲染，没有元素graph自然也就渲染结束了，用户可以统一做后续处理，就像html渲染结束并不意味着html中必须有任何元素
不过可能是牛角尖了，也可以close，或者记录为次要

我认同这种说法，应该有一个画布准备完毕的回调函数，或者fromJSON应该有一个回调函数是整体渲染完毕的回调

render:done 其实给我的感觉就是ready，感觉应该需要。

render:done事件,好像每个节点完成了都会触发吧,我想要的是一个全部渲染完成后的事件

我们需要流程图数据进行计算,如果没有全部渲染完成后的事件,有时初始化计算的时候会因为拿不到画布的数据而出现报错

我的流程图有的是会节点数量+1次调用这个函数,有的就会正确的只调用一次这个函数,我想知道区别是什么

迷惑了  这个东西还会触发n次  因为开启了async  有没有只触发一次的api 啊。 既然都done了，还触发多次，感觉有歧义

迷惑了 这个东西还会触发n次 因为开启了async 有没有只触发一次的api 啊。 既然都done了，还触发多次，感觉有歧义

我也遇到了这个问题😂请问您最后是怎么解决的

authors: America-first-melon NewByVector apankun bighhhh cuidong626 pygc wenhaoxu1


text:
AntV f2 使用交互行为引入相应的行为依旧报错

F2 Version:

Platform: 移动端 Vue

错误信息如下：

恩，这个问题我们同学也遇到了。你尝试下如下解决方案：

删除 node_modules, 重新安装最新版 beta 版本 F2 3.2.0-beta.8
按需引入的话，全部引用 @antv/f2/lib 下的代码模块，如下：

import F2 from '@antv/f2/lib/index';
require('@antv/f2/lib/interaction/pan');

@simaQ 好的 ，我试一下 . 我试着输出chart对象，没有行为这个属性方法

@simaQ  谢谢，昨天已经解决。

你好，我在vue里面使用使用
import F2 from '@antv/f2/lib/index';
require('@antv/f2/lib/interaction/pan');
这样引入，解决了'interaction' is not a function这个报错，但是又出来一个新的报错：Ctor is not a constructor"。
你是怎么解决的呀？请教一下


text:
Extreme Performance Issues With 600+ Nodes & Edges

Compared to G6 (GGEditor) and cytoscapeJS with similar data structure the load of graph is almost instant (< 1 second).
Here is example static page I created with the same data across 3 different graph frameworks.
Graph Tests
Created also example page with X6 problem:
https://x6grapheditor.z16.web.core.windows.net/
Expected Behavior

the data to be loaded instantly and zoom in/out is smooth with 600+ nodes & edges

Current Behavior

when trying to zoom out the page completely freezes for good minute and each action afterwards is beyond slow

Steps To Reproduce

clone X6 repo
cd x6/examples/x6-example-drawio
npm install
add JSON data file to project:
https://pastebin.com/jiwuQ98G
modify editor.ts file to render data from JSON:

  renderGraph() {
    const graph = this.graph
    graph.render(data)
  }

  start() {
    this.renderGraph()
  }

npm run start

Additional Context
This data with nodes & edges is real life example I'm using in my projects. I have anonimized the data so I can put it publicly.
Your Environment

cloned official example of this repository
x6/examples/x6-example-drawio

node: 10.15.0

npm: 6.11.3

x6: 0.3.0

OS: Windows 10 Ver 1909

Browser: Chrome Version 80.0.3987.149 (Official Build) (64-bit)

@bubkoo any ideas what is causing this?

@avilde  you can use G6:  https://github.com/antvis/g6。

@dxq613 I have already tested out both G6 and GG-Editor.
Your suggestion does not answer the issue.

@avilde The possible reason may be each node/edge will be re-rendered when zoom in/out, we are working on only scale the top-most group g instead of re-rendering each nodes.

authors: avilde bubkoo dxq613


text:
Juggling Act: Various updates to stripe-scala.

This includes a handful of updates to stripe-scala that gets it in line with the API.
This gets specs passing with the 2014-01-31 API, including the following changes:

Bump to 2.10 only so I could use string interpolation.
Ability to explicitly specify apiVersion.
Updates to Charge to support dispute.
Updates to how invoice line items are presented so its in sync with the API.

Cheers!

Please merge it in and rollout a public 2.10 build :+1:

If you're able to use 2.11 I started rolling a Stripe library of my own: https://github.com/farmdawgnation/streifen
Still not 100% complete and needs some updates for the latest API version, but it shouldn't be too far off the mark.
This stripe-scala library seems to be abandoned...

Unfortunately not all of my project's dependencies have been released for 2.11.
Do you know of any Stripe library compatible with 2.10?
Yep, seems like it...

No, sorry. Mine can't be backported due to the 22-param case class limit. :(

Closing this due to age.

authors: farmdawgnation phedoreanu

repo: anurag/stripe-scala


text:
🛠FIX : Literary Realms carousel's autoplay added

Related Issue
“None”
Fixes:  #4677
Description
Literary Realms carousel is now auto-playing.
Type of PR

[x] Bug fix
[x] Feature enhancement

Screenshots / videos (if applicable)
https://github.com/user-attachments/assets/747042cc-6b9e-49d1-be0c-6281c158af7a
Checklist:

[x] I have made this change from my own.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] My changes generate no new warnings.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers and screenshots after making the changes.

@abhi03ruchi  please review and merge this PR with the level 2 label.


text:
Added a chatbot using Dialogflow to enhance user interaction on the website.The updates improve user interaction by providing a more guided experience through the addition of follow-up suggestions and context management. #Issue-1248

Changes Made

Integrated the Dialogflow agent into the homepage.
Ensured that the chatbot UI is user-friendly and positioned at the bottom left of the page.
Fixed whitespace issues in the HTML file.

Training Phrases Added

Enhanced the following intents with new training phrases to capture user inquiries accurately:

Services Inquiry Intent:

"Tell me about your services"
"What services do you offer?"
"Services"
"I want to know about services"

Events Inquiry Intent:

"Tell me about your events"
"What events are coming up?"
"How can I register for events?"
"Upcoming events"

Contact Inquiry Intent:

"How can I contact you?"
"What’s your contact information?"
"Contact details"

How to Test

Open the chatbot on the WildGuard website.
Start by typing a greeting or simply click on the chat window.
Follow the initial greeting suggestions:

Type "services" to inquire about the available services.
Type "events" to get information on upcoming events.
Type "contact" to retrieve contact information.

Additional Information
These updates aim to enhance user engagement and streamline information access. By guiding users through common queries, the chatbot will now provide a more intuitive experience for visitors to WildGuard.

@sohailshk Please resolve the conflicts in the mentioned files. Ensure you pull all the latest changes before pushing your feature/bug fixes to the main branch. After resolving the conflicts, please open a new PR for review.
Thank you!

authors: rahulkhandait-sde sohailshk


text:
Fix Docker file

Fixes #666 .
Changes proposed in this pull request:

Move Dockerfile into its own directory -- smaller build context -> faster build
Use Debian Buster instead of Alpine. Fixes previous startup problems
Build sysl inside container instead of copying unknown version
Using Debian makes it easier to build on top of this image using apt

Buld image with
 docker image build -t anzbank/sysl:<tag> docker-build-context

Notes: This version defaults to Go 1.4 and Sysl 0.7.1. Override version
defaults with build-args.
My image size is  96.9MB, which seems OK to me. The current official image is 379MB but it does not work.
Checklist:

[ ] Added related tests  -- N/A
[ ] Made corresponding changes to the documentation  N/A

@anz-bank/sysl-developers

Use Debian Buster instead of Alpine. Fixes previous startup problems

What problems are these? Is this the issue described in #666? If so, this is not a good "fix" - a Dockerfile should never depend on the outside build environment, so even if this PR would make that work, lets not document that.

Using Debian makes it easier to build on top of this image using apt

You could say that it should instead use a redhat image as that would be easier to build on top of using yum/rpm. My point here is that introducing Debian into the image for personal reasons should not be done. It needs a good justification. And keeping a small tight image (alpine) is generally preferable that a larger one. A user who wants a Debian-based image could easily build their own.
Some other comments incoming on the specific of the commit...

You could say that it should instead use a redhat image as that would be easier to build on top of using yum/rpm. My point here is that introducing Debian into the image for personal reasons should not be done. It needs a good justification. And keeping a small tight image (alpine) is generally preferable that a larger one. A user who wants a Debian-based image could easily build their own.

This is not a "personal preference".

The apt package manager is very well understood, apk is not. It is likely that Sysl image will be tailored by users with additional tools for their own environment. Apt makes this easier.

Alpine has had security issues and many people are nervous about using it.

What problems are these? Is this the issue described in #666? If so, this is not a good "fix" - a Dockerfile should never depend on the outside build environment, so even if this PR would make that work, lets not document that.

The problem you described in #666. The Alpine image will need additional libs to work. Debian does not.

Just notice I skipped pandoc install.

The Alpine image will need additional libs to work. Debian does not.

Additional libs are not needed - it just needs to be built in an alpine container if it going to be deployed in an alpine container. For example, the following works in alpine (after removing pkg from .dockerignore):
FROM golang:1.13.8-alpine3.11 as builder

WORKDIR /src
COPY . .
RUN go install ./...

FROM alpine:3.11.3
COPY --from=builder /go/bin/sysl /sysl
ENTRYPOINT ["/sysl"]
CMD ["help"]

This is not a "personal preference".

Sorry, I didn't mean to imply anything in particular here, just that with no justification for a change, I can only assume a personal preference.
If you would like to change the base image for the sysl container image from alpine to debian, that can be discussed, but should be done independently of getting the Dockerfile working.

Alpine has had security issues and many people are nervous about using it.

Sorry if this is too blunt, but this comment is just too hand-wavey. If there are specific issues, they should be pointed out. Alpine is often chosen as a base as it presents a smaller attack surface than a larger distro like Debian. If alpine has security issues, I'd be more inclined to go distroless (https://github.com/GoogleContainerTools/distroless) than add a fat base.

It is likely that Sysl image will be tailored by users with additional tools for their own environment.

I'd like to understand this more before accepting it. The sysl container image is there to make it easy to run the sysl binary. It has defined inputs and outputs. Any pre-processing of the input or post-processing of the output does not need to be done in the sysl container image.
This is not to say there may not be use cases for a broader image, but I'd like to understand them first before making a change to move in that direction.

So I've been working with the Alpine image and getting Plantuml installed (well Java really) is a PITA. I don't want to go through that

Good. I don't think it's really necessary. The container image should just contain sysl, IMHO. PlantUML is different software that belongs in its own image. I usually run the sysl tests using plantuml/plantuml-server:tomcat from dockerhub when I don't want to use the public www.plantuml.com url.

Still using debian base image because it's too hard to install plantuml in Alpine
Still using Git Checkout because

It seems better practice to build directly from a know tag instead of what could be lying around in the build context
I can't build successfully from the build context on my machine -- be great if someone else could test

I think we are there?

Build:
  Version      : a20c98f-docker-build-2020-03-06T01:03:12Z
  Git Commit   : a20c98f
  Date         : 2020-03-06T01:03:12Z
  Go Version   : unspecified
  OS           : unspecified
Runtime:
  GOOS/GOARCH  : linux/amd64
  Go Version   : go1.13.8

That seems to be fixed

If you can just fix those two outstanding issues, I'll merge this PR.

If this PR were to be merged as is, the goreleaser machinery would break because when it builds the dockerfile it does not pass the build arg for the sysl tag. While I strongly believe having the docker build only be able to build official source from existing refs is wrong, this PR is still incomplete without updating the goreleaser config.

Not sure what you need here? Is this fixed by using make to build?

Not sure what you need here? Is this fixed by using make to build?

You can ignore that. It was an out-of-date comment based I think on my review crossing with your commit. It was referencing that the docker build required an argument and would fail without it. The goreleaser configs were not providing that argument. But it is moot now that the argument is not required.

K -- so we are done? I think I've fixed the rest

K -- so we are done? I think I've fixed the rest

Still waiting for those two gitignore changes.

Already done AFIK
**/node_modules
/dist/
/pkg/antlr*.jar
/release/
/scripts/generate

Already done AFIK

That's odd. I don't see that change. When I look at the "Files Changed" tab, I still see the original content, and github is not marking the review comments as "Outdated". Can you double-check you've pushed your latest commit?

Already done AFIK

That's odd. I don't see that change. When I look at the "Files Changed" tab, I still see the original content, and github is not marking the review comments as "Outdated". Can you double-check you've pushed your latest commit?

Looks good on GitHub?
https://github.com/alecthegeek/sysl/blob/docker-build/.dockerignore

Looks good on GitHub?
https://github.com/alecthegeek/sysl/blob/docker-build/.dockerignore

This is what I see. And I see it from two separate computers. What do you see?

Something went squiffy.
I pulled merged and pushed. Seems to be ok now?

authors: alecthegeek camh-anz

repo: anz-bank/sysl


text:
Msgpack Support

Initial pr for msgpack support, hang tight a bit, I'll add some more commits just now.

Ok basic MsgPack support is there.
just a thought I have it as res.WriteMsgpack, do you think it should be res.WriteMsgPack instead?

I prefer the Response#WriteMsgpack().
Msgpack -> MessagePack
Protobuf -> ProtocolBuffers

https://stackoverflow.com/questions/6888900/mime-type-for-msgpack
We can add support for both, since msgpack is a lesser known format and people will have varying opinions.

From StackOverflow

From Wikipedia : According to RFC 6838 (published in January 2013 : http://tools.ietf.org/html/rfc6838),
any use of types in the "x." tree is strongly discouraged. Media types with names beginning with "x-" are
no longer considered to be members of this tree since January 2013.
Then use directly application/msgpack

However, if the mime-type is still a concern we could add in the following, just to give people the benefit of the doubt:
switch mt {
case "application/json":
	err = json.NewDecoder(r.Body).Decode(v)
case "application/msgpack", "application/x-msgpack":
	err = msgpack.NewDecoder(r.Body).Decode(v)
case "application/xml":
	err = xml.NewDecoder(r.Body).Decode(v)
case "application/x-www-form-urlencoded", "multipart/form-data":
	err = b.bindParams(v, r.Params())
default:
	return errors.New("unsupported media type")
}

Not sure how to go about it with responses though

case "application/msgpack", "application/x-msgpack":, I think we should do this.

authors: SaulDoesCode aofei

repo: aofei/air


text:
Cannot make addons

It won't let me use Origins in my mod, as when I attempt to add it, I get errors when I build my mod:
A problem occurred configuring root project 'MOD NAME HERE'.
> Could not resolve all files for configuration ':modCompileClasspath'.
   > Could not resolve com.terraformersmc:modmenu:2.0.0-beta.7.
     Required by:
         project : > com.github.MerchantPug:origins-fabric:282d1384ff
      > Could not resolve com.terraformersmc:modmenu:2.0.0-beta.7.
         > Could not get resource 'https://dl.bintray.com/adriantodt/maven/com/terraformersmc/modmenu/2.0.0-beta.7/modmenu-2.0.0-beta.7.pom'.
            > Could not GET 'https://dl.bintray.com/adriantodt/maven/com/terraformersmc/modmenu/2.0.0-beta.7/modmenu-2.0.0-beta.7.pom'. Received status code 403 from server: Forbidden
   > Could not resolve net.adriantodt.fabricmc:fallflyinglib:3.0.0-beta.2.
     Required by:
         project : > com.github.MerchantPug:origins-fabric:282d1384ff > com.github.apace100:apoli:v1.0.3
      > Could not resolve net.adriantodt.fabricmc:fallflyinglib:3.0.0-beta.2.
         > Could not get resource 'https://dl.bintray.com/adriantodt/maven/net/adriantodt/fabricmc/fallflyinglib/3.0.0-beta.2/fallflyinglib-3.0.0-beta.2.pom'.
            > Could not GET 'https://dl.bintray.com/adriantodt/maven/net/adriantodt/fabricmc/fallflyinglib/3.0.0-beta.2/fallflyinglib-3.0.0-beta.2.pom'. Received status code 403 from server: Forbidden
   > Could not resolve io.github.onyxstudios.Cardinal-Components-API:cardinal-components-base:3.0.0.
     Required by:
         project : > com.github.MerchantPug:origins-fabric:282d1384ff > com.github.apace100:apoli:v1.0.3
      > Could not resolve io.github.onyxstudios.Cardinal-Components-API:cardinal-components-base:3.0.0.
         > Could not get resource 'https://dl.bintray.com/adriantodt/maven/io/github/onyxstudios/Cardinal-Components-API/cardinal-components-base/3.0.0/cardinal-components-base-3.0.0.pom'.
            > Could not GET 'https://dl.bintray.com/adriantodt/maven/io/github/onyxstudios/Cardinal-Components-API/cardinal-components-base/3.0.0/cardinal-components-base-3.0.0.pom'. Received status code 403 from server: Forbidden
   > Could not resolve io.github.onyxstudios.Cardinal-Components-API:cardinal-components-entity:3.0.0.
     Required by:
         project : > com.github.MerchantPug:origins-fabric:282d1384ff > com.github.apace100:apoli:v1.0.3
      > Could not resolve io.github.onyxstudios.Cardinal-Components-API:cardinal-components-entity:3.0.0.
         > Could not get resource 'https://dl.bintray.com/adriantodt/maven/io/github/onyxstudios/Cardinal-Components-API/cardinal-components-entity/3.0.0/cardinal-components-entity-3.0.0.pom'.
            > Could not GET 'https://dl.bintray.com/adriantodt/maven/io/github/onyxstudios/Cardinal-Components-API/cardinal-components-entity/3.0.0/cardinal-components-entity-3.0.0.pom'. Received status code 403 from server: Forbidden
   > Could not resolve io.github.ladysnake:PlayerAbilityLib:1.3.0-nightly.1.17-rc1.
     Required by:
         project : > com.github.MerchantPug:origins-fabric:282d1384ff > com.github.apace100:apoli:v1.0.3
      > Could not resolve io.github.ladysnake:PlayerAbilityLib:1.3.0-nightly.1.17-rc1.
         > Could not get resource 'https://dl.bintray.com/adriantodt/maven/io/github/ladysnake/PlayerAbilityLib/1.3.0-nightly.1.17-rc1/PlayerAbilityLib-1.3.0-nightly.1.17-rc1.pom'.
            > Could not GET 'https://dl.bintray.com/adriantodt/maven/io/github/ladysnake/PlayerAbilityLib/1.3.0-nightly.1.17-rc1/PlayerAbilityLib-1.3.0-nightly.1.17-rc1.pom'. Received status code 403 from server: Forbidden

See the updated workspace setup guide: https://origins.readthedocs.io/en/latest/guides/addon/workspace_setup/


text:
AMBARI-24847 - Logsearch: Cannot search for a term that includes spaces or dashes

What changes were proposed in this pull request?

On Logsearch UI if a sentence like "Some terrible error has happened" is used for message filtering a huge amount of irrelevant log entries is pulled, this is mainly because the search term is tokenized and finds every line that contains any of the words.
The UI replaces all whitespace characters in the search terms to + char. It is a boolean operator: The + symbol (also known as the "required" operator) requires that the term after the + symbol exist somewhere in a field in at least one document in order for the query to return a match.
Special solr query characters can not be used in search terms because the log_message field is tokenized by StandardTokenizer and it eliminates the special characters.

Fix.:

switch to WhiteSpaceTokenizer for log_message field.
do not replace whitespace chars in the search term on the UI
phrase search terms can be specified using " like:
Message:"Some terrible error has happened (line:42)"

Integration tests for query converters are added.
How was this patch tested?
UTs and ITs re passed
Manually:

Deploy ambari and a cluster: logsearch, zookeeper, infra solr
Login to Logsearch and performs searches: specify search term for the Message field
Check the result contains only the relevant entries

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/30/
Test PASSed.

authors: asfgit kasakrisz

repo: apache/ambari-logsearch


text:
[Improvement]: The commit time of the file is displayed incorrectly

Search before asking

[X] I have searched in the issues and found no similar issues.

What would you like to be improved?

1970-01-01 07:59:59 is a strange time.
How should we improve?
I think it is OK not to show the commit time when api return -1.
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Subtasks
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

If this issue not solved yet, let me take this.

repo: apache/amoro


text:
docs: Why Is Apache APISIX the Best API Gateway?

Changes:
added a blog "Why Is Apache APISIX the Best API Gateway?"

Hi @SkyeYoung, can you please check what happened on Netlify? It failed again 🤔。

Hi @SkyeYoung, can you please check what happened on Netlify? It failed again 🤔.

The GitHub token on Netlify expired, and I've regenerated a new one.

authors: SkyeYoung juzhiyuan

repo: apache/apisix-website


text:
Compute error: Overflow happened in tpcds_logical_q58

Describe the bug
after disable skip_failed_rules.
tpcds_logical_q58 failed in
'simplify_expressions' failed due to unexpected error: Arrow error: Compute error: Overflow happened on: 100 * 100000000000000000000000000000000000000"
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.

I investigated it, I found bug in ConstEvaluator.
It will failed in expr.rewrite(&mut const_evaluator)?

cc @HaoYang670 .
After we fix this problem, we can disable skip_failed_rules

After we fix this problem, we can disable skip_failed_rules
This is really a great news! Great works @jackwener.

failed in "CAST(Int64(100) AS Decimal128(38, 38))"
"ArrowError(ComputeError("Overflow happened on: 100 * 100000000000000000000000000000000000000"))"

We've dealt with such issue in #3690.

failed in expression "CAST(Int64(100) AS Decimal128(38, 38))"

This sounds like a type inference bug, as opposed to something related to the handling of decimal arithmetic. In particular where has this scale of 38 come from? Such a datatype cannot represent anything greater 1 and so the query is perfectly correct to bail out with an error.
The original schema has a precision of 2, as it represents a monetary quantity, and so ending up with a scale of 38 is extremely suspect...

skip_failed_rules is now disabled by default by #6265
https://github.com/apache/datafusion/blob/19356b26f515149f96f9b6296975a77ac7260149/datafusion/common/src/config.rs#L546-L549
And tpcds_logical_q58 is not ignored and runs successfully in test suite:
https://github.com/apache/datafusion/blob/19356b26f515149f96f9b6296975a77ac7260149/datafusion/core/tests/tpcds_planning.rs#L319-L322
Closing issue

authors: HaoYang670 Jefffrey jackwener tustvold viirya

repo: apache/arrow-datafusion


text:
Poor reported performance of DataFusion against DuckDB and Hyper

Describe the bug
There is a blog post that reports relatively poor performance of DataFusion compared to DuckDB and Hyper:
https://www.architecture-performance.fr/ap_blog/tpc-h-benchmark-of-hyper-duckdb-and-datafusion-on-parquet-files/
To Reproduce
I would like someone to try and reproduce the DataFusion performance reported in the blog and propose ways to improve the performance of DataFusion (perhaps by enabling some of the options that are off by default)
Expected behavior
No response
Additional context
@Dandandan  suggests on slack that with parallelized scan of a parquet file this benchmark may go faster

There was a question on slack asking if the Python bindings enable parallel scans.
The Python SessionContext constructor is here. It uses SessionConfig::default if no config is provided, so yes, parallel scans should be enabled if a recent version is being used.
https://github.com/apache/arrow-datafusion-python/blob/main/src/context.rs#L216-L233

Looking at https://github.com/apache/arrow-datafusion/blob/main/dev/changelog/19.0.0.md , it should be enabled since 19.0.0, while the article mentions it's using 20.0.0

FYI @djfrancesco we are looking into your article.
Is there any code you could share to replicate the results?

I am sorry because the code is a little messy... I might have use DataFusion in a wrong way. I tried several config options but without success. I write these blog posts mostly as notes to myself. I would be glad to update the post with better timings. The main command is the following:

No worries -- clearly if you can't get good performance out of DataFusion we have some things to improve (even if it is only UX).
Thank you for the article

Anybody has a preliminary idea why the figures look this bad? Mostly config related, algorithmic issues or code structure/performance related?

Related (in a sense that trying Datafusion with defaults yields questionable results): https://github.com/apache/arrow-datafusion/discussions/5141

Looks like the blog is broken, I can not open it now.

I took a quick look at the code, and have two observations:

The configs being created are not actually being used anywhere i.e. they are not being passed into the context
There is a comment that DataFusion does not support multiple Parquet files for a table, which is not true, but maybe this means the DataFusion tests are not comparable to the other tests if they are being run against one parquet file per table?

I created a quick PR to improve the docs around creating a context and will spend more time on this over the next few days. The UX could be improved.
https://github.com/apache/arrow-datafusion-python/pull/321

Thanks for your help @andygrove! I tried versions 20 and 21 and got this Error when using a SessionConfig instance:
      config.set("datafusion.execution.parquet.pushdown_filters", "true")
      ^^^^^^^^^^
  AttributeError: 'datafusion.SessionConfig' object has no attribute 'set'

Maybe it is a new feature ?

Hi @djfrancesco, how many partitions do you set for each engine? Are they the same? And could you share the cpu usage?

FWIW I am running the scripts in https://github.com/aetperf/sql_olap_bench now on my own machine (they are very nicely written @djfrancesco ) and will report back any interesting findings

So I have two interesting pieces of information:

I can reproduce the reported performance difference on my 8 core cloud machine

I also see DataFusion using only a single core
+--------------+--------------+----------+-----------------+-------------------+---------------------+--------------------+--------------+----------+-------------+
| l_returnflag | l_linestatus | sum_qty  | sum_base_price  | sum_disc_price    | sum_charge          | avg_qty            | avg_price    | avg_disc | count_order |
+--------------+--------------+----------+-----------------+-------------------+---------------------+--------------------+--------------+----------+-------------+
| A            | F            | 37734107 | 56586554400.73  | 53758257134.8700  | 55909065222.827692  | 25.522005853257337 | 38273.129734 | 0.049985 | 1478493     |
| N            | F            | 991417   | 1487504710.38   | 1413082168.0541   | 1469649223.194375   | 25.516471920522985 | 38284.467760 | 0.050093 | 38854       |
| N            | O            | 74476040 | 111701729697.74 | 106118230307.6056 | 110367043872.497010 | 25.50222676958499  | 38249.117988 | 0.049996 | 2920374     |
| R            | F            | 37719753 | 56568041380.90  | 53741292684.6040  | 55889619119.831932  | 25.50579361269077  | 38250.854626 | 0.050009 | 1478870     |
+--------------+--------------+----------+-----------------+-------------------+---------------------+--------------------+--------------+----------+-------------+
4 rows in set. Query took 2.841 seconds.
❯

However, when I run datafusion against "datafusion created" parquet files from https://github.com/apache/arrow-datafusion/tree/main/benchmarks it is 3x faster though much less fast than hyper (like 100ms vs 928ms)
alamb@aal-dev:~/tpch_data/parquet_data_SF1$ datafusion-cli -f ~/sql_olap_bench/q1.txt
+--------------+--------------+-------------+-----------------+-------------------+---------------------+-----------+--------------+----------+-------------+
| l_returnflag | l_linestatus | sum_qty     | sum_base_price  | sum_disc_price    | sum_charge          | avg_qty   | avg_price    | avg_disc | count_order |
+--------------+--------------+-------------+-----------------+-------------------+---------------------+-----------+--------------+----------+-------------+
| A            | F            | 37734107.00 | 56586554400.73  | 53758257134.8700  | 55909065222.827692  | 25.522005 | 38273.129734 | 0.049985 | 1478493     |
| N            | F            | 991417.00   | 1487504710.38   | 1413082168.0541   | 1469649223.194375   | 25.516471 | 38284.467760 | 0.050093 | 38854       |
| N            | O            | 74476040.00 | 111701729697.74 | 106118230307.6056 | 110367043872.497010 | 25.502226 | 38249.117988 | 0.049996 | 2920374     |
| R            | F            | 37719753.00 | 56568041380.90  | 53741292684.6040  | 55889619119.831932  | 25.505793 | 38250.854626 | 0.050009 | 1478870     |
+--------------+--------------+-------------+-----------------+-------------------+---------------------+-----------+--------------+----------+-------------+
4 rows in set. Query took 0.928 seconds.

The explain plan clearly shows DataFusion trying to parallelize the scan:
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan  | Sort: lineitem.parquet.l_returnflag ASC NULLS LAST, lineitem.parquet.l_linestatus ASC NULLS LAST                                                                                |
|               |   Projection: lineitem.parquet.l_returnflag, lineitem.parquet.l_linestatus, SUM(lineitem.parquet.l_quantity) AS sum_qty, SUM(lineitem.parquet.l_extendedprice) AS sum_base_price, SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount) AS sum_disc_price, SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax) AS sum_charge, AVG(lineitem.parquet.l_quantity) AS avg_qty, AVG(lineitem.parquet.l_extendedprice) AS avg_price, AVG(lineitem.parquet.l_discount) AS avg_disc, COUNT(UInt8(1)) AS count_order                                                                                |
|               |     Aggregate: groupBy=[[lineitem.parquet.l_returnflag, lineitem.parquet.l_linestatus]], aggr=[[SUM(lineitem.parquet.l_quantity), SUM(lineitem.parquet.l_extendedprice), SUM(CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))lineitem.parquet.l_discountDecimal128(Some(100),23,2)CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4))lineitem.parquet.l_extendedprice AS lineitem.parquet.l_extendedprice * Decimal128(Some(100),23,2) - lineitem.parquet.l_discount) AS SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount), SUM(CAST(CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))lineitem.parquet.l_discountDecimal128(Some(100),23,2)CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4))lineitem.parquet.l_extendedprice AS lineitem.parquet.l_extendedprice * Decimal128(Some(100),23,2) - lineitem.parquet.l_discount AS Decimal128(38, 6)) * CAST(Decimal128(Some(100),23,2) + CAST(lineitem.parquet.l_tax AS Decimal128(23, 2)) AS Decimal128(38, 6))) AS SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax), AVG(lineitem.parquet.l_quantity), AVG(lineitem.parquet.l_extendedprice), AVG(lineitem.parquet.l_discount), COUNT(UInt8(1))]] |
|               |       Projection: CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4)) AS CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))lineitem.parquet.l_discountDecimal128(Some(100),23,2)CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4))lineitem.parquet.l_extendedprice, lineitem.parquet.l_quantity, lineitem.parquet.l_extendedprice, lineitem.parquet.l_discount, lineitem.parquet.l_tax, lineitem.parquet.l_returnflag, lineitem.parquet.l_linestatus                                                                                |
|               |         Filter: lineitem.parquet.l_shipdate <= Date32("10471")                                                                                |
|               |           TableScan: lineitem.parquet projection=[l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate], partial_filters=[lineitem.parquet.l_shipdate <= Date32("10471")]                                                                                |
| physical_plan | SortPreservingMergeExec: [l_returnflag@0 ASC NULLS LAST,l_linestatus@1 ASC NULLS LAST]                                                                                |
|               |   SortExec: expr=[l_returnflag@0 ASC NULLS LAST,l_linestatus@1 ASC NULLS LAST]                                                                                |
|               |     ProjectionExec: expr=[l_returnflag@0 as l_returnflag, l_linestatus@1 as l_linestatus, SUM(lineitem.parquet.l_quantity)@2 as sum_qty, SUM(lineitem.parquet.l_extendedprice)@3 as sum_base_price, SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount)@4 as sum_disc_price, SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax)@5 as sum_charge, AVG(lineitem.parquet.l_quantity)@6 as avg_qty, AVG(lineitem.parquet.l_extendedprice)@7 as avg_price, AVG(lineitem.parquet.l_discount)@8 as avg_disc, COUNT(UInt8(1))@9 as count_order]                                                                                |
|               |       AggregateExec: mode=FinalPartitioned, gby=[l_returnflag@0 as l_returnflag, l_linestatus@1 as l_linestatus], aggr=[SUM(lineitem.parquet.l_quantity), SUM(lineitem.parquet.l_extendedprice), SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount), SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax), AVG(lineitem.parquet.l_quantity), AVG(lineitem.parquet.l_extendedprice), AVG(lineitem.parquet.l_discount), COUNT(UInt8(1))]                                                                                |
|               |         CoalesceBatchesExec: target_batch_size=8192                                                                                |
|               |           RepartitionExec: partitioning=Hash([Column { name: "l_returnflag", index: 0 }, Column { name: "l_linestatus", index: 1 }], 8), input_partitions=8                                                                                |
|               |             AggregateExec: mode=Partial, gby=[l_returnflag@5 as l_returnflag, l_linestatus@6 as l_linestatus], aggr=[SUM(lineitem.parquet.l_quantity), SUM(lineitem.parquet.l_extendedprice), SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount), SUM(lineitem.parquet.l_extendedprice * Int64(1) - lineitem.parquet.l_discount * Int64(1) + lineitem.parquet.l_tax), AVG(lineitem.parquet.l_quantity), AVG(lineitem.parquet.l_extendedprice), AVG(lineitem.parquet.l_discount), COUNT(UInt8(1))]                                                                                |
|               |               ProjectionExec: expr=[CAST(l_extendedprice@1 AS Decimal128(38, 4)) * CAST(Some(100),23,2 - CAST(l_discount@2 AS Decimal128(23, 2)) AS Decimal128(38, 4)) as CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4)) * CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))CAST(Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2)) AS Decimal128(38, 4))Decimal128(Some(100),23,2) - CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))CAST(lineitem.parquet.l_discount AS Decimal128(23, 2))lineitem.parquet.l_discountDecimal128(Some(100),23,2)CAST(lineitem.parquet.l_extendedprice AS Decimal128(38, 4))lineitem.parquet.l_extendedprice, l_quantity@0 as l_quantity, l_extendedprice@1 as l_extendedprice, l_discount@2 as l_discount, l_tax@3 as l_tax, l_returnflag@4 as l_returnflag, l_linestatus@5 as l_linestatus]                                                                                |
|               |                 CoalesceBatchesExec: target_batch_size=8192                                                                                |
|               |                   FilterExec: l_shipdate@6 <= 10471                                                                                |
|               |                     ParquetExec: limit=None, partitions={8 groups: [[home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:0..20797508], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:20797508..41595016], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:41595016..62392524], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:62392524..83190032], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:83190032..103987540], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:103987540..124785048], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:124785048..145582556], [home/alamb/tpch_data/parquet_data_SF1/lineitem.parquet:145582556..166380062]]}, predicate=l_shipdate@10 <= 10471, pruning_predicate=l_shipdate_min@0 <= 10471, projection=[l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate]                                                                                |
|               |                                                                                |
+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Here is the duckdb generated schema:
❯  create external table lineitem stored as parquet location 'lineitem.parquet';
0 rows in set. Query took 0.003 seconds.
❯ describe lineitem;
+-----------------+-------------------+-------------+
| column_name     | data_type         | is_nullable |
+-----------------+-------------------+-------------+
| l_orderkey      | Int32             | YES         |
| l_partkey       | Int32             | YES         |
| l_suppkey       | Int32             | YES         |
| l_linenumber    | Int32             | YES         |
| l_quantity      | Int32             | YES         |
| l_extendedprice | Decimal128(15, 2) | YES         |
| l_discount      | Decimal128(15, 2) | YES         |
| l_tax           | Decimal128(15, 2) | YES         |
| l_returnflag    | Utf8              | YES         |
| l_linestatus    | Utf8              | YES         |
| l_shipdate      | Date32            | YES         |
| l_commitdate    | Date32            | YES         |
| l_receiptdate   | Date32            | YES         |
| l_shipinstruct  | Utf8              | YES         |
| l_shipmode      | Utf8              | YES         |
| l_comment       | Utf8              | YES         |
+-----------------+-------------------+-------------+
16 rows in set. Query took 0.001 seconds.

Here is the datafusion generated schema:
❯ describe lineitem;
+-----------------+-------------------+-------------+
| column_name     | data_type         | is_nullable |
+-----------------+-------------------+-------------+
| l_orderkey      | Int64             | NO          |
| l_partkey       | Int64             | NO          |
| l_suppkey       | Int64             | NO          |
| l_linenumber    | Int32             | NO          |
| l_quantity      | Decimal128(15, 2) | NO          |
| l_extendedprice | Decimal128(15, 2) | NO          |
| l_discount      | Decimal128(15, 2) | NO          |
| l_tax           | Decimal128(15, 2) | NO          |
| l_returnflag    | Utf8              | NO          |
| l_linestatus    | Utf8              | NO          |
| l_shipdate      | Date32            | NO          |
| l_commitdate    | Date32            | NO          |
| l_receiptdate   | Date32            | NO          |
| l_shipinstruct  | Utf8              | NO          |
| l_shipmode      | Utf8              | NO          |
| l_comment       | Utf8              | NO          |
+-----------------+-------------------+-------------+

So it seems to me there are two follow ups to explore:

Why is datafusion only using one core on this single large file?
Why is the time different with the two files?

I will write a smaller self contained reproducer for the "slow query on a single file" and maybe someone clever can look at that.

I wonder if running parquet-layout against the parquet file might prove insightful.
DataFusion is currently limited to row group level parallelism, and there certainly are parquet writers that write very large row groups which would cause issues for this - https://github.com/apache/arrow/issues/34280

The layout is on: https://github.com/apache/arrow-datafusion/issues/5995#issuecomment-1507321881

So it seems to me there are two follow ups to explore:

Why is datafusion only using one core on this single large file?
Why is the time different with the two files?

Why Decimal128 is introduced for l_quantity in Datafusion but DuckDB uses int32?

Why Decimal128 is introduced for l_quantity in Datafusion but DuckDB uses int32?

I do not know -- the parquet files for these tests are created within duckdb somehow (it looks like maybe they have included the dbgen data generator in their system natively)

Filed https://github.com/apache/arrow-datafusion/issues/5995 with a reproducer for the "one core scanning" issue

the parquet files for these tests are created within duckdb somehow (it looks like maybe they have included the dbgen data generator in their system natively)

Yes I used the DuckDB tpch extension : https://github.com/duckdb/duckdb/tree/master/extension/tpch
Each table is exported as a Parquet file using a COPY ( ) TO, with row group size 122880 (default in DuckDB) and snappy compression

https://github.com/apache/arrow-datafusion/pull/5997 should help with this, the file range logic was partitioning row groups based on the location of their ColumnMetadata, which is normally written at the end of a ColumnChunk, not their actual data. This causes issues because duckdb appears to not write the un-inlined metadata and instead just writes a file offset of 0. Technically this is probably not spec-compliant, but more reliable to partition based on where the actual page data is.

So it seems to me there are two follow ups to explore:

Why is datafusion only using one core on this single large file?
Why is the time different with the two files?

Why Decimal128 is introduced for l_quantity in Datafusion but DuckDB uses int32?

I think l_quantity should be of type Decimal.  I checked the schema generated by SparkSQL tpch tool.

col_name
data_type
comment

l_orderkey
bigint
null

l_partkey
bigint
null

l_suppkey
bigint
null

l_linenumber
int
null

l_quantity
decimal(12,2)
null

l_extendedprice
decimal(12,2)
null

l_discount
decimal(12,2)
null

l_tax
decimal(12,2)
null

l_returnflag
string
null

l_linestatus
string
null

l_commitdate
date
null

l_receiptdate
date
null

l_shipinstruct
string
null

l_shipmode
string
null

l_comment
string
null

l_shipdate
date
null

So it seems to me there are two follow ups to explore:

Why is datafusion only using one core on this single large file?
Why is the time different with the two files?

Why Decimal128 is introduced for l_quantity in Datafusion but DuckDB uses int32?

I think l_quantity should be of type Decimal.  I checked the schema generated by SparkSQL tpch tool.

col_name
data_type
comment

l_orderkey
bigint
null

l_partkey
bigint
null

l_suppkey
bigint
null

l_linenumber
int
null

l_quantity
decimal(12,2)
null

l_extendedprice
decimal(12,2)
null

l_discount
decimal(12,2)
null

l_tax
decimal(12,2)
null

l_returnflag
string
null

l_linestatus
string
null

l_commitdate
date
null

l_receiptdate
date
null

l_shipinstruct
string
null

l_shipmode
string
null

l_comment
string
null

l_shipdate
date
null

I wonder if running parquet-layout against the parquet file might prove insightful.
DataFusion is currently limited to row group level parallelism, and there certainly are parquet writers that write very large row groups which would cause issues for this - apache/arrow#34280. Longer-term I would like to eventually get back to #2504 but that is not likely in the next couple of months.

The flexibility of the parquet file causes different Writers to use different file generation strategies. The data in a Parquet file can be spread over the row groups and the pages using any encoding and compression the writer or user wants.
If the physical layout of the parquet file affects the way different query engines scan, should we introduce a standard TPC-H Parquet file and re-run the performance comparison test?
I also saw this issue in this paper: https://dl.gi.de/bitstream/handle/20.500.12116/40316/B3-1.pdf?sequence=1&isAllowed=y

we look at three different Parquet writers to show how much Parquet files differ even though they store the same data. Parquet Writer Comparison:

Generator
Rows per Row Group
Pages per Row Group
File Sizes(SF1,SF10,SF100)

Spark
3,000,000
150
192 MB, 2.1 GB, 20 GB

Spark uncompressed
3,000,000
150
333 MB, 3.3 GB, 33 GB

DuckDB
100,352
1
281 MB, 2.8 GB, 28 GB

Arrow
67,108,864
15 - 1800
189 MB, 2.0 GB, 20 GB

For each generator, we measure the number of rows and the number of pages that are stored per row group. The Spark and DuckDB Parquet writers store a fixed number of elements per page and a fixed number of pages per row group. Since Parquet does not force synchronization between the column chunks, there are writers such as Arrow that do not store the same number of elements per page. Arrow uses a fixed data page size between roughly 0.5MB and 1 MB. For DuckDB and Spark, the page sizes vary from 0.5 MB to 6 MB.
Even though we only cover three different Parquet writers, we have already observed two extremes. DuckDB and Arrow do not take advantage of the hierarchical data layout: DuckDB will only use one page per row group, and Arrow stores the entire dataset in one row group for scale factor 1 and 10 since each row group stores 67 million rows.

I think l_quantity should be of type Decimal. I checked the schema generated by SparkSQL tpch tool.

It is also what is specified in the TPC-H document pdf:
Column Name	Datatype Requirements
L_QUANTITY	decimal

In the TPC-H documentation, l_quantity is defined as a decimal type.
However, duckdb incorrectly defines it as an INTEGER type, as can be seen in the duckdb source code: https://github.com/duckdb/duckdb/blob/eaf507009fd06573d74cc5742a8643481883a0ff/extension/tpch/dbgen/dbgen.cpp#L405-L415

Arrow stores the entire dataset in one row group for scale factor 1 and 10 since each row group stores 67 million rows

FWIW this was a bug and should be fixed by

DuckDB not making use of the hierarchical layout is perplexing, I wonder if this was an intentional design decision, it certainly isn't in the spirit of the specification

I don't understand this statement
This is my understanding of the difference:
┌──────────────────────────────────┐         ┌──────────────────────────────────┐
│        RowGroup Metadata         │         │        RowGroup Metadata         │
│                                  │         │                                  │
├──────────────────────────────────┤         ├──────────────────────────────────┤
│ ┌───────────────────────────────┐│         │        RowGroup Metadata         │
│ │                               ││         │                                  │
│ └───────────────────────────────┘│         ├──────────────────────────────────┤
│ ┌───────────────────────────────┐│         │        RowGroup Metadata         │
│ │                               ││         │                                  │
│ └───────────────────────────────┘│         └──────────────────────────────────┘
│ ┌───────────────────────────────┐│         ┌──────────────────────────────────┐
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│         RowGroup Data            │         │ └───────────────────────────────┘│
└──────────────────────────────────┘         │ ┌───────────────────────────────┐│
┌──────────────────────────────────┐         │ │                               ││
│        RowGroup Metadata         │         │ └───────────────────────────────┘│
│                                  │         │ ┌───────────────────────────────┐│
├──────────────────────────────────┤         │ │                               ││
│ ┌───────────────────────────────┐│         │ └───────────────────────────────┘│
│ │                               ││         │         RowGroup Data            │
│ └───────────────────────────────┘│         └──────────────────────────────────┘
│ ┌───────────────────────────────┐│         ┌──────────────────────────────────┐
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│ ┌───────────────────────────────┐│         │ └───────────────────────────────┘│
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│         RowGroup Data            │         │ └───────────────────────────────┘│
└──────────────────────────────────┘         │ ┌───────────────────────────────┐│
┌──────────────────────────────────┐         │ │                               ││
│        RowGroup Metadata         │         │ └───────────────────────────────┘│
│                                  │         │         RowGroup Data            │
├──────────────────────────────────┤         └──────────────────────────────────┘
│ ┌───────────────────────────────┐│         ┌──────────────────────────────────┐
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│ ┌───────────────────────────────┐│         │ └───────────────────────────────┘│
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│ ┌───────────────────────────────┐│         │ └───────────────────────────────┘│
│ │                               ││         │ ┌───────────────────────────────┐│
│ └───────────────────────────────┘│         │ │                               ││
│         RowGroup Data            │         │ └───────────────────────────────┘│
└──────────────────────────────────┘         │         RowGroup Data            │
                                             └──────────────────────────────────┘

               Arrow                                        DuckDB

Wouldn't the DuckDB implementation actually potentially be better (as it allows the metadata to be fetched with one contiguous IO.
Perhaps I am mis understanding

I don't understand this statement

Was in response so a single page per row group, not the fact it also has a dubious interpretation of the statistics specification. Inlining statistics makes a lot of sense, single page per row group seems to sacrifice a lot

Proposal for making the settings "fast by default": https://github.com/apache/arrow-datafusion/issues/6287

I think we have addressed this particular concern a long time ago -- we can work on additional performance improvements as we find them. CLosing this one as I don't think it is actionable now

authors: Dandandan alamb alippai andygrove djfrancesco mingmwang ozankabak r4ntix tustvold yahoNanJing


text:
parquet should depend on a small set of arrow features

parquet has an optional dependency on arrow. But when users do depend on parquet with arrow support, they get all of the default features for arrow, even though parquet by itself only needs the ipc feature.
My project depends on parquet and needs the arrow feature as well, but this makes my build times very large since arrow has so many default features that I don't need.
I'd like to change parquet's optional arrow dependency to only depend on the ipc feature and not the full set of default features. This is a one line change!

My proposed fix is in https://github.com/apache/arrow-rs/pull/801

repo: apache/arrow-rs


text:
Cherry pick Allow casting Timestamp arrays into String to active_release

Automatic cherry-pick of f569e8f

Originally appeared in https://github.com/apache/arrow-rs/pull/664: Allow casting Timestamp arrays into String

Codecov Report

Merging #698 (bca820d) into active_release (3c0a8bd) will increase coverage by 0.01%.
The diff coverage is 100.00%.

:exclamation: Current head bca820d differs from pull request most recent head 987ecc2. Consider uploading reports for the commit 987ecc2 to get more accurate results

@@                Coverage Diff                 @@
##           active_release     #698      +/-   ##
==================================================
+ Coverage           82.39%   82.40%   +0.01%
==================================================
  Files                 167      167
  Lines               46568    46600      +32
==================================================
+ Hits                38371    38403      +32
  Misses               8197     8197

Impacted Files
Coverage Δ

arrow/src/compute/kernels/cast.rs
94.63% <100.00%> (+0.09%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c0a8bd...987ecc2. Read the comment docs.

authors: alamb codecov-commenter


text:
[C++][Parquet] Follow API changes in ARROW-1808

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=wesm) / @wesm
PRs and other links:

GitHub Pull Request #418

Note: This issue was originally created as PARQUET-1164. Please see the migration documentation for further details.

Wes McKinney / @wesm:
Issue resolved by pull request 418
https://github.com/apache/parquet-cpp/pull/418

repo: apache/arrow


text:
ARROW-14157: [C++] Refactor Abseil to its own macro

We will need Abseil for the GCS C++ SDK (aka google-cloud-cpp), so
split it to its own macro. Use the abseil_ep target as a sentinel to
make the macro idempotent.
Abseil creates many (over 100) libraries, each relatively small. These
libraries depend on each other. Document how to extract this information
using a shell command-line for future updates. We will not need all
these libraries, but it is easier to describe all their dependencies
that pick and choose which ones are used.

@github-actions crossbow submit -g nightly

authors: coryan emkornfield


text:
ARROW-5830: [C++] Stop using memcmp in TensorEquals for tensors with float values

memcmp is an inappropriate way to examine the equality of float tensors because float tensors can have NaNs.  In addition to fix this issue, I made the following fixes in this pull request:

Change the namespace of StridedIntegerTensorContentsEquals, that is renamed from StridedTensorContentsEquals, to anonymous because it isn't referred from the outside of compare.cc
Let the result of Tensor::Equals be false when two tensors has the different shapes.

I think this pull-request is ready for code review.
@wesm Could you please have a look?

Codecov Report

Merging #5166 into master will decrease coverage by 21.89%.
The diff coverage is 62.26%.

@@            Coverage Diff             @@
##           master    #5166      +/-   ##
==========================================
- Coverage   87.65%   65.76%   -21.9%
==========================================
  Files        1025      554     -471
  Lines      146568    71019   -75549
  Branches     1437        0    -1437
==========================================
- Hits       128480    46707   -81773
- Misses      17726    24312    +6586
+ Partials      362        0     -362

Impacted Files
Coverage Δ

cpp/src/arrow/compare.cc
52.91% <62.26%> (-27%)
:arrow_down:

cpp/src/arrow/util/memory.h
0% <0%> (-100%)
:arrow_down:

cpp/src/gandiva/date_utils.h
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/util/memory.cc
0% <0%> (-100%)
:arrow_down:

cpp/src/gandiva/decimal_type_util.h
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/filesystem/util_internal.cc
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/compute/logical_type.h
0% <0%> (-100%)
:arrow_down:

cpp/src/parquet/hasher.h
0% <0%> (-100%)
:arrow_down:

cpp/src/gandiva/basic_decimal_scalar.h
0% <0%> (-100%)
:arrow_down:

cpp/src/arrow/compute/kernels/boolean.cc
0% <0%> (-100%)
:arrow_down:

... and 720 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0f9c4a9...bde8ff8. Read the comment docs.

@wesm I noticed that TensorEquals should also have an optional parameter of EqualOptions.
I want to make a change to add it by this Wednesday, please wait a moment.
And I'll look at ARROW-6359.  Thank you for telling me.

I've finished working for EqualOptions support in Tensor::Equals.
This is finally ready to review.

Thank you @mrkn !

authors: codecov-io mrkn pitrou


text:
ARROW-6214: [R] Add R sanitizer docker image

The PR creates a docker image to replicate R sanitizer errors. One can iterate with arrow local sources without installing CRAN arrow package.
A real undefined behaviour was fixed in array_to_vector.cpp. The other errors were false positives (to best of my knowledge).

@jeroen, @romainfrancois

I think this fixes one issue, but there may be more.
I've been exploring using the pyarrow manylinux wheel (or something similar) as a source of C++ binaries for the R package. After some trial-and-error, I got the package to build successfully, but then it core dumped when running the test suite, very similar to the situation we observed on the CRAN macOS server that led to this JIRA.
I pulled this change to array_to_vector.cpp onto the arrow fork/branch I was using and ran the build again, and it core-dumped again... but differently! This makes me suspect that there's another problem left to sanitize.
I'll investigate more on my side and see if I can identify where this is failing. Could be unrelated, of course.

This feels like an out of memory issue (both happen in allocation path + non determinism).

If you have a local mac, you can try to reproduce with ulimit, e.g.
(ulimit -m $((1024 * 1024 * 256)); R < testthat.R)

You might have to find the right ulimit flag (-m, -v)

The failures were on xenial ubuntu, not macOS. I was able to use the wheel as a binary source on macOS just fine.
I ran the tests with more verbosity and incrementally skipped the ones that triggered a core dump. It turned out that the specific exception was somewhat random even when the same test was failing, either bad_alloc or length_error, so the fix I pulled in from this branch might not have made the difference I thought before.
I eventually got the build to pass once I skipped enough tests: https://github.com/nealrichardson/arrow/compare/0.14.1.1-rpkg...nealrichardson:wheel-debug
All of the failures were tests that were trying to do something (mostly, and perhaps only, cast()) that was expected to fail and then assert the error message. For some reason this passes on most platforms and C++ build setups, but is failing on Travis on xenial using the pyarrow manylinux2010 wheel as the source for C++ binaries.
I don't know that I can debug this further, but maybe this information is useful to you/others who are more fluent in C++.

After some spelunking, we've come to suspect a mismatch between the (old) compiler used in manylinux2010 and the one used here in xenial. François added some gdb debugging, and we saw that the problem was coming from the exception handler inside Rcpp::stop: https://travis-ci.org/nealrichardson/sandbox/builds/586674948#L1546. https://stackoverflow.com/questions/56494095/rcppstop-crashes-r-under-g suggests that compiler version mismatch could be to blame.
Anyway, this turns out to be irrelevant for the current pull request.

@ursabot crossbow submit docker-r

@ursabot crossbow submit docker-r-sanitizer

This is now passing travis' lint: https://travis-ci.org/fsaintjacques/arrow/builds/587033835?utm_medium=notification&utm_source=email

authors: fsaintjacques nealrichardson


text:
Implement RedisIO on top of Splittable DoFn

Imported from Jira BEAM-11997. Original Jira may contain additional context.
Reported by: boyuanz.

This issue was resolved in 2.39.0 release (though, related jira is still open I guess my mistake).

authors: aromanenko-dev damccorm

repo: apache/beam


text:
[Task]: Cut Playground output

What needs to happen?
Text fields in Flutter are slow with long text:

https://github.com/flutter/flutter/issues/128575

Since the user can potentially produce very long output by putting print in a loop, we should limit the visible output. Since this is a frontend issue, this should be done at the frontend.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[X] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

.take-issue


text:
[BEAM-8337] publish Flink job server container images

This will make it easier to get started on Beam on Flink on Kubernetes.
cc @udim

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---

---
---

Java

Python

---

---
---

XLang
---
---
---

---
---
---

Pre-Commit Tests Status (on master branch)

---
Java
Python
Go
Website

Non-portable

Portable
---

---
---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

cc @angoenka

Run Python Spark ValidatesRunner

Run Python PreCommit

I triggered Python Spark ValidatesRunner Tests by mistake.
Java failures are a known flake: BEAM-8035