text:
wr.s3.read_parquet() do not read partition column

Describe the bug
Following table has three columns: day, month, year, where year is the paritionKey.
! aws s3 ls s3://ocean-qualiu/spark/test_wr_2_test_parquet_write_a/
                           PRE year=2018/
                           PRE year=2019/
                           PRE year=2020/

When run following python code to read the table path. Pandas show a dataframe with only day, month.
PS. Redshift Spectrum and Athena was able to recognize partition column as a column.
wr.s3.read_parquet("s3://ocean-qualiu/spark/test_wr_2_test_parquet_write_a/").head()

Environment
Package                            Version

alabaster                          0.7.12
anaconda-client                    1.7.2
anaconda-project                   0.8.3
argh                               0.26.2
asn1crypto                         1.4.0
astroid                            2.4.2
astropy                            4.0
atomicwrites                       1.3.0
attrs                              20.3.0
Automat                            20.2.0
autopep8                           1.4.4
autovizwidget                      0.17.1
awscli                             1.18.197
awswrangler                        2.8.0
Babel                              2.8.0
backcall                           0.1.0
backports.shutil-get-terminal-size 1.0.0
bcrypt                             3.2.0
beautifulsoup4                     4.8.2
bitarray                           1.2.1
bkcharts                           0.2
bleach                             3.2.1
bokeh                              1.4.0
boto                               2.49.0
boto3                              1.16.37
botocore                           1.19.37
Bottleneck                         1.3.2
cached-property                    1.5.2
certifi                            2020.11.8
cffi                               1.14.0
chardet                            3.0.4
Click                              7.0
cloudpickle                        1.3.0
clyent                             1.2.2
colorama                           0.4.3
contextlib2                        0.6.0.post1
cryptography                       2.8
cycler                             0.10.0
Cython                             0.29.15
cytoolz                            0.10.1
dask                               2.11.0
decorator                          4.4.1
defusedxml                         0.6.0
diff-match-patch                   20181111
dill                               0.3.3
distributed                        2.11.0
distro                             1.5.0
docker                             4.4.0
docker-compose                     1.27.4
dockerpty                          0.4.1
docopt                             0.6.2
docutils                           0.15.2
entrypoints                        0.3
environment-kernels                1.1.1
et-xmlfile                         1.0.1
fastcache                          1.1.0
filelock                           3.0.12
flake8                             3.7.9
Flask                              1.1.1
fsspec                             0.6.2
future                             0.18.2
gevent                             1.4.0
glob2                              0.7
gmpy2                              2.0.8
google-pasta                       0.2.0
greenlet                           0.4.15
h5py                               2.10.0
hdijupyterutils                    0.17.1
HeapDict                           1.0.1
html5lib                           1.0.1
hypothesis                         5.5.4
idna                               2.10
imageio                            2.6.1
imagesize                          1.2.0
importlib-metadata                 3.1.0
intervaltree                       3.0.2
ipykernel                          5.1.4
ipyparallel                        6.3.0
ipython                            7.12.0
ipython-genutils                   0.2.0
ipywidgets                         7.5.1
isort                              4.3.21
itsdangerous                       1.1.0
jdcal                              1.4.1
jedi                               0.14.1
jeepney                            0.4.2
Jinja2                             2.11.1
jmespath                           0.10.0
joblib                             0.14.1
json5                              0.9.1
jsonschema                         3.2.0
jupyter                            1.0.0
jupyter-client                     5.3.4
jupyter-console                    6.1.0
jupyter-core                       4.6.1
jupyterlab                         1.2.6
jupyterlab-server                  1.0.6
keyring                            21.1.0
kiwisolver                         1.1.0
lazy-object-proxy                  1.4.3
libarchive-c                       2.8
lief                               0.9.0
llvmlite                           0.34.0
locket                             0.2.0
lxml                               4.6.2
MarkupSafe                         1.1.1
matplotlib                         3.1.3
mccabe                             0.6.1
mistune                            0.8.4
mkl-fft                            1.0.15
mkl-random                         1.1.0
mkl-service                        2.3.0
mock                               4.0.1
more-itertools                     8.2.0
mpmath                             1.1.0
msgpack                            0.6.1
multipledispatch                   0.6.0
nb-conda                           2.2.1
nb-conda-kernels                   2.2.4
nbconvert                          5.6.1
nbformat                           5.0.4
networkx                           2.4
nltk                               3.4.5
nose                               1.3.7
notebook                           6.0.3
numba                              0.51.2
numexpr                            2.7.1
numpy                              1.19.4
numpydoc                           0.9.2
nvidia-ml-py                       10.418.84
olefile                            0.46
opencv-python                      4.2.0.32
openpyxl                           3.0.3
packaging                          20.7
pandas                             1.1.5
pandocfilters                      1.4.2
paramiko                           2.7.2
parso                              0.5.2
partd                              1.1.0
path                               13.1.0
pathlib2                           2.3.5
pathtools                          0.1.2
patsy                              0.5.1
pep8                               1.7.1
pexpect                            4.8.0
pg8000                             1.19.5
pickleshare                        0.7.5
Pillow                             7.2.0
pip                                20.3
pkginfo                            1.5.0.1
plotly                             4.13.0
pluggy                             0.13.1
ply                                3.11
prometheus-client                  0.7.1
prompt-toolkit                     3.0.3
protobuf                           3.14.0
protobuf3-to-dict                  0.1.5
psutil                             5.6.7
psycopg2                           2.7.5
ptyprocess                         0.6.0
py                                 1.8.1
py3cli                             1.0.1
py4j                               0.10.7
pyarrow                            2.0.0
pyasn1                             0.4.8
pycodestyle                        2.5.0
pycosat                            0.6.3
pycparser                          2.19
pycrypto                           2.6.1
pycurl                             7.43.0.5
pydocstyle                         4.0.1
pyfiglet                           0.7
pyflakes                           2.1.1
pyfunctional                       1.4.2
pygal                              2.4.0
Pygments                           2.5.2
pykerberos                         1.2.1
pylint                             2.6.0
PyMySQL                            1.0.2
PyNaCl                             1.4.0
pyodbc                             4.0.0-unsupported
pyOpenSSL                          19.1.0
pyparsing                          2.4.7
pyrsistent                         0.15.7
PySocks                            1.7.1
pyspark                            2.3.4
pytest                             5.3.5
pytest-arraydiff                   0.3
pytest-astropy                     0.8.0
pytest-astropy-header              0.1.2
pytest-doctestplus                 0.5.0
pytest-openfiles                   0.4.0
pytest-remotedata                  0.3.2
python-dateutil                    2.8.1
python-dotenv                      0.15.0
python-jsonrpc-server              0.3.4
python-language-server             0.31.7
pytz                               2021.1
PyWavelets                         1.1.1
pyxdg                              0.26
PyYAML                             5.3.1
pyzmq                              18.1.1
QDarkStyle                         2.8
QtAwesome                          0.6.1
qtconsole                          4.6.0
QtPy                               1.9.0
redshift-connector                 2.0.881
requests                           2.25.0
requests-kerberos                  0.12.0
retrying                           1.3.3
rope                               0.16.0
rsa                                4.5
Rtree                              0.9.3
ruamel-yaml                        0.15.87
s3fs                               0.4.2
s3transfer                         0.3.3
sagemaker                          2.19.0
sagemaker-pyspark                  1.4.1
scikit-image                       0.16.2
scikit-learn                       0.22.1
scipy                              1.4.1
scramp                             1.4.0
seaborn                            0.10.0
SecretStorage                      3.1.2
Send2Trash                         1.5.0
setuptools                         50.3.2
simplegeneric                      0.8.1
singledispatch                     3.4.0.3
six                                1.15.0
sklearn                            0.0
smclarify                          0.1
smdebug-rulesconfig                1.0.0
snowballstemmer                    2.0.0
sortedcollections                  1.1.2
sortedcontainers                   2.1.0
soupsieve                          1.9.5
sparkmagic                         0.15.0
Sphinx                             3.3.1
sphinxcontrib-applehelp            1.0.1
sphinxcontrib-devhelp              1.0.1
sphinxcontrib-htmlhelp             1.0.2
sphinxcontrib-jsmath               1.0.1
sphinxcontrib-qthelp               1.0.2
sphinxcontrib-serializinghtml      1.1.3
sphinxcontrib-websupport           1.2.0
spyder                             4.0.1
spyder-kernels                     1.8.1
SQLAlchemy                         1.3.13
statsmodels                        0.11.0
sympy                              1.5.1
tables                             3.6.1
tabulate                           0.8.7
tblib                              1.6.0
terminado                          0.8.3
testpath                           0.4.4
texttable                          1.6.3
toml                               0.10.2
toolz                              0.10.0
tornado                            6.0.3
tqdm                               4.42.1
traitlets                          4.3.3
typed-ast                          1.4.1
ujson                              1.35
unicodecsv                         0.14.1
urllib3                            1.25.11
watchdog                           0.10.2
wcwidth                            0.1.8
webencodings                       0.5.1
websocket-client                   0.57.0
Werkzeug                           1.0.0
wheel                              0.34.2
widgetsnbextension                 3.5.1
wrapt                              1.11.2
wurlitzer                          2.0.0
xlrd                               1.2.0
XlsxWriter                         1.2.7
xlwt                               1.3.0
yapf                               0.28.0
zict                               1.0.0
zipp                               3.4.0
WARNING: You are using pip version 20.3; however, version 21.1.2 is available.
You should consider upgrading via the '/home/ec2-user/anaconda3/envs/python3/bin/python -m pip install --upgrade pip' command.
To Reproduce

create a paritioned parquet with data wrangler.
Read the parquet files with wr.s3.read_parquet(table path)

P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.

You must add dataset=True as a parameter to the call. From the read_parquet docs:
dataset (bool) – If True read a parquet dataset instead of simple file(s) loading all the related partitions as columns.

Example:
import awswrangler as wr
import pandas as pd

wr.s3.to_parquet(
    df=pd.DataFrame({
        "col": [1, 2, 3],
        "col2": ["A", "A", "B"]
    }),
    path="s3://bucket/prefix",
    dataset=True,
    partition_cols=["col2"]
)

df1 = wr.s3.read_parquet("s3://bucket/prefix/")
print(df1.head())
   col
0    1
1    2
2    3

df2 = wr.s3.read_parquet("s3://bucket/prefix/", dataset=True)
print(df2.head())
   col col2
0    1    A
1    2    A
2    3    B

authors: jaidisido samliuq

repo: awslabs/aws-data-wrangler


text:
feat(docs): update docs when creating a new Layer ARN

Issue number: #1549
Summary
This PR automates the publishing of the layer version in our documetation for every release.
This will reduce maintance overhead, and make the publishing of the new layer version faster.
Changes

Please provide a summary of what's being changed

Update action-lint to support multiple nested workflow calls https://github.blog/changelog/2022-08-22-github-actions-improvements-to-reusable-workflows-2/
Added arm64 layers to docs
Added method to republish docs after a new published Layer

User experience

Please share what the user experience looks like before and after this change

After this change, the new layer version will be update don the documentation automatically, as soon as the Layer is created and published.
Checklist
If your change doesn't seem to apply, please leave them unchecked.

[x] Meet tenets criteria
[x] I have performed a self-review of this change
[x] Changes have been tested
[x] Changes are documented
[x] PR title follows conventional commit semantics

Is this a breaking change?
RFC issue number:
Checklist:

[ ] Migration process documented
[ ] Implement warnings (if it can live side by side)

Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

View rendered docs/index.md

Note: update the rest of the SAM sample code on docs/index.md

Note: remove "Can't find our Lambda Layer for your preferred AWS region?" because we are now publishing in all regions.

Docs looks excellent. Perhaps let's just increase md-grid (where the content is) to 90vw to give a bit more room, otherwise the layer per region table will look too tight

Consider it approved after these last changes to ease maintenance.

source: gharchive/pull-request

authors: heitorlessa rubenfonseca

repo: awslabs/aws-lambda-powertools-python

license_type: permissive

license_source: github-api


text:
ci: add release new version action

Description

Add release new version action

General Checklist

[ ] Added new tests to cover change, if needed
[ ] Security oriented best practices and standards are followed (e.g. using input sanitization, principle of least privilege, etc)
[ ] Documentation update for the change if required
[x] PR title conforms to conventional commit style
[ ] If breaking change, documentation/changelog update with migration instructions

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
Patch and project coverage have no change.

Comparison is base (6b3b527) 99.65% compared to head (6762a45) 99.65%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #12   +/-   ##
=======================================
  Coverage   99.65%   99.65%
=======================================
  Files          24       24
  Lines         882      882
  Branches      151      151
=======================================
  Hits          879      879
  Misses          3        3

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter zhu-xiaowei

repo: awslabs/clickstream-web


text:
Use custom NDK version instead of action default NDK

PyTorch Android native build only works on NDK version 20.
Action uses version 22 by default.

Have you tested the android pipeline?

Have you tested the android pipeline?

Have you tested the android pipeline?

I tested it on my local repo https://github.com/stu1130/djl/runs/1705801591?check_suite_focus=true you can see it only failed on upload

Have you tested the android pipeline?

I tested it on my local repo https://github.com/stu1130/djl/runs/1705801591?check_suite_focus=true you can see it only failed on upload

repo: awslabs/djl


text:
Fix spelling error in ListDataset docstring

Issue #, if available: No issue created
Description of changes: I found a very small spelling error in the ListDatasetdocumentation and fixed it.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thank you!

authors: flrs lostella

repo: awslabs/gluon-ts


text:
Add Offerings to InstanceType to hold zone and capacity type, remove OS from InstanceType, promote capacity-type label

1. Issue, if available:
Part of https://github.com/awslabs/karpenter/issues/371
2. Description of changes:

OS is not an instance type level concept - this seems to have been shoehorned in and causes unnecessary confusion. Removing it until we figure out proper support for OS constraints.
When we do ICE caching we need to be able to filter out combinations of zone and capacity type when we see ICEs. Since packing otherwise does not care about them it seemed preferable to put a slice of these tuples ("Offerings") inside InstanceType rather than having to permute all the possible combinations in bin packing (i.e. if there are 3 zones and 2 capacity types then we can still consider just a single InstanceType for an actual instance type with this Offering re-factoring rather than having to create 5 InstanceType instances in the status quo to cover the permutations of zone and capacity type).
Added a note on tailing logs to the dev guide. Maybe obvious to those of you very familiar with K8s, but was not obvious to me =P
In talking to Ellis, decided to bite the bullet and re-name the capacity-type label to make it a first class citizen rather than being relegated to the AWS cloud provider. This simplifies some of the scheduler code a little and honestly just makes more sense.

I'm not entirely sure if I need to add additional unit tests at this time - I think we're covered with what's there but open to feedback.
3. Does this change impact docs?

[x] Yes, PR includes docs updates
[ ] Yes, issue opened: link to issue
[ ] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I appreciate you including docs changes, it's helping me keep the thread.

authors: eptiger geoffcline

repo: awslabs/karpenter


text:
Cleanups

Some minor cleanups I ran into while working on the extables fixes.
Boilerplate disclamer follows:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Had a braino in the top commit, so dropped it.

repo: awslabs/ktf


text:
Fargate support

Tell us more about this new feature.
At present, mountpoint-s3 cannot be used with the Fargate runtime because it doesn't support privileged mode. There have been numerous discussions on s3-fargate integration over the past few years without any tangible efforts. Implementing this feature would greatly benefit many existing AWS customers. Are you planning to work on it?

Thanks for the feature request, Damian! I can definitely see the benefits of this integration.
I don't have anything to share on Fargate support right now. We are tracking interest for Fargate support in #450. I will close this issue in favor of that one.

source: gharchive/issue

authors: dannycjones kalupad


text:
docs: Add security comment to ISSUE_TEMPLATE

Resolved issues:
Internal
Description of changes:
Copy the security reporting wording from the README to the issue template.
Also, Update to newer GitHub template format.
Call-outs:
There is the option of creating distinct issue templates for Bugs vs Features, for now, just went with Custom to reduce confusion and duplication.
Testing:
Added to my fork.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #2167 into master will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2167      +/-   ##
==========================================
- Coverage   81.09%   81.09%   -0.01%
==========================================
  Files         248      248
  Lines       18497    18497
==========================================
- Hits        15001    15000       -1
- Misses       3496     3497       +1

Looks like you would have to rebase your PR to obtain the latest fix for sidetrail https://github.com/awslabs/s2n/pull/2170.

authors: codecov-commenter dougch ttjsu-aws


text:
[Bug] TRE when deploying the TRE solution with devops profile and shared ami parameters the deployment fails

Describe the bug
TRE when deploying the TRE solution with devops profile and enableAmiSharing parameters the deployment fails
The Reason:
The TRE deployment creates 2 IAM roles that are required for iam sharing over the devops account, this roles are created after running the enableAmiSharing and it fails as the IAM roles are missing.
Possible Resolution:
Create IAM Roles workflowLoopRunnerRole & apiHandlerRole before running the main/solution/prepare-devops-account
Workaround:
Deploy TRE environment without parameters devops profile and enableAmiSharing
After successful deployment, change the parameters evops profile and enableAmiSharing = true  in the <stage.yml> and run deploy again .
To Reproduce
Steps to reproduce the behavior:
Deploy environment from scratch :
Create  new environment <stage.yml> set enableAmiSharing = true with devops profile
Run ./scripts/environment-deploy

See error

Expected behavior
The deployment creates prepare devops account as part of the deployment
Versions (please complete the following information):

Release Version installed [5.2.7]

Thank you for reaching out. Your request has been added to our backlog for research computing solutions. Our team curates these requests for fit with our solutions vision on a regular basis. Please watch this space for new updates. If you require immediate assistance, please reach out to your AWS account team. Please note that security issues should be reported directly to AWS Security at aws-security@amazon.com.

repo: awslabs/service-workbench-on-aws


text:
fix: scripts not finding the right one

Description
Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

We identified new issues on unchanged lines of code. Navigate to the Amazon CodeGuru Reviewer console to view the recommendations to fix them.

authors: NingLu alvindaiyan

repo: awslabs/stable-diffusion-aws-extension


text:
aws spring cloud 3.x : There is no queue stop timeout, that is creating problems by increasing buid timeout.

Type: Feature
Is your feature request related to a problem? Please describe.
public void stop(String logicalQueueName) {
stopQueue(logicalQueueName);
	try {
		if (isRunning(logicalQueueName)) {
			Future<?> future = this.scheduledFutureByQueue.remove(logicalQueueName);
			if (future != null) {
				future.get(this.queueStopTimeout, TimeUnit.MILLISECONDS);
			}
		}
	}
	catch (InterruptedException e) {
		Thread.currentThread().interrupt();
	}
	catch (ExecutionException | TimeoutException e) {
		getLogger().warn("Error stopping queue with name: '" + logicalQueueName + "'", e);
	}
}

There used to be a queueStopTimeout in the previous version, that let us, shut the queue down by interrupting the thread, when the application shuts down. In v3

@Override
public void stop() {
	if (!this.isRunning) {
		return;
	}
	logger.debug("Stopping container {}", this.id);
	synchronized (this.lifecycleMonitor) {
		this.isRunning = false;
		doStop();
	}
	logger.info("Container {} stopped", this.id);
}

Please advice if this exists, leave implementation, do you see a "hacky" way?

Another hacky thing I had to do: https://github.com/awspring/spring-cloud-aws/issues/1055

@maciejwalkowiak

@VaibhavTheVicar , please take a look at the ContainerOptions.
You'll find listenerShutdownTimeout and acknowledgementShutdownTimeout there which you can configure to achieve your goal. Set to Duration.ZERO if you don't want any wait.
Let me know if that works for you.

Closing due to lack of feedback.

authors: VaibhavTheVicar tomazfernandes

repo: awspring/spring-cloud-aws


text:
events_unbound io_ring_exit_work task kworker blocked for more than 120 seconds

Follow up on the issue from #326 - but this is only triggered when SQPOLL is enabled.
This is running with
https://git.kernel.dk/cgit/linux-block/commit/?h=poll-multiple&id=283fc84eaeb1031e1f2389e4e365e44cd4398b9c
I have one ring that is only deallocated by process exit, the rest is "properly" closed.
[  633.583870] ------------[ cut here ]------------
[  633.583873] WARNING: CPU: 4 PID: 6485 at fs/io_uring.c:8611 io_ring_exit_work+0xe6/0x500
[  633.583880] Modules linked in: binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua isst_if_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl ppdev parport_pc ena parport serio_raw i2c_piix4 sch_fq_codel drm i2c_core ip_tables x_tables autofs4
[  633.583896] CPU: 4 PID: 6485 Comm: kworker/u16:77 Not tainted 5.12.0-rc6-5.13-uring-210407 #1
[  633.583898] Hardware name: Amazon EC2 c5.2xlarge/, BIOS 1.0 10/16/2017
[  633.583899] Workqueue: events_unbound io_ring_exit_work
[  633.583902] RIP: 0010:io_ring_exit_work+0xe6/0x500
[  633.583904] Code: 8b bb f0 fd ff ff e8 99 7d ff ff 4c 8d ab 40 fe ff ff 31 d2 31 f6 4c 89 e7 e8 b6 e9 ff ff 48 8b 05 7f 03 29 01 49 39 c7 79 02 <0f> 0b be 0c 00 00 00 4c 89 ef e8 5b 96 7c 00 48 85 c0 74 d4 4c 8d
[  633.583906] RSP: 0018:ffffb30e01bebdd0 EFLAGS: 00010297
[  633.583908] RAX: 00000001000144f7 RBX: ffff97d63d8095a8 RCX: 0000000000000008
[  633.583909] RDX: 0000000000000001 RSI: ffff97d63d809088 RDI: ffff97d63d8095a0
[  633.583910] RBP: ffffb30e01bebe50 R08: 00000093848a6640 R09: 0000000000000000
[  633.583911] R10: ffffffff9126bfa0 R11: 0000000000000000 R12: ffff97d63d809000
[  633.583912] R13: ffff97d63d8093e8 R14: 0000000000000000 R15: 00000001000144f3
[  633.583913] FS:  0000000000000000(0000) GS:ffff97d90bd00000(0000) knlGS:0000000000000000
[  633.583915] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  633.583916] CR2: 000000c00033d000 CR3: 0000000115010004 CR4: 00000000007706e0
[  633.583919] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  633.583920] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  633.583920] PKRU: 55555554
[  633.583921] Call Trace:
[  633.583925]  ? __switch_to+0x192/0x460
[  633.583930]  ? __switch_to_asm+0x36/0x70
[  633.583934]  process_one_work+0x220/0x3c0
[  633.583938]  worker_thread+0x4d/0x3f0
[  633.583940]  kthread+0x114/0x150
[  633.583943]  ? process_one_work+0x3c0/0x3c0
[  633.583945]  ? kthread_park+0x90/0x90
[  633.583947]  ret_from_fork+0x22/0x30
[  633.583950] ---[ end trace 5225fbee45c60487 ]---
[ 2840.166730] show_signal: 36 callbacks suppressed
[ 2840.166735] traps: swift-nioPackag[143299] trap invalid opcode ip:7f1d15b3c7d2 sp:7ffc59b93ea0 error:0 in libswiftCore.so[7f1d159ea000+525000]
[ 3626.789652] INFO: task kworker/u16:15:5478 blocked for more than 120 seconds.
[ 3626.792583]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.795536] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.798758] task:kworker/u16:15  state:D stack:    0 pid: 5478 ppid:     2 flags:0x00004000
[ 3626.798763] Workqueue: events_unbound io_ring_exit_work
[ 3626.798769] Call Trace:
[ 3626.798772]  __schedule+0x2de/0x890
[ 3626.798777]  ? check_preempt_wakeup+0xfd/0x210
[ 3626.798781]  schedule+0x4f/0xc0
[ 3626.798783]  schedule_timeout+0x202/0x290
[ 3626.798786]  ? ttwu_do_activate+0xb5/0x3f0
[ 3626.798789]  wait_for_completion+0x94/0x100
[ 3626.798792]  io_ring_exit_work+0x18c/0x500
[ 3626.798793]  ? io_uring_del_task_file+0xc0/0xc0
[ 3626.798798]  process_one_work+0x220/0x3c0
[ 3626.798801]  worker_thread+0x4d/0x3f0
[ 3626.798803]  kthread+0x114/0x150
[ 3626.798805]  ? process_one_work+0x3c0/0x3c0
[ 3626.798807]  ? kthread_park+0x90/0x90
[ 3626.798809]  ret_from_fork+0x22/0x30
[ 3626.798816] INFO: task kworker/u16:29:5506 blocked for more than 120 seconds.
[ 3626.801729]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.804603] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.807777] task:kworker/u16:29  state:D stack:    0 pid: 5506 ppid:     2 flags:0x00004000
[ 3626.807779] Workqueue: events_unbound io_ring_exit_work
[ 3626.807782] Call Trace:
[ 3626.807783]  __schedule+0x2de/0x890
[ 3626.807786]  ? enqueue_entity+0x115/0x660
[ 3626.807789]  schedule+0x4f/0xc0
[ 3626.807791]  schedule_timeout+0x202/0x290
[ 3626.807793]  ? ttwu_do_activate+0xb5/0x3f0
[ 3626.807797]  wait_for_completion+0x94/0x100
[ 3626.807799]  io_ring_exit_work+0x18c/0x500
[ 3626.807803]  ? io_uring_del_task_file+0xc0/0xc0
[ 3626.807806]  process_one_work+0x220/0x3c0
[ 3626.807808]  worker_thread+0x4d/0x3f0
[ 3626.807811]  kthread+0x114/0x150
[ 3626.807813]  ? process_one_work+0x3c0/0x3c0
[ 3626.807815]  ? kthread_park+0x90/0x90
[ 3626.807817]  ret_from_fork+0x22/0x30
[ 3626.807835] INFO: task kworker/u16:98:138912 blocked for more than 120 seconds.
[ 3626.810838]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.813798] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.816976] task:kworker/u16:98  state:D stack:    0 pid:138912 ppid:     2 flags:0x00004000
[ 3626.816978] Workqueue: events_unbound io_ring_exit_work
[ 3626.816980] Call Trace:
[ 3626.816981]  __schedule+0x2de/0x890
[ 3626.816983]  ? enqueue_entity+0x115/0x660
[ 3626.816986]  schedule+0x4f/0xc0
[ 3626.816987]  schedule_timeout+0x202/0x290
[ 3626.816989]  ? ttwu_do_activate+0xb5/0x3f0
[ 3626.816992]  wait_for_completion+0x94/0x100
[ 3626.816994]  io_ring_exit_work+0x18c/0x500
[ 3626.816996]  ? io_uring_del_task_file+0xc0/0xc0
[ 3626.816999]  process_one_work+0x220/0x3c0
[ 3626.817000]  worker_thread+0x4d/0x3f0
[ 3626.817002]  kthread+0x114/0x150
[ 3626.817004]  ? process_one_work+0x3c0/0x3c0
[ 3626.817005]  ? kthread_park+0x90/0x90
[ 3626.817007]  ret_from_fork+0x22/0x30
[ 3626.817015] INFO: task kworker/u16:129:227858 blocked for more than 120 seconds.
[ 3626.820039]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.822968] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.826156] task:kworker/u16:129 state:D stack:    0 pid:227858 ppid:     2 flags:0x00004000
[ 3626.826159] Workqueue: events_unbound io_ring_exit_work
[ 3626.826161] Call Trace:
[ 3626.826162]  __schedule+0x2de/0x890
[ 3626.826164]  ? enqueue_entity+0x115/0x660
[ 3626.826168]  schedule+0x4f/0xc0
[ 3626.826170]  schedule_timeout+0x202/0x290
[ 3626.826172]  ? ttwu_do_activate+0xb5/0x3f0
[ 3626.826176]  wait_for_completion+0x94/0x100
[ 3626.826179]  io_ring_exit_work+0x18c/0x500
[ 3626.826182]  ? io_uring_del_task_file+0xc0/0xc0
[ 3626.826185]  process_one_work+0x220/0x3c0
[ 3626.826187]  worker_thread+0x4d/0x3f0
[ 3626.826189]  kthread+0x114/0x150
[ 3626.826191]  ? process_one_work+0x3c0/0x3c0
[ 3626.826192]  ? kthread_park+0x90/0x90
[ 3626.826194]  ret_from_fork+0x22/0x30
[ 3626.826200] INFO: task kworker/u16:145:240639 blocked for more than 120 seconds.
[ 3626.829201]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.833454] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.839410] task:kworker/u16:145 state:D stack:    0 pid:240639 ppid:     2 flags:0x00004000
[ 3626.839413] Workqueue: events_unbound io_ring_exit_work
[ 3626.839415] Call Trace:
[ 3626.839416]  __schedule+0x2de/0x890
[ 3626.839418]  ? enqueue_entity+0x115/0x660
[ 3626.839420]  schedule+0x4f/0xc0
[ 3626.839422]  schedule_timeout+0x202/0x290
[ 3626.839424]  ? ttwu_do_activate+0xb5/0x3f0
[ 3626.839427]  wait_for_completion+0x94/0x100
[ 3626.839429]  io_ring_exit_work+0x18c/0x500
[ 3626.839431]  ? io_uring_del_task_file+0xc0/0xc0
[ 3626.839433]  process_one_work+0x220/0x3c0
[ 3626.839435]  worker_thread+0x4d/0x3f0
[ 3626.839436]  kthread+0x114/0x150
[ 3626.839439]  ? process_one_work+0x3c0/0x3c0
[ 3626.839440]  ? kthread_park+0x90/0x90
[ 3626.839442]  ret_from_fork+0x22/0x30
[ 3626.839446] INFO: task kworker/u16:157:262491 blocked for more than 120 seconds.
[ 3626.845169]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.849467] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.855352] task:kworker/u16:157 state:D stack:    0 pid:262491 ppid:     2 flags:0x00004000
[ 3626.855354] Workqueue: events_unbound io_ring_exit_work
[ 3626.855357] Call Trace:
[ 3626.855358]  __schedule+0x2de/0x890
[ 3626.855362]  ? enqueue_entity+0x115/0x660
[ 3626.855365]  schedule+0x4f/0xc0
[ 3626.855367]  schedule_timeout+0x202/0x290
[ 3626.855369]  ? ttwu_do_activate+0xb5/0x3f0
[ 3626.855374]  wait_for_completion+0x94/0x100
[ 3626.855376]  io_ring_exit_work+0x18c/0x500
[ 3626.855378]  ? io_uring_del_task_file+0xc0/0xc0
[ 3626.855381]  process_one_work+0x220/0x3c0
[ 3626.855384]  worker_thread+0x4d/0x3f0
[ 3626.855386]  kthread+0x114/0x150
[ 3626.855389]  ? process_one_work+0x3c0/0x3c0
[ 3626.855390]  ? kthread_park+0x90/0x90
[ 3626.855392]  ret_from_fork+0x22/0x30
[ 3626.855397] INFO: task kworker/u16:169:274169 blocked for more than 120 seconds.
[ 3626.861101]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.865360] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.871249] task:kworker/u16:169 state:D stack:    0 pid:274169 ppid:     2 flags:0x00004000
[ 3626.871252] Workqueue: events_unbound io_ring_exit_work
[ 3626.871254] Call Trace:
[ 3626.871255]  __schedule+0x2de/0x890
[ 3626.871257]  ? check_preempt_wakeup+0xfd/0x210
[ 3626.871259]  schedule+0x4f/0xc0
[ 3626.871261]  schedule_timeout+0x202/0x290
[ 3626.871263]  ? ttwu_do_activate+0xb5/0x3f0
[ 3626.871265]  wait_for_completion+0x94/0x100
[ 3626.871267]  io_ring_exit_work+0x18c/0x500
[ 3626.871269]  ? io_uring_del_task_file+0xc0/0xc0
[ 3626.871272]  process_one_work+0x220/0x3c0
[ 3626.871273]  worker_thread+0x4d/0x3f0
[ 3626.871275]  kthread+0x114/0x150
[ 3626.871277]  ? process_one_work+0x3c0/0x3c0
[ 3626.871278]  ? kthread_park+0x90/0x90
[ 3626.871280]  ret_from_fork+0x22/0x30
[ 3626.871283] INFO: task kworker/u16:171:274174 blocked for more than 120 seconds.
[ 3626.877023]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.881287] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.887192] task:kworker/u16:171 state:D stack:    0 pid:274174 ppid:     2 flags:0x00004000
[ 3626.887196] Workqueue: events_unbound io_ring_exit_work
[ 3626.887198] Call Trace:
[ 3626.887199]  __schedule+0x2de/0x890
[ 3626.887201]  ? check_preempt_wakeup+0xfd/0x210
[ 3626.887203]  schedule+0x4f/0xc0
[ 3626.887207]  schedule_timeout+0x202/0x290
[ 3626.887209]  ? ttwu_do_activate+0xb5/0x3f0
[ 3626.887212]  wait_for_completion+0x94/0x100
[ 3626.887216]  io_ring_exit_work+0x18c/0x500
[ 3626.887218]  ? io_uring_del_task_file+0xc0/0xc0
[ 3626.887221]  process_one_work+0x220/0x3c0
[ 3626.887222]  worker_thread+0x4d/0x3f0
[ 3626.887225]  kthread+0x114/0x150
[ 3626.887228]  ? process_one_work+0x3c0/0x3c0
[ 3626.887229]  ? kthread_park+0x90/0x90
[ 3626.887232]  ret_from_fork+0x22/0x30
[ 3626.887236] INFO: task iou-sqp-278149:278150 blocked for more than 120 seconds.
[ 3626.892947]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.897266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.903175] task:iou-sqp-278149  state:D stack:    0 pid:278150 ppid:     1 flags:0x00004002
[ 3626.903177] Call Trace:
[ 3626.903178]  __schedule+0x2de/0x890
[ 3626.903180]  schedule+0x4f/0xc0
[ 3626.903182]  io_uring_cancel_sqpoll+0xdb/0x110
[ 3626.903184]  ? wait_woken+0x80/0x80
[ 3626.903187]  io_sqpoll_cancel_cb+0x24/0x30
[ 3626.903188]  io_run_task_work_head+0x28/0x50
[ 3626.903191]  io_sq_thread+0x4ef/0x720
[ 3626.903194]  ? wait_woken+0x80/0x80
[ 3626.903195]  ? recalc_sigpending+0x1c/0x60
[ 3626.903198]  ? io_submit_sqes+0x14c0/0x14c0
[ 3626.903200]  ret_from_fork+0x22/0x30
[ 3626.903204] INFO: task NIO-ELT-1-#0:278151 blocked for more than 120 seconds.
[ 3626.907496]       Tainted: G        W         5.12.0-rc6-5.13-uring-210407 #1
[ 3626.911758] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3626.917652] task:NIO-ELT-1-#0    state:D stack:    0 pid:278151 ppid:     1 flags:0x00024000
[ 3626.917654] Call Trace:
[ 3626.917655]  __schedule+0x2de/0x890
[ 3626.917657]  ? free_pages_and_swap_cache+0xb9/0xd0
[ 3626.917662]  schedule+0x4f/0xc0
[ 3626.917664]  schedule_timeout+0x202/0x290
[ 3626.917667]  wait_for_completion+0x94/0x100
[ 3626.917669]  __io_uring_files_cancel+0x17d/0x2e0
[ 3626.917672]  ? blk_finish_plug+0x26/0x40
[ 3626.917676]  ? io_uring_cancel_sqpoll+0x110/0x110
[ 3626.917678]  do_exit+0xc0/0xaf0
[ 3626.917681]  ? exit_to_user_mode_prepare+0x3d/0x1a0
[ 3626.917685]  __x64_sys_exit+0x1b/0x20
[ 3626.917687]  do_syscall_64+0x38/0x90
[ 3626.917690]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 3626.917695] RIP: 0033:0x7f1b2570a6c6
[ 3626.917697] RSP: 002b:00007f1b22077e80 EFLAGS: 00000246 ORIG_RAX: 000000000000003c
[ 3626.917699] RAX: ffffffffffffffda RBX: 00007f1b22078700 RCX: 00007f1b2570a6c6
[ 3626.917700] RDX: 000000000000003c RSI: 00000000007fb000 RDI: 0000000000000000
[ 3626.917701] RBP: 00007f1b21878000 R08: 0000000000000006 R09: 0000000100000000
[ 3626.917702] R10: 00007f1b257054e0 R11: 0000000000000246 R12: 00007ffd3c3f149e
[ 3626.917704] R13: 00007ffd3c3f149f R14: 00007ffd3c3f14a0 R15: 00007f1b22077f40

Was thinking it's because of not applied https://github.com/isilence/linux/commit/42c74362eb3997a543b648260fe2ad8d2266b786, but actuallt it looks different.
There is a very alike syzbot report, makes sense to solve it first. I'll let you know when it's sorted out.

Ok, super - let me know and I'll retest then.

What I see in logs:
userspace task1: exit() -> cancel -> waits sqpoll1 to cancel (via task work)
sqpoll1: io_uring_cancel_sqpoll() -> hangs in req cancel (probably, unable to cancel some requests)
exit_work1: short on refs as well, maybe same reason as for sqpoll1
exit_workN: waits for some task to task_work_run(), maybe for userspace task1
It very well may be a not put poll request that we forgot somewhere. I need to take care of 5.12 first, but can you run a branch for some debug output? It's on top of the Jens' patch. Logs may overflow, so better to dmesg > file from the beginning of a run.
https://github.com/isilence/linux/commits/poll_bugs_test

Sure, here is the output from the run - I didn't get the usual bells in the console but there is info in the output about the usual hangs (and basically each process just hung hard taking the shell with it when the process trying to exit, 100%). For the record, run with:
commit ed68098faf1f661231a7e2cc3485861e09425baa (HEAD -> poll_bugs_test, origin/poll_bugs_test)
dmesg.txt

@hassila, do you poll pipes or std[in,out,err]? If so, do you have poll requests having  both read and write in its poll mask?

Sockets - read, write and both at times - it's a big test suite...

@hassila, can you try a new branch below? The last patch, which also you CC'ed and sent by email, fixes one kind of sqpoll related hangs. Let's see if it's yours.
https://github.com/isilence/linux/tree/sqpoll_hangs_tests

Nope, not quite - this time processes gets stuck with cpu:s running, attaching screenshot and dmesg output.

dmesg.txt

Are you sure you used right kernel version? That branch doesn't have debug printk()'s like below.
what's uname -r?
[  605.598521] p 0000000000000000 sqd 0000000000000000 ctx 0000000023460f9d | exit work hang 1

Ouch. Sorry, mea culpa. I had two lines in my grub config to easily switch between kernels and managed to update the commented out one... Instance rebooting now, retest in progress.

Ok, now this time with the right kernel - no CPU, but my processes hangs on exit. dmesg:
dmesg.txt

And missing from that dmesg.txt something came right after:
[  364.310769] R13: 00007ffd338fe2ff R14: 00007ffd338fe300 R15: 00007f0cb4807f40
[  502.613079] ------------[ cut here ]------------
[  502.613082] WARNING: CPU: 2 PID: 8 at fs/io_uring.c:8592 io_ring_exit_work.cold+0x0/0x16
[  502.613089] Modules linked in: binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua isst_if_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl ppdev parport_pc parport ena i2c_piix4 serio_raw sch_fq_codel drm i2c_core ip_tables x_tables autofs4
[  502.613104] CPU: 2 PID: 8 Comm: kworker/u16:0 Not tainted 5.12.0-rc6-5.13-uring-210413 #1
[  502.613106] Hardware name: Amazon EC2 c5.2xlarge/, BIOS 1.0 10/16/2017
[  502.613107] Workqueue: events_unbound io_ring_exit_work
[  502.613111] RIP: 0010:io_ring_exit_work.cold+0x0/0x16
[  502.613114] Code: a7 6a ff ff 48 83 c8 ff e9 7b 31 87 ff 48 c7 c7 90 8b d8 b0 c6 05 c4 af e0 00 01 e8 8b 6a ff ff b8 02 00 00 00 e9 f7 4e 87 ff <0f> 0b 48 c7 c7 00 96 d8 b0 4c 89 e6 e8 70 6a ff ff e9 21 43 88 ff
[  502.613115] RSP: 0018:ffffa63f8004bdd0 EFLAGS: 00010283
[  502.613117] RAX: 000000010000c531 RBX: ffff904ed5fd5df0 RCX: 0000000000000009
[  502.613119] RDX: 0000000000000001 RSI: ffff904ed5fd58c8 RDI: ffff904ed5fd5de8
[  502.613120] RBP: ffffa63f8004be50 R08: 0000000000000000 R09: 0000000000000000
[  502.613121] R10: 000000000000000f R11: 0000000000000000 R12: ffff904ed5fd5800
[  502.613121] R13: ffff904ed5fd5c30 R14: 0000000000000000 R15: 000000010000c52c
[  502.613123] FS:  0000000000000000(0000) GS:ffff9051cbc80000(0000) knlGS:0000000000000000
[  502.613124] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  502.613125] CR2: 00007faab237d9dc CR3: 0000000107c80002 CR4: 00000000007706e0
[  502.613128] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  502.613129] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  502.613130] PKRU: 55555554
[  502.613131] Call Trace:
[  502.613135]  ? __switch_to+0x192/0x460
[  502.613140]  ? __switch_to_asm+0x36/0x70
[  502.613143]  process_one_work+0x220/0x3c0
[  502.613147]  worker_thread+0x4d/0x3f0
[  502.613149]  kthread+0x114/0x150
[  502.613152]  ? process_one_work+0x3c0/0x3c0
[  502.613154]  ? kthread_park+0x90/0x90
[  502.613156]  ret_from_fork+0x22/0x30
[  502.613159] ---[ end trace ff7f99fcda4828df ]---
[  502.613160] task failed NIO-ELT-0-#3
[  502.665076] task failed NIO-ELT-0-#3
[  502.717070] task failed NIO-ELT-0-#3
[  502.769067] task failed NIO-ELT-0-#3
[  502.821066] task failed NIO-ELT-0-#3
[  502.873065] task failed NIO-ELT-0-#3
[  502.925064] task failed NIO-ELT-0-#3
[  502.977064] task failed NIO-ELT-0-#3
[  503.029062] task failed NIO-ELT-0-#3
[  503.081061] task failed NIO-ELT-0-#3
[  503.133060] task failed NIO-ELT-0-#3
[  503.185058] task failed NIO-ELT-0-#3
[  503.237059] task failed NIO-ELT-0-#3
[  503.289056] task failed NIO-ELT-0-#3
[  503.341055] task failed NIO-ELT-0-#3
[  503.393055] task failed NIO-ELT-0-#3
[  503.445053] task failed NIO-ELT-0-#3
[  503.497052] task failed NIO-ELT-0-#3
[  503.549052] task failed NIO-ELT-0-#3
[  503.601050] task failed NIO-ELT-0-#3
[  503.653048] task failed NIO-ELT-0-#3
[  503.705047] task failed NIO-ELT-0-#3
[  503.757046] task failed NIO-ELT-0-#3
[  503.809045] task failed NIO-ELT-0-#3
[  503.861045] task failed NIO-ELT-0-#3
[  503.913043] task failed NIO-ELT-0-#3
[  503.965042] task failed NIO-ELT-0-#3
[  504.017041] task failed NIO-ELT-0-#3
[  504.069039] task failed NIO-ELT-0-#3
[  504.121039] task failed NIO-ELT-0-#3
[  504.173038] task failed NIO-ELT-0-#3
[  504.225036] task failed NIO-ELT-0-#3
[  504.277034] task failed NIO-ELT-0-#3
[  504.329033] task failed NIO-ELT-0-#3
[  504.381032] task failed NIO-ELT-0-#3
[  504.433031] task failed NIO-ELT-0-#3
[  504.485030] task failed NIO-ELT-0-#3
[  504.537028] task failed NIO-ELT-0-#3
[  504.589027] task failed NIO-ELT-0-#3
[  504.641026] task failed NIO-ELT-0-#3
[  504.693027] task failed NIO-ELT-0-#3
[  504.745024] task failed NIO-ELT-0-#3
[  504.797024] task failed NIO-ELT-0-#3
[  504.849023] task failed NIO-ELT-0-#3
[  504.901022] task failed NIO-ELT-0-#3
[  504.953021] task failed NIO-ELT-0-#3
[  505.005018] task failed NIO-ELT-0-#3
[  505.057017] task failed NIO-ELT-0-#3
[  505.109014] task failed NIO-ELT-0-#3
[  505.161014] task failed NIO-ELT-0-#3
[  505.213014] task failed NIO-ELT-0-#3
[  505.265012] task failed NIO-ELT-0-#3
[  505.317011] task failed NIO-ELT-0-#3
[  505.369010] task failed NIO-ELT-0-#3
[  505.421010] task failed NIO-ELT-0-#3
[  505.473008] task failed NIO-ELT-0-#3
[  505.525007] task failed NIO-ELT-0-#3
[  505.577006] task failed NIO-ELT-0-#3
[  505.629006] task failed NIO-ELT-0-#3
[  505.681006] task failed NIO-ELT-0-#3
[  505.733004] task failed NIO-ELT-0-#3
[  505.785003] task failed NIO-ELT-0-#3
[  505.837001] task failed NIO-ELT-0-#3
[  505.889000] task failed NIO-ELT-0-#3
[  505.941000] task failed NIO-ELT-0-#3
[  505.992998] task failed NIO-ELT-0-#3
[  506.044997] task failed NIO-ELT-0-#3
[  506.096996] task failed NIO-ELT-0-#3
[  506.148994] task failed NIO-ELT-0-#3
... repeating ...

Blood, how many of them is in there... Don't worry, 5.13 hasn't even started yet, we'll get it nailed, was focusing on things that are more time critical

@hassila, the last patch set you've been CC'ed did went in, and the problem it solves looks pretty much as yours. Can you test it out?
https://git.kernel.dk/cgit/linux-block/log/?h=for-5.13/io_uring

Sure @isilence - tomorrow or Thursday latest I think - will let you know.

Perfect, thanks Joakim

Ok, I've run tests now for 20 minutes hammering it using f2a48dd09b8e933f59570692e1382b81d4fddc49 - it's been rock solid and no dmesg output. It definitely would have had issues with earlier versions, so seems you nailed it, well done! I think we can close this. Thanks!

authors: hassila isilence

repo: axboe/liburing

license: mit

license_source: bigquery


text:
utilizing wq_fd together with IORING_SETUP_ATTACH_WQ returns EINVAL

We are trying to utilize multiple threads to submit and reap i/o events. As suggested we initialize the first ring as normal and it works. But when we set the above on the second uring it returns error EINVAL. Does this feature even work at the moment?
#define IO_URING_DEPTH 4
struct io_uring_params params;
memset(&params, 0, sizeof(params));
params.flags |= IORING_SETUP_SQPOLL;
params.sq_thread_idle = __SQ_THRD_IDLE;
if(__first_ring_fd_set) {
/* this allows multiple user io_urings to utilize the same async
* backend. Thus it should utilize the same SQLPOLL thread as well */
params.wq_fd = cntxt->ring.ring_fd;
params.flags = IORING_SETUP_ATTACH_WQ;
}
if((rc = io_uring_queue_init_params(IO_URING_QDEPTH, &cntxt->ring, &params))!=0) {
rc = -1;
goto cleanup;
}
if((rc = io_uring_register_files(&cntxt->ring, &fd, 1))!=0) {
rc = -1;
goto cleanup;
}

What kernel? And are the rings created from the same task?

Thanks for the response, This is being done in linux 5.4 in AWS linux

5.4 is too old to support that feature. Basically -EINVAL here means "some flag you set in the ring setup is not supported".

What is the oldest linux version that supports that feature. We really need to get this working as soon as we can.

5.6 would be the oldest, but honestly if you're swapping kernels, I'd recommend using the latest stable. Even 5.6 is pretty old in that regard, that's almost 2 years ago.

We can utilize linux kernel-5.10. Hopefully it will work well enough there. Do we have to download a specific version of liburing code for that kernel? Or can we simply download the most recent version of liburing and it will know that we are working on kernel-5.10 and support features that it can?
Basically we create the first ring single threaded and the remaining rings are created concurrently... hopefully there is no issue with concurrently created/destroying rings. If there is we can isolate those code points as they are quite infrequent.
regards

5.10 seems reasonable. The current release of liburing will work fine, there are really no library dependencies for this feature.

thanks so much for the quick response. We will download the most recent version of liburing from Github ...and i will close this issue.

I spent the whole of last day upgrading to Linux 5.10 in AWS and i also downloaded the latest version of liburing from Github and rebuilt everything from scratch and i still get EINVAL when utilizing wq_fd together with IORING_SETUP_ATTACH_WQ.
The code is as indicated in the first comment. And this is what i see with uname -r:
5.10.50-44.132.amzn2.x86_64
We are doing this in docker containers. However the host is also the same kernel.  Is it possible that this is not working because we are utilizing docker containers?
Any help is greatly appreciated.

Could very well be that it's not working since it's in a docker container, I really have very little idea about that. I'd recommend cloning the liburing repo and building it all, then run test/shared-wq from there. That's a basic functionality test. If that works, then you are likely doing something wrong. If it doesn't, then likely docker is doing something funky.

It sounds strange that it should be an issue. We are also utilizing this in combination with  IORING_SETUP_SQPOLL.
Is it known to work together with IORING_SETUP_ATTACH_WQ?
Allright i will rebuild and run. So far we were running in Docker and without ATTACH_WQ some of our tests pass.

Yes, it'll work with SQPOLL, in fact liburing also has a test case for that, test/sq-poll-share

When rebuilding liburing i do run into the following problem:
io_uring_register.c:509:10: warning: implicit declaration of function 'memfd_create'; did you mean  timer_create'? [-Wimplicit-function-declaration]
memfd = memfd_create("uring-shmem-test", 0);
^~~~~~~~~~~~
timer_create
/tmp/cc91MvfX.o: In function `test_shmem':
after a bit of looking around it seems that including <syscall.h> together with utilizing SYS_memfd_create seems to make it work and additionally all the tests appear to pass, i ran
source ./test/runtests.sh
So the tests seem to pass ...
So inside the Docker container, all the tests pass. But when we run this in the product it fails with EINVAL. Is there anyway we can debug this further to see what may be causing it to return EINVAL. Is there some system call it is attempting which returns EINVAL. If so we can poke around to see what may be causing it to fail.

should we fall back on strace to see why it failed ... let me try that?

Looking at your example:
if(__first_ring_fd_set) {
/* this allows multiple user io_urings to utilize the same async
* backend. Thus it should utilize the same SQLPOLL thread as well */
params.wq_fd = cntxt->ring.ring_fd;
params.flags = IORING_SETUP_ATTACH_WQ;
}

if((rc = io_uring_queue_init_params(IO_URING_QDEPTH, &cntxt->ring, &params))!=0) {
rc = -1;
goto cleanup;
}

it's not clear to me what is being set in params->wq_fd - just to be clear, that one should be set to the ring fd of the original ring, and I can't tell if that's what you're doing as cnxt appears to be the new ring being registered. But it could just be that you're not including all of the code, and cnxt is allocated after that setting and it's the original ring to begin with?
In any case, if the test cases work, then you're doing something wrong in your code. -EINVAL is a horrible error code, but there's really nothing else that covers it. It'll get set for any invalid field in the params or setup.

io_uring_queue_init_params() boils down to a io_uring_setup(), and then later ring mmaps if that succeeds. If you strace, you're most likely seeing io_uring_setup() return -EINVAL due to some invalid parameter.

You are right about that one. That is a copy paste error on my part. Sorry to bother you. Let me try to address that one.

Thanks so much for your help. Im really sorry for taking up your time for what is my bug. Hopefully this fix will work. regards.

It's Friday, wasting a bit of time on this is healthier than drinking beer ;-)

We are hoping that the SQPOLL will setup one thread which will be polling for the sqes. And our application on the other hand has potentially 100 threads each one of which has a tiny seq, cqe ring of length 4/8. All threads are actively submitting and reaping ... hopefully we should be able to leverage the random i/o aspects of SSDs with that. If you are interested i can keep you posted on any performance reads we see.
We are looking at doing random i/o on massive databases. In the ballpark of a few TBs.
You have done a great service to the community by providing this library. Incidentally we also intend to look closely at SPDK which is utilizing a user mode polling model. In some usage scenarios that might offer some further advantages.

Definitely, keep us updated. If completion polling ends up being interesting as well, you can use IOPOLL with SQPOLL, fwiw.

Sorry i read some articles which indicate that IOPOLL should not be combined with SQPOLL ... i don't know what the reasoning was provided ...  if you search for io_uring samsung, performance on google you will find the pdf document that talks about it ... it also compares it with SPDK.
Don't know why they suggest not combining IOPOLL with SQPOLL.

Below is the link, they suggest do not utilize IOP with SQPOL
https://www.usenix.org/sites/default/files/conference/protected-files/vault20_slides_lund.pdf

After the fix i get the error: ENXIO ... probably some incorrect fd sent it ... let me check ...

Here is what i am finding about ENXIO:
The test sq-poll-share.c  does not utilize io_uring_register_files() to register fd.  From what i can see we have to utilize this API in order to register files for SQPOLL. If we do not utilize this API we can't even make a most basic test work. So there is something amiss.

Incidentally we have a separate test that utilizes SQPOLL and it works very well and we follow nearly the same steps.
Its when we try to go multi-threaded that we run into issues. At the moment i am getting error ENXIO when invoking
io_uring_queue_init_params().

closing the issue for now. It looks like sq-poll-share.c test is not working on linux. We have to understand that, and hopefully that will resolve this issue.


text:
link to axelarscan when deploying tokens

When deploying a token, link to axelarscan through the Initiated/Confirmed status, instead of just the top right buttons

Not sure if i understand this correctly, but we already had multiple links to axelarscan. Feel free to reopen the issue if i misunderstand this 🙇🏻

authors: canhtrinh npty

repo: axelarnetwork/axelarjs


text:
Limit Sign up form width

4 grid columns

Fixed in commit https://github.com/axelpale/tresdb/commit/218bfcc5fa14bc1f446138746e23d8a6e7bfb4c0

repo: axelpale/tresdb


text:
Changing bacticks to (double) quotes

Hi,
Are there any plans to support converting bacticks to quotes when user deletes a variable? I noticed in the gif (see below) from vscode plugin that it has the option to go both ways. Would be great to have both.

Very nice and simple plugin btw, very useful.

Yeah, I'd like to implement this feature, but I need to think more about how to implement it to avoid problems like these: meganrogge/template-string-converter#43

Wasn't aware of those issues. I think there should still be a branch that has it implemented as simple as possible, and have people test it out. Most probably it will not have some of those issues vscode plugin has, but also some other issues might arise.

Awesome, thanks!

@axelvc I tried using the feature but cant manage to make it work. Can you give me a simple explanation or add it to the README? Appreciate a lot your effort 💪🏼

Can you give me a simple explanation or add it to the README?

You need to enable the option in the setup function. That should be enough
require('template-string').setup {
  remove_template_string = true,
}

I made it work by deleting the character with x under normal mode. Is it the way its supposed to be working?

The feature should work in Normal and Insert mode, but there are some cases where the plugin doesn't remove the backticks. You can see them here: https://github.com/axelvc/template-string.nvim/pull/4#issue-1342876268
If you've found a new issue, feel free to report it by opening an issue.

authors: axelvc camiloaromero23 kristijanhusak

license: MIT


text:
Windows Universal Platform (UWP) Support?

Can you add Windows Universal Platform (UWP)?
It was available on older Cocos2d-x 3 versions

Yes, we have plan to support UWP: https://github.com/c4games/engine-x/projects/2

Usefull for Xbox One/X Platform.

Yes and also Windoes store. In the past Cocos2dx support uwp, I think it had support in version 3.15

Maybe this is helping to convert/add UWP to adxe:
https://github.com/Microsoft/DesktopBridgeToUWP-Samples
Desktop Conversion Extensions is a bridge that enables you to convert your classic desktop application (like Win32, Windows Forms, and WPF) or game to a Universal Windows Platform (UWP) app or game.

It works for Desktop applications, Xbox still requires the full implementation

I confirm, winRT support that allows to produce UWP format was available in 3.15 and is easy to copy/paste for support in 3.17.

There is a problem with the audio player as of 3.17, but taking advantage of the unified OpenAL the project has, it might be easier to debug caching problems. There is also no gamepad support.

I will share you my cocos2d-x 4.0 UWP port, and start working on a fork, I just don't have a few prebuilt libraries from the dependencies/thirdparty folder, it's stuck at the linking phase, I think I have the lua thing built already

This one is it:
https://github.com/Greentwip/time-trio
UWP Port of 4.0 version.
Problem is some libraries need to be compiled from source, I've already compiled angle, lua, openssl I gues? But bullet, and some other need their precomps. I'll try to follow up with the adxe structure, right now I don't know how to disable glfw and purely add the missing UWP files using OpenGL.

Not probably UWP (I'm almost done with the port) but you can try this Xbox port I've finished compiling, you'll just need the open Microsoft GDK (or sign up for the internal Microsoft GDK):
https://github.com/balancedwolf/adxe
See: CMakeGDKXboxOne.cmake and CMakeLists.txt from the time-trio repo to build your game.

@hal99,
Hope this is still active and part of axys 1.1

@halx99
please add labels:
pinned
help wanted

https://github.com/Greentwip/cocos2d-x
XBOX one port

axmol UWP screenshot:

axmol UWP screenshot:

Amazing!

Time to clean my XBOX One ;)

authors: Obg1 aismann appakabar balancedwolf bensgigi halx99

repo: axmolengine/axmol


text:
Refactor dist init and split out dist migrate

dist init still works as on main
dist migrate does the migration part of dist init
dist init's migration functionality is now implemented in terms of dist migrate

This shows dist migrate on its own:

And this shows that dist migrate is only a migration, and explicitly does not do the re-init part:

Once #1612 is merged, I'll rebase this off main and those changes won't be in this PR's diff.

repo: axodotdev/cargo-dist


text:
bitwise ops across arrays

[1 0 1][1 0 0] &
[1 0 1][1 0 0] |

Implemented in #78

repo: aya-lang/aya


text:
Transfer of ViennaRNA PyPI ownership

Dear Ayaan Hossain,
I wonder whether you are willing to transfer ownership and maintainership of the PyPI ViennaRNA package project to us as the upstream development team.
You haven't been updating this repository since May 2022 and it seems you even provide a sort of pre-release. From what I can see at the moment, there is no easy way to tell whether you include the official upstream releases into your PyPI package or whether you patch them to some extent.
We believe it makes much more sense if we integrate submissions and updates to PyPI into our regular release workflow instead. This will ensure that all updates that are released with new versions of the ViennaRNA Package find their way into the PyPI package within a reasonable time. At the same time it would avoid any confusions of our users who don't know that the PyPI package is not maintained by us.
Please let me know about your decision.
Best,
Ronny

Hi Ronny, thanks for reaching out. Yes, I usually only try to maintain the latest version. I last updated the 2.5.0 version and since then there's been 2.5.1 and now 2.6.0. I can consider transferring ownership of the name in PyPI so your team can push your releases to PyPI, but I was wondering about the following:

Currently, I compile the source on the user's machine and this installs all the relevant SWIG interface code to the user. Afterwards, I delete the compiled files, and the python interface is the only thing that remains. Will you make sure the release you push to PyPI just installs the python interface?

There is no good way to specify the parameter files, other than the user somehow having to download them separately and point to them to the RNA.read-parameter_file function (apologies if I get the name wrong, it's mentioned in the README with this repo). Can you make sure when the user installs the package, it is as easy as specifying a string to load the parameters?

The python interface is now just a SWIG interface to the underlying C code. Will your team spend some time to flesh out a more curated python interface? It'd help if you have a series of documents explaining all the different interface functions, their proper parameters, usage, purpose, return type etc.

Let me know, what you think.
Ayaan

@RaumZeit @ayaanhossain
I trust you are both doing well. As a user and enthusiast of the ViennaRNA project, I have been closely following your conversation regarding the Python package support.
Firstly, let me express my heartfelt gratitude for the valuable work you both have done. Ayaan, your initiative to package ViennaRNA for PyPI and maintain it has been of immense value. Ronny, your willingness to take over the PyPI package to provide official updates is a testament to the commitment of the original development team towards the project and its users.
I noticed that it's been a week since your last exchange. As an interested party and a user of the ViennaRNA package, I would like to encourage you both to continue this important conversation. Your ongoing dialogue on this topic is vital for the future development and support of the ViennaRNA Python package.
Furthermore, I am excited to see the integration of GitHub actions into your project pipelines. It would be great to see the Python package become an integral part of the official CI/CD. This would not only ensure the efficient and effective maintenance of the project but would also solidify the trust of the user community in the ViennaRNA package.
Thank you both for your dedication and commitment to this project. I am sure many users like myself eagerly await the outcomes of your ongoing discussion and collaboration.
Looking forward to the continued growth and success of the ViennaRNA project.
Best regards,
Abed

Hi @ayaanhossain and @AH-Merii ,
I'm pretty busy these days with giving lectures at the university, so I can't dedicate too much time to this project during the next two weeks or so. But, we are still willing to proceed on that matter.
So let me first answer the questions @ayaanhossain mentioned:

Sure, for the PyPI version, we will only install the Python interface and no additional other parts of the ViennaRNA Package, That should be easily possible, since we also do that for our binary packages that we distribute for several linux distros.

While we ship energy parameter files, those are mostly intended to be used by our executable programs. In our library, we already provide dedicated API functions that load these parameters independently without the need for these files. So, in essence, we already include all parameter files into our library. They have just not been exposed as strings so far but required the call of high-level functions that would load these parameters, e.g. RNA.params_load_Andronescu2007() would load the Andronescu et al. 2007 parameter set.
For the next release, I even added support to actually export these parameters as strings that represent the content of the parameter files. Long story short, no parameter files are required for the Python interface, everything is already compiled into the library and available through the API.

While we use SWIG to create the Python bindings for the ViennaRNA C-library we add an object oriented layer that transforms many of our C functions to methods of the respective objects in Python. I understand that this renders the available reference manual documentation somewhat useless for the Python user. However, with version 2.6.0 we introduced an automatic conversion of the Doxygen documentation into Python docstrings that, for the most part, recognizes these transformations from functions to methods. This conversion and the creation of docstrings finally allowed us to provide a Python API documentation which will be growing in the future. You can already find the latest documentation at Read the Docs

As for the github actions and the creation of the PyPI packages, this is exactly the way we want to go. We want to fully automate the process such that whenever we create a new release, this will trigger a github action that then uploads the required artifacts to PyPI.

That's all for now from my side and I'll be more focused on bringing this project forward in about 2 weeks from now.
Cheers,
Ronny

I forgot to mention that for point 3. we plan to eventually merge both of the created documentations into one.
I'm already in the process to transform our doxygen output to sphinx-generated html websites via the breathe bridge. But this is a tedious task that requires many changes in the way we currently document the API. But once this is done, a merge of both the C-API with all its extra descriptions and example codes and the Python documentation we extract from the generated docstrings should be easy to accomplish.

@RaumZeit
Thank you for taking the time to provide such a detailed update amidst your busy schedule. It's truly appreciated!
I'm incredibly excited about the upcoming changes you've outlined for the ViennaRNA Python package. The improvements in the PyPI version, API documentation, and the automated CI/CD practices sound very promising.
Looking forward to these developments in the coming weeks. 😁

@AH-Merii @ayaanhossain:
Dear Ayaan and Abed,
I started to refactor the Python interface build process for the ViennaRNA Package and everything seems good so far. I use setuptools to build RNAlib and the swig extension and prepare/compile the sdist and bdist_wheel packages. Now, that I wanted to test whether everything works correctly by pushing it to test.pypi.org, I realized that the ViennaRNA project at test.pypi.org is taken by @AH-Merii . I tried reaching out via email a few days ago and today. but didn't get a response yet. That's why I write here now. Would you, @AH-Merii, be so kind to add me to the list of maintainers for your project at test.pypi.org?! This would allow me to generate an API token to upload the artifacts. Otherwise, I'd need to create a package with a different package name that would render testing much more tedious.
As soon as everything looks right, I start the github actions integration via cibuildwheel to build binary packages for Linux, macosx and hopefully windows. Once that also works out as expected, I'd need access to the ViennaRNA project at pypi.org from @ayaanhossain .
Just as a side note, I intend to keep the
import ViennaRNA

thing next to our official
import RNA

Both will actually load the same set of functions, classes and variables. Also, the RNA/__init__.py now is an actual package description file instead of a copy of the RNA.py generated by swig. So, the swig wrapper (RNA.py) itself is now an actual module (RNA.RNA) and part of the RNA package.
For future purposes, I think splitting the library into smaller sub packages is the way to go, e.g.:
from ViennaRNA import plotting

and things alike. But this requires some more restructuring in the SWIG interfaces and a shared library for libRNA.  So, this change will likely come not earlier than version 3.0.0.
Thanks for your understanding and commitment!
Best,
Ronny

Hi Ronny, thanks for the initiative. I'm happy to see that you're focusing on a nice python interface for the RNAlib functions. I have added you to the pypi package for ViennaRNA, please feel free to start playing around with it. I have added you as a Maintainer, and once you confirm that you've received the invite, I will promote you to full owner. Best, Ayaan

Thanks a lot to the both of you!
I very much appreciate your trust in me/us working on the ViennaRNA Package.
The current state for the PyPI packages is that my pipeline runs fine for Linux and MacOS. For the latter, I was also able to build arm64 wheels today, so they will be included as well. What doesn't work is support for PyPy, since our C Extension doesn't seem compatible with their C layer. Windows builds need some more changes to the source code and will most likely follow in the future, once I find the time to rewrite the upstream code to support building with MSVC. I'll test upload of the sdist and wheel packages to test.pypi.org later this week.
I'll let you know once the tests for test.pypi.org are successful.
Best,
Ronny

Well, @ayaanhossain:
My pipelines are set up and working, publishing to both TestPyPI and PyPI works (they show version 2.6.0 now) and will be triggered automatically whenever a new release is issued on github. The good thing is that most users might not even have to compile the package, since the new build pipeline already provides Python 3.8 - 3.11 builds for Linux and MacOS X. As mentioned earlier, I'll have a look at Windows builds as well, but this might take some more time as it requires more changes in our code base....
Thanks again for everything! I'll have a look at the content of your git repo later to see which parts might be interesting to keep for the https://viennarna-python.readthedocs.io site. Also I try to add more documentation/examples on the energy parameter settings via the API.
Best,
Ronny
P.S.: you can make me owner of the PyPI project now
P.S.S.: The github actions workflow that builds and publishes everything is in our development branch and will be accessible once the next release is pushed to github. Just in case you are interested. The same is true for the setup.py, setup.cfg and pyproject.toml files I used to create the Python packages (Although they are already included in the sdist packages on PyPI)

Congrats Ronny! I've promoted you to owner on the PyPI. Thank you and good luck! Hope the new python interface is really pythonic and easy to use for design and evaluation of RNA sequences. Ayaan

authors: AH-Merii RaumZeit ayaanhossain

repo: ayaanhossain/ViennaRNA


text:
evaluate clang build with qt

in time of r13 Qt was able to build only with gcc/r10 - check compatibility

Imagemagick7 test was tested with clang/gcc on real hardware(armeabi-v7a) and found to be working with Qt 5.9.2.
detailed evaluation, including performance comparation, will be a separate task inside IMTest project


text:
🛑 Obase is down

In 56879c9, Obase (https://www.obase.com/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Obase is back up in 28175ee after 10 minutes.

repo: aydgn/upptime


text:
Cannot Mock Class which has Array paramater in constructor

Code :
public class Service{
    public Service(Command[] commands) {  }
}

public abstract class Command {  }

var mock = MockRepository.GenerateMock<Service>(new Command[] { }); // This fails

This is just a syntax issue. You passed an empty object array as a parameter - so the exception stating that there is no parameterless constructor is totally right.
Just write:
var mock = MockRepository.GenerateMock(new object[] {new Command[] {}});

Thanks. I figured it out some time ago.

authors: alaendle ashishs

repo: ayende/rhino-mocks


text:
Feature: sort interface properties, object keys and array values if code has annotation

Describe the rule
There is existing package that implements similar feature: https://github.com/ronparkdev/eslint-plugin-annotation
I personally think it would be great if eslint-plugin-perfectionist provided thie rule since it's also related to sorting.
Code example
|  |  |
Additional comments
No response
Validations

[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

Currently, you can use ESLint comments to define plugin rule settings.
/*
  eslint perfectionist/sort-objects: [
    'error',
    {
      type: 'alphabetical',
      order: 'asc',
      partitionByNewLine: true
    }
  ]
*/

Using annotations looks beautiful and convenient. But I'm not sure it will be in demand.
I would like to hear more opinions from other users.

I understand the use case for sort-array-includes: it would allow users to sort arrays who do not have .include if the user wishes.
However, I'm not sure to understand the use case with the other rules such as sort-objects: all objects get sorted if the rule is active, why would an annotation be needed?

@sardor01 Perhaps the ESLint comment solves your problem?

repo: azat-io/eslint-plugin-perfectionist


text:
Version info in 1.2.0.3 is 1.2.0.0

Download https://github.com/azerg/NppBplistPlugin/releases/download/1.2.0.3/NppBplistPlugin_x86.zip
Unzip to C:\Downloads\NppBplistPlugin.dll, found:
Size: 2054656 Byte (1.96 MB)
Modified: 2017-01-27 21:34
Version: 1.2.0.0
MD5: db2d3ea89a526b9f315d3b0053770a93
SHA1: 6427201900041159d11d488e976b9a85dbda5be9
SHA256: 0c7f53d480d199c0cda37596747873d3e7e6365f3225abfa23e7a47edf0d3a44
CRC32: 77e20d7a

Also in x64 build, version info says 1.2.0.0.

Thanks. Will recompile with updated version info soon 🏎

fixed in 1.2.0.4

authors: azerg sunjw

repo: azerg/NppBplistPlugin


text:
【每日一题】- 2020-07-01 - TypeScript 编译器是用 TypeScript 写的，那是先有编译器还是TS？

TypeScript 编译器是用 TypeScript 写的，比如：
https://github.com/microsoft/TypeScript/blob/7b942b4fa875f2877a90d201cf146e6196b0c07b/src/compiler/scanner.ts
那么这个TS写的编译器是怎么被解析的呢？

先有 TypeScript。 TS 编译器的发展大致如下：

用别的语言（比如JS）写一个第一版编译器 V1
用 TS 写第二版编译器 V2，TS 需要被 V1 先编译
用 TS 写第三版编译器 V3，TS 需要被 V2 先编译
至此，就完成了自举（bootstrapping）


text:
ParamsSpec: Store Mapping Functions Separately

Enables the developer to not have to pass in params specs for items that had mapping functions.
Currently ParamsSpec and related types -- ParamsSpecFieldless, ValueSpec, FieldWiseSpec (generated by proc macro) -- hold MappingFns within their type.
This forces users to respecify the params specs with mapping functions in subsequent CmdCtx::builder_* instantiation.
This change is to:

Get developers to instantiate a Map<MappingFnK, Box<dyn MappingFn>>, and pass that to CmdCtx::builder_*.
ParamsSpec and related types will hold the MappingFnK.
Developers should define the MappingFnK, an all-unit-variant enum with sensible names to identify the mapping function.

Considering the value of this.
With It
Developers

Define an enum to name the function keys:
enum MappingFunctions {
    BucketNameFromBucketState,
    IamPolicyArnFromIamPolicyState,
}

Define the mappings:
let mapping_functions = {
    let mut mapping_functions = MappingFunctions::new();
    mapping_functions.insert(BucketNameFromBucketState, S3BucketState::bucket_name);
    // ..

    mapping_functions
};

Pass MappingFunctions to CmdCtxBuilder, for each code instantiation (may be just one):
cmd_ctx_builder.with_mapping_functions(mapping_functions);

Not have to call .with_item_params::<TheItem>(..) in subsequent calls.

Users

Get runtime error if the mapping function type doesn't match, but it should be caught by tests.

Framework Maintainers

MappingFunctions map will have magic logic to store the function argument types and return type.
Error reporting when types are mismatched.

Without It
Developers

Define the item params spec:
// First execution
let s3_object_params_spec = S3ObjectParams::<WebApp>::field_wise_spec()
    .with_file_path(web_app_path_local)
    .with_object_key(object_key)
    .with_bucket_name_from_map(S3BucketState::bucket_name)
    .build();

// Subsequent executions
let s3_object_params_spec = S3ObjectParams::<WebApp>::field_wise_spec()
    .with_bucket_name_from_map(S3BucketState::bucket_name)
    .build();

Pass the item params spec to CmdCtxBuilder, for every separate code instantiation:
cmd_ctx_builder
    .with_item_params::<S3ObjectItem<WebApp>>(
        item_id!("s3_object"),
        s3_object_params_spec,
    )

This is somewhat of an inconvenience, because if this isn't done, the user / developer will have a runtime error, which looks like this:
peace_rt_model::params_specs_mismatch

  × Item params specs do not match with the items in the flow.
  help: The following items either have not had a params spec provided previously,
        or had contained a mapping function, which cannot be loaded from disk.

        So the params spec needs to be provided to the command context for:

        * s3_object

When the closure passed to with_*_from_map doesn't have the argument type specified, or mismatches, the compilation error is still unclear. https://github.com/rust-lang/rust/pull/119888 will allow us to return a useful compilation error.
Users
No runtime error, because it will be caught at compile time.
Framework Maintainers

Error messages / diagnostics showing which CmdCtx is missing which item spec for which field, should be made clear.


text:
🛑 Amelia Bot is down

In 807b2ca, Amelia Bot (https://ameliabot-discord.uzumekiulee.repl.co/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Amelia Bot is back up in f0cede8 after 4 minutes.

repo: azrielbsi/monitor


text:
Unable to set timestamp for vm_extension and vmss_extension custom_script

It seems there is a bug in both vm_extension and vmss_extension custom_script. There is a function used (toint) which is not existing in terraform, there is only tonumber. The reason why it was not seen before is that try will supress the error.
   settings = jsonencode(
     {
-      "fileUris" : local.fileuris,
-      "timestamp" : try(toint(var.extension.timestamp), 12345678)
+      fileUris  = local.fileuris,
+      timestamp = try(tonumber(var.extension.timestamp), 1234568)
     }
   )

I will raise a PR to fix this issue.

Assigning to you waiting for the PR? ;)

authors: arnaudlh wasfree

repo: aztfmod/terraform-azurerm-caf


text:
ypromiseを外す

URL : http://azu.github.io/promises-book/#promise-library
流石に古いような感じ。問題はないんだけど、あんまりメンテされる空気感ではない(仕様変更ないので問題ないとは言えるけど、推奨とは言い難い感じ)
代わりを一ついれたい

[x] ypromise依存を外す -> getify/native-promise-onlyを使う
[x] ypromiseの紹介文を変更

es6-promisesではなくnative-promise-onlyを使う理由はない特にないのだけど、
es6以降もpromiseは微妙に変わっていってるのと、コードの読みやすさはNPOの方がよい。
今の仕様に一番厳密なのはcore-jsやes6-shim系の予感はある。これはspeicesとかの考慮が関係するので、promiseのpolyfillの範囲外に思える

repo: azu/promises-book


text:
🛑 azuma doa: Mailing Service is down

In c8eb444, azuma doa: Mailing Service (https://pie.azuma-health.tech/health/doa-mailing) was down:

HTTP code: 0
Response time: 0 ms

Resolved: azuma doa: Mailing Service is back up in 52baf2f.

repo: azuma-healthtech-public/uptime-pie


text:
Documentation or interface goodies for setting configurations on external IPC connections

Need a way to specify in code how to change configuration parameters for IPC connection to another node
Need a way to achieve same functionality listed above over the CLI

Either the above functionality, or explicitly do it in code + add docs on how to manipulate these internal conf params + allow users of library to create their own gen_server to handle this functionality in a more generic way for other gen_servers.

Resolved.

repo: azuqua/clusterluck


text:
Custom config?

How can I use pretty-quick with a custom eslint-prettier (.eslintrc) config?
My commits code is different than my development environment... :sweat_smile:

This would require prettier-eslint support, so I'll close this as a duplicate of #22.

authors: azz juandc

repo: azz/pretty-quick


text:
Using an external vector in selections was deprecated in tidyselect 1.1.0.

In perform_bootstrap_ts(): tidyr::pivot_wider(names_from = temporal_col_name,
Warning: Using an external vector in selections was deprecated in tidyselect 1.1.0.
## ℹ Please use `all_of()` or `any_of()` instead.
##   # Was:
##   data %>% select(temporal_col_name)
##
##   # Now:
##   data %>% select(all_of(temporal_col_name))

fixed in #13

repo: b-cubed-eu/indicator-uncertainty


text:
Source map does not resolve correctly for imported vulnerable contracts

We have a 2 contract setup, in which the main contract inherits a vulnerable contract:
vulnerableImport.sol
pragma solidity 0.5.0;

import "./imported.sol";

contract C is Imported {}

imported.sol
pragma solidity 0.5.0;

contract Imported {
    function f() public {
        selfdestruct(msg.sender);
    }
}

Scanning finds the correct selfdestruct problem:
$ sabre ./vulnerableImport.sol
✔ Compiled with solc v0.5.0 successfully
[    ] Analyzing C(node:30670) V8: :3 Invalid asm.js: Invalid member of stdlib

./vulnerableImport.sol
  6:2  error  The contract can be killed by anyone  https://smartcontractsecurity.github.io/SWC-registry/docs/SWC-106

✖ 1 problem (1 error, 0 warnings)

However the source mapping does not point to the correct line number. Reported location is 6:2 but line 6 does not exist in the vulnerableImport.sol file.
Observation
Doing a few more tests I found an interesting behavior. Adding an extra empty line in the vulnerable contract (before the vulnerable code) will increase 6:2 to 6:3.

Yeah, it looks like there is something wrong with it, I will try to find out the reason and fix it If no one has started to solve this problem yet.

@cleanunicorn -- Thanks for the example. I'll take a quick look at it and see how I can resolve the source map issue. The problem with imports and their source map linking is something that I haven't debugged yet. Unfortunately solc doesn't support resolving imports out of the box. So we're handling it within sabre, hence all such issues 😅

I have exactly the same problem in mythos ☺

authors: Skyge cleanunicorn eswarasai

repo: b-mueller/sabre


text:
Crashes when hovering over a map

crash-2022-07-24_18.21.52-client.txt

This is probably due to an incompatibility with another mod. I will look into fixing these incompatibilities in the coming weeks

I did not comply with a changed API so my mod does not recognize custom tooltip data. I will change that as soon as I am able to get back to my PC, which will probably be in a week.


text:
New User: React-Bootstrap

Please fill the following out:

Name: React-Bootstrap
Homepage url: http://react-bootstrap.github.io/
Brand Guidelines/Licensing: MIT
Logo:

Thanks!


text:
Improve parsing error message (T6710)

Issue originally made by @markelog

Ref https://github.com/babel/babel/issues/3774
Input code
require("babylon").parse("2++");

Description
Will throw a error -
"SyntaxError: Assigning to rvalue (1:0)"
Which is pretty cryptic, parsers like esprima, or any other JS engine will show more meaningful message  - "Invalid left-hand side expression in postfix operation"

Moving to https://github.com/babel/babylon/issues/119

Moved where?

.. sigh closed this one.

Yey!

authors: hzoo markelog

repo: babel/babylon


text:
add relative information about add-module-exports to support ordinary…

… lib, fixes #362
As asked here https://github.com/babel/generator-babel-boilerplate/issues/362
Tell me if  I should put it somewhere else.

done. Hope you like it.

Thanks @gabrielstuff ! I'll merge this in once CI passes :)

Well, no idea what's up with Node 5. I'm gonna merge this and fix that in a subsequent PR.

authors: gabrielstuff jmeas

repo: babel/generator-babel-boilerplate


text:
Wish animation can be turned off and pop-up can show up in center of window

thx

I agree with removing the animation. There's a lot of redrawing happening while it opens up, and it makes the extension feel slow, since you can't start using it instantly.
Ideally, the animation would be an option disabled by default.

I don't believe the animation can be customized by chrome extensions, as per: https://stackoverflow.com/questions/43867226/is-it-possible-to-customize-my-chrome-extension-popup-windows-opening-animation?rq=1
That being said, I'm just learning the chrome extension APIs now...

I imagine it's unlikely they will add support to the API, closing this in the meantime.

authors: babyman jgthms markwatson spjspjspj

repo: babyman/quick-tabs-chrome-extension


text:
🛑 SM Service is down

In 881698d, SM Service (http://smservice.de) was down:

HTTP code: 0
Response time: 0 ms

Resolved: SM Service is back up in d5f35fe.

repo: bachmannschumacher/upptime


text:
🛑 SM Service is down

In 146853a, SM Service (http://smservice.de) was down:

HTTP code: 0
Response time: 0 ms

Resolved: SM Service is back up in db0959f.


text:
🔌 Plugin: Apply git patch

🔖 Summary
For templates that publish to existing repos with publish:github:pull-request, some files might already exist that we'd like to modify, e.g. adding a workflow to .circleci/config.yml. At the moment, if a directory pulled by fetch:template contains that file, it will fully replace the existing file, which makes it a bit painful to merge.
It seems like it might be relatively low-effort to implement an plugin that git applys provided patch before / during a publish:*.
The biggest question in my head is whether this could be implemented as fully separate from publish:* or if it would need to patch those actions. That would probably have a big impact on the overall level of effort
🌐 Project website (if applicable)
No response
✌ Context
No response
👀 Have you spent some time to check if this plugin request has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
No, but I'm happy to collaborate on a PR with someone else

This seems reasonable, I would suggest that this be a separate scaffolder action however. I wonder if this is possible in the current setup however, we might need to break up the bigger actions into smaller steps so that you can apply the git patch after the git repo has been created but before the push to the remote.

authors: benjdlambert zdufour-asp

repo: backstage/backstage


text:
Status field not work

Hi, I added the status field in the entity, but I got an error below.

I don't know if I can get the function like below. we want to have status column to show if our service is down or up.

Hi!
The status field is not meant to ever be written in a YAML file. It is a generated readonly field, where some errors that can happen inside the processing loops of the catalog are surfaced.
We have been considering a general mechanism for getting errors into entities from "the outside", e.g. via an API or similar but there is no such thing in place yet.


text:
catalog-plugin: Hide "Create Component" button using config

Hey 👋🏻 We're deploying Backstage internally but we're not currently using the scaffolder functionality. Because of this we'd like to remove the "Create Component" button from the catalog plugin home page.
This PR makes it possible to hide the button using config.
Config missing or true:

Config false:

:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)

Tugboat has finished building the preview for this pull request!
Link:

https://pr4677-bbxsvcojki1xtmghlmof1nv5lwkfeyqc.tugboat.qa

Dashboard:

https://dashboard.tugboat.qa/603647f4f285af3e6596e855

👍 We also wanne get rid of it in some cases.
As it's using external route refs, how about supporting something like "optional" external route refs?

Should we generalize this all the way to instead a flag readonly: true?
This does not tie it into a specific implementation (the words "show", "create", "component", and "link" all tell a story about deeper specifics). Also, a flag like this could hide the Create link in the sidebar, and help the demo site be instructive about "hey you can't do that because it's intentionally disabled" etc

Should we generalize this all the way to instead a flag readonly: true?

Thats sounds perfect to me 👌 I can update this PR to do that if you'd like?

I think so. @Fox32 agree?

Oh and one more thing. You need to add this new parameter to the config.d.ts file in the a same package, and mark it @visibility frontend

I think so. @Fox32 agree?

I think that would be good for now

Wanna throw in that the intention is to implement this by making the external component creation route optional, and signal this that way instead, removing the need for config.
Tbh optional external route refs shouldn't be that tricky so could take a look at implementing them if we wanna avoid having to deprecate and remove this in the future?

That makes sense. So the button would only be present if the Create route was registered at all?

Ended up taking a stab at optional external routes, so that we're able to avoid the need for config.
Shipped in #4700, which would supersede this PR

Amazing! Thank you. I'll close this one.

authors: Fox32 Rugvip backstage-service freben robinjmurphy


text:
catalog-model: Remove serializeEntityRef

removes serializeEntityRef which have been deprecated for almost a year, long live stringifyEntityRef.

rebased (didn't fix test yet tho)

authors: freben jhaals


text:
Telegram don’t start

ubuntu@ip-172-31-36-140:~$ sudo TelegramTUI/./telegramTUI������������������������Traceback (most recent call last):�����������������������������������������������  File "/usr/lib/python3.4/configparser.py", line 1116, in _unify_values���������    sectiondict = self._sections[section]����������������������������������������KeyError: 'telegram_api'������������������������������������������������������������������������������������������������������������������������������������������During handling of the above exception, another exception occurred:�����������������������������������������������������������������������������������������������Traceback (most recent call last):�����������������������������������������������  File "TelegramTUI/./telegramTUI", line 2, in ��������������������������    from src.ui import App�������������������������������������������������������  File "/home/ubuntu/TelegramTUI/src/ui.py", line 2, in �����������������    from src.MainForm import MainForm��������������������������������������������  File "/home/ubuntu/TelegramTUI/src/MainForm.py", line 4, in �����������    from src.telegramApi import client�������������������������������������������  File "/home/ubuntu/TelegramTUI/src/telegramApi.py", line 200, in ������    client = TelegramApi()�������������������������������������������������������  File "/home/ubuntu/TelegramTUI/src/telegramApi.py", line 20, in __init__�������    api_id = config.get('telegram_api', 'api_id')��������������������������������  File "/usr/lib/python3.4/configparser.py", line 754, in get��������������������    d = self._unify_values(section, vars)����������������������������������������  File "/usr/lib/python3.4/configparser.py", line 1119, in _unify_values���������    raise NoSectionError(section)������������������������������������������������configparser.NoSectionError: No section: 'telegram_api'��������������������������ubuntu@ip-172-31-36-140:~$ �

Are you install all dependensies?
Please, send me traceback in utf8.

repo: bad-day/TelegramTUI


text:
optional text in cucumber expression does not match

Current behavior
Omitted optional text in cucumber expressions do not match.
Expected behavior
Omitted optional text in cucumber expressions should match.
Test code to reproduce
I assume /this is a cucumber epression/ as indicated in the preprocessor docs and defined in the cucumber docs.
Then(/I am on the start page (again)/, () => {});
does not match
Then I am on the start page

Step implementation missing for "I am on the start page".

(However, it does match Then I am on the start page again.)
Versions

Cypress: 10.3.1
Preprocessor: 12.0.0
Node: 14.17.0

Checklist

[x] I've read the FAQ.
[x] I've read Instructions for logging issues.
[x] I'm not using cypress-cucumber-preprocessor@4.3.1 (package name has changed and it is no longer the most recent version, see #689).

I assume /this is a cucumber epression/

The docs are inaccurate, cucumber expressions are strings.

Additionally, if you were to write a step Then("I am on the start page (again)", () => {});, then I think the last space would be required.

Save your opinion of it for the authors.
https://github.com/cucumber/cucumber-expressions
And the rest of the docs look fine.

Save your opinion of it for the authors.
https://github.com/cucumber/cucumber-expressions

Their specification does not imply if my use case should work or not. Common sense tells me that it should. Sorry, but may I ask if it is their or your code we are talking about? Just to clarify.

And the rest of the docs look fine.

You say “cucumber expressions are strings”
The docs says “A step definition’s expression can either be a regular expression or a cucumber expression. The examples in this section use cucumber expressions.”
My IDE says /^a table step$/ (from that page) is not a string but a regular expression.
You say “the resut of the docs looks fine”

Although I’m new to Typescript, I think one of those four statements must be wrong.

Given, When and Then accepts both strings and regular expressions. Strings automatically are turned into regular expressions, this is what's referred to as "cucumber expressions". This is handled by https://github.com/cucumber/cucumber-expressions, IE. it is not my invention.

So /^a table step$/ is a regular expression but not a cucumber expression, right?

So /^a table step$/ is a regular expression but not a cucumber expression, right?

That's right.

The thing I do not understand is: Do you use some original cucumber code to parse those expressions? Something you cannot touch? Or do you just reject my let’s call it request for improvement?

You can find multiple cucumber libs listed as dependencies in package.json, including @cucumber/cucumber-expressions.

Thanks for your information! One last question, if I may: I am thinking about workarounds. Is it possible for me to transform the string before it goes to cucumber-expressions? I mean without needing to fork this project?

Sure, like this
Given(transform("I am on the start page (again)"), function () {
  ...
});

Thank you! My workaround now looks like this: (“Dann” is German for “Then”.)
interface IStepDefinitionBody<T extends unknown[]> {
    (this: Mocha.Context, ...args: T): void;
}
function toCucumberExpOrRegExp(description: string | RegExp) {
    return typeof description === 'string' ? description.replace(' (', '( ') : description;
}
export const Dann = function (description: string | RegExp, implementation: IStepDefinitionBody<any>) {
    return Then(toCucumberExpOrRegExp(description), implementation);
};

Usage:
Dann('erscheint (wieder) die Startseite', function () {
    // ...
});

I wonder if I am losing anything by having implementation: IStepDefinitionBody<any> as parameter instead of implementation: IStepDefinitionBody<T>. Everything seems to be just fine, though.

No, I don't think so. It's not possible to infer the types of the args of a step-def-body statically. I might have replaced any with unknown and require each dev to explicitly state each expected type, although it's not going to give you any runtime guarantees.

Good point! I replaced implementation: IStepDefinitionBody<any> with implementation: IStepDefinitionBody<unknown[]>.

authors: badeball iomedico-beyer

repo: badeball/cypress-cucumber-preprocessor


text:
Greenkeeper badge

https://greenkeeper.io/
https://github.com/greenkeeperio/greenkeeper/issues/416
The badge could say "dependencies: monitored", similar to the david-dm badge.

I assume you'd want a badge that shows something like "dependencies: up to date" like the david-dm one. Does Greenkeeper provide an API for that?

The badge could say "dependencies: monitored",

If all you want is "dependencies: monitored", you could simply use a static badge:

https://img.shields.io/badge/dependencies-monitored-green.svg

@Daniel15 I don't think there currently is, but there's this: https://github.com/greenkeeperio/greenkeeper/issues/422

I'm marking all the needs-upstream-help issues closed.
If there's new information from the upstream service, please post in the thread. If they're actionable by Shields, a maintainer will reopen the issue.
If anyone wants to follow up with these vendors, feel free to do that! Even though the unresolved issues are closed, they are easy to find:
https://github.com/badges/shields/issues?q=label%3Aneeds-upstream-help+is%3Aclosed

authors: Daniel15 paulmelnikow stevenvachon

repo: badges/shields


text:
fix wercker examples

refs #1994
Still need to think about how to improve the base class, but lets fix this one example as a first step.

Warnings

:warning:

This PR modified service code for wercker but not its test code. That's okay so long as it's refactoring existing code.

Messages

:book:

:sparkles: Thanks for your contribution to Shields, @chris48s!

  Generated by :no_entry_sign: dangerJS