text:
Icon

Type
No response
Info
Here are some icons some are for web page. Feel free to keep or discard.
Search engine: Marginalia
Icon(s)

Hey, could you give a link to the source of this icon? Can't seem to find anything about it.

@Kilthor reminder.

I will look for it

How's

this

Hey, I'm talking about a link to the search engine, as I can't seem to find the company.

I am still trying to find it hahah
I deleted some files and that was one of them

Found it
If you do a google search "marginalia logo png" it comes up
The company URL https://search.marginalia.nu/

authors: Kilthor walkxcode

repo: walkxcode/Dashboard-Icons


text:
bestmodel.pth size shrank while it was being uploaded

wandb --version && python --version && uname

Weights and Biases version: 0.8.25
Python version: 3.7.6
Operating System: Linux

Description
While running a fastai callback to train a model in jupyter notebook by doing something like:
learn_clas.fit_one_cycle(10, slice(2e-3/decay_factor,2e-3), moms=moms, callbacks=WandbCallback(learn_clas))
#... do some other stuff
learn_clas.fit_one_cycle(10, slice(3e-5/decay_factor,3e-5), moms=moms, callbacks=WandbCallback(learn_clas))

In the results that get written out to the notebook I often see errors such as:

Here's the text of the error message.
wandb: ERROR Error uploading "bestmodel.pth": CommError, File /home/fastai/keyword-extraction/notebooks/wandb/run-20200206_091629-h6xl4gbi/bestmodel.pth size shrank from 239376385 to 232153088 while it was being uploaded.
wandb: ERROR Error uploading "bestmodel.pth": CommError, File /home/fastai/keyword-extraction/notebooks/wandb/run-20200206_091629-h6xl4gbi/bestmodel.pth size shrank from 239376385 to 80355328 while it was being uploaded.

It would appear that either there's some kind of race condition - or that the file is being overwritten by the second training loop while still being uploaded by the first.
Update:
It's also happening with lots of other files, eg:
wandb: ERROR Error uploading "___batch_archive_1.tgz": CommError, File /tmp/tmpnn7k_ppewandb/___batch_archive_1.tgz size shrank from 416086 to 301398 while it was being uploaded.
Exception in thread Thread-78:
Traceback (most recent call last):
  File "/home/fastai/anaconda3/envs/keyword-extraction/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/fastai/anaconda3/envs/keyword-extraction/lib/python3.7/site-packages/wandb/file_pusher.py", line 85, in run
    self.cleanup_file()
  File "/home/fastai/anaconda3/envs/keyword-extraction/lib/python3.7/site-packages/wandb/file_pusher.py", line 154, in cleanup_file
    os.unlink(self.tgz_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpnn7k_ppewandb/batch-1.tgz'

Hello! Thank you for letting us know about this error. We will take a look and get back to you as soon as we can.

Can you try to add the callback to the learner instead of at the fit step?

Also if you want to save as a separate run, you can just add a wandb.join() in between to close the previous run.

Thanks @borisdayma - I've been using a single learner and freezing and fine tuning it, so it's not that I'm during different runs, but the same model with multiple fine-tuning passes. But adding the callback at the learner level does seem to help the problem. I haven't seen the error once I've done that.
If I call wandb.join() after each fine tuning pass will that create a brand new run? I'm not sure that's what I want with the multiple fine-tuning passes I think.

Actually I'm still getting the issue, it's just happening on the wandb.join() that I've added at the bottom of my jupyter notebook. It doesn't seem to occur at any other place though. This is the literally the last line in my notebook:

The error message is:
Exception in thread Thread-810:
Traceback (most recent call last):
  File "/home/fastai/anaconda3/envs/keyword-extraction/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/fastai/anaconda3/envs/keyword-extraction/lib/python3.7/site-packages/wandb/file_pusher.py", line 85, in run
    self.cleanup_file()
  File "/home/fastai/anaconda3/envs/keyword-extraction/lib/python3.7/site-packages/wandb/file_pusher.py", line 154, in cleanup_file
    os.unlink(self.tgz_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp6ip8rshjwandb/batch-1.tgz'

If you have it as a single run, you should just add the callback to the learner and you should not need any join at all but you could have one at the end
If you save it as multiple runs, you should add it to the training loop "fit" and you will add a join after each loop.
Let me know if you still have that issue.

@borisdayma
I am having the same issue with two different scenarios:

I use a sweep, it was working just fine. Then suddenly, it stopped syncing and started showing the same error message. I checked the debug.log in my docker container and found:

2020-04-18 19:34:02,918 INFO    MainThread:112 [run_manager.py:_sync_etc():1433] syncing files to cloud storage
2020-04-18 19:41:26,057 ERROR   Thread-21 :112 [retry.py:__call__():108] Retry attempt failed:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 300, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7efd822ce550>: Failed to establish a new connection: [Errno 110] Connection timed out

I don't use a sweep, but I am trying to use Optuna for hyper-parameter optimization and log with wandb. Optuna will use 32 CPU cores, so multiple threads. I saw the same error and the same thing in debug.log.

Honestly, I am not sure if this is an internet issue from my side or something is going wrong?

I see a similar error with images:
wandb: ERROR Error uploading "media/images/train_mosiac-generated6_5721_c2138903.png": CommError, File /tmp/tmpd5dpje7xwandb/1cak8igk-media/images/train_mosiac-generated6_5721_c2138903.png size shrank from 459037 to 377117 while it was being uploaded.

Hey guys, can you try our new client library.  We're about a week away from official release, but the errors you're seeing should be fixed.  You can install with: pip install wandb -U --pre

Hey guys, can you try our new client library. We're about a week away from official release, but the errors you're seeing should be fixed. You can install with: pip install wandb -U --pre

I am still experiencing the same issue in the original post after running the pre release install

@maxzzze Can you confirm which version of fastai you are using?

@borisdayma I dont see fastai installed

Oh sorry I thought it was a fastai related issue.
In that case how do you save your files? Do you have a simple demo?
In particular I'm interested to know if it's when you log images/data visualization or if it's when logging models.

@borisdayma I just now switched from using optuna to ray and I am not experiencing any issues like this one anymore.
Im tunning transformers 3.5.0 now but was experiencing the above issue before using transformers 3.4.0 (optuna again).

source: gharchive/issue

authors: borisdayma eugeneware jramapuram kennazilis ktobah maxzzze vanpelt

repo: wandb/client

license: MIT

license_type: permissive

license_source: github-api


text:
[CLI]: Artifact.download gives OSError: [Errno 22] Invalid argument: './artifacts/Files_txt:v2'

Describe the bug

I am learning how to use Artifacts and wanted to make a simple example.

In a first run I load an artifact with a wandb.Table.
In a second run I want to access that table by downloading the whole artifact but when I use "artifact.download()" the file shows me : <*** OSError: [Errno 22] Invalid argument: './artifacts/Files_txt:v2'> ("Files_txt:v2" is the name of my artifact).
Therefore I decide to specify a root and it works, it downloads the artifact in root, however, I don't know how to access to my table. I have tried the following options:

my_table = artifact.get('Table')

Output
*** OSError: [Errno 22] Invalid argument: './artifacts/Files_txt:v2'.

Using the full file name
my_table = artifact.get('Table.table.json')

Output
*** OSError: [Errno 22] Invalid argument: './artifacts/Files_txt:v2'.

Finally I tried the two previous steps but now using the full path where the artifact was downloaded
my_table = artifact.get('root/Table') # root is my path
my_table = artifact.get('root/Table.table.json') # root is my path

Output: It gives me a NoneType object
I think it may be a bug related to the path used by W&B, any suggestions?

    if FIRST_STEP:

        with wandb.init(project='tests', entity='deividbotina', job_type = 'artifact_test') as run:

            artifact = wandb.Artifact(
                name='Files_txt',
                type="test",
                description='Just some files txt',
                metadata={"meta1":'Testing metadata', "meta2":'Testing metadata 2'}
            )

            my_table = wandb.Table(columns=["a", "b"], data=[["1a", "1b"], ["2a", "2b"]])
            artifact.add(my_table,"Table")

            run.log_artifact(artifact)

    elif SECOND_STEP:

        with wandb.init(project='tests', entity='deividbotina', job_type = 'artifact_test') as run:

            artifact = run.use_artifact('deividbotina/tests/Files_txt:latest', type='test')

            # artifact_dir = artifact.download() # Gives "OSError: [Errno 22] Invalid argument: './artifacts/Files_txt:v2'"
            artifact_dir = artifact.download(root='/media/deivid/files/repos/WNB/Artifacts_1/download')

            # Access to the table
            my_table = artifact.get('Table') # Gives "OSError: [Errno 22] Invalid argument: './artifacts/Files_txt:v2'"
            my_table = artifact.get('Table.table.json') # Gives "OSError: [Errno 22] Invalid argument: './artifacts/Files_txt:v2'"
            my_table = artifact.get(r'/media/deivid/files/repos/WNB/Artifacts_1/download/again/Table') # Gives an NoneType Object
            my_table = artifact.get(r'/media/deivid/files/repos/WNB/Artifacts_1/download/again/Table.table.json') # Gives an NoneType Object

wandb: Tracking run with wandb version 0.13.9
wandb: Run data is saved locally in /media/deivid/files/repos/WNB/Artifacts_1/wandb/run-20230128_182941-ya777m8z
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run glistening-kumquat-8
wandb: ⭐ View project at https://wandb.ai/deividbotina/tests
wandb: 🚀 View run at https://wandb.ai/deividbotina/tests/runs/ya777m8z
*** OSError: [Errno 22] Invalid argument: './artifacts/Files_txt:v2'

Additional Files
No response
Environment
WandB version: 0.13.9
OS: Linux Mint 21 Cinnamon
Python version: 3.10.8
Versions of relevant libraries:
Additional Context
No response

Hi @deividbotina-alv,
Looks like artifact.get is being used incorrectly over here - I would suggest you have a look at our documentation and specifically this example to understand usage.
Thanks,
Ramit

Hi @deividbotina-alv,
Looks like artifact.get is being used incorrectly over here - I would suggest you have a look at our documentation and specifically this example to understand usage.
Thanks, Ramit

Hi @ramit-wandb
Thank you for quick answer, I read the documentation before, maybe I am misunderstanding something?, following your advice I ran the following code (just copy pasting the example but with my project and entity information):
    with wandb.init(project='tests', entity='deividbotina', job_type = 'artifact_test') as r:
        artifact = wandb.Artifact('my_dataset', type='dataset')
        table = wandb.Table(columns=["a", "b", "c"], data=[[i, i*2, 2**i]])
        artifact.add(table, "my_table")
        wandb.log_artifact(artifact)

    # Run using the artifact
    with wandb.init(project='tests', entity='deividbotina', job_type = 'artifact_test') as r:
        artifact = r.use_artifact('my_dataset:latest')
        table = r.get('my_table')

And I got the following error
AttributeError: 'Run' object has no attribute 'get'

I believe there is an error in the example, to be specific, in the last line, so I changed it to table = artifact.get('my_table') and I am getting the same error as before:
OSError: [Errno 22] Invalid argument: './artifacts/my_dataset:v0'

Do you know what I could be doing wrong?
Bests,
Deivid

Hi @deividbotina-alv,
I just tested this on my end, and while the example does have a minor error - it should be artifact.get('my_table'), that seems to work for me on MacOS and Ubuntu.
Let's do the following to try to debug this:

Could you share the debug-internal.log file of one of the runs which crashed with OSError 22?
If possible, could you uninstall and reinstall the wandb SDK?

Based on the error message, this is an error when calling open() internally to download this artifact.
There is a good chance this issue might also be coming from the Operating System level in which case this could be an error with the permissions you have on the filesystem, something worth checking and it might be worth trying on another machine if you run into the same issue.
Ramit

Hi @deividbotina-alv, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

artifact.get('my_table')

Hello @ramit-wandb, sorry I couldn't reply faster. Here the debug-internal.log file.
debug-internal.log
I am thinking to do the same experiment in google colab also, just to check if it is related with my operative system.
Thanks.

Hey @deividbotina-alv,
Are you sure this is the correct debug log? From what I see, this seems to be the logs for the process that creates and uploads an artifact, not the one that ingests it. The log would be of in your working directory in the format wandb/run-<DATETIME>-<RUNID>/logs/debug-internal.log where <RUNID> should correspond to the Run ID of the run ingesting the artifacts.
Thanks,
Ramit

Hi @ramit-wandb
Thanks again for quick response. Sorry I had a lot of folders and I took the wrong one. Just in case, I deleted all my runs and ran again the code. Here the two files:
debug-internal_1.log
debug-internal_2.log
Bests,
Deivid.

Hi @deividbotina-alv,
I just looked into the 2 files, and they both seem healthy. You should definitely run the test above in a colab environment, I don't think this error is occuring from the wandb SDK's end.

Hi @deividbotina-alv,
I just looked into the 2 files, and they both seem healthy. You should definitely run the test above in a colab environment, I don't think this error is occuring from the wandb SDK's end.
If you could send the whole stack trace, that would be helpful as well.

Hi @ramit-wandb
Thanks for you answer. I did run the same experiment in a colab environment and it worked just fine, the artifact was saved in /content/artifacts/my_dataset:v0. The problem is only happening in my computer with linux mint, I am going to try in a new conda environment and I'll let you know.
Thanks,
Deivid.

Hey Deivid,
Thanks for confirming this. It might be worthwhile to do containerize your program with Docker and run this on another machine. Maybe this will let you run the program on your current machine too but I'm not sure about that.

Hey Deivid,
Thanks for confirming this. It might be worthwhile to do containerize your program with Docker and run this on another machine. Maybe this will let you run the program on your current machine too but I'm not sure about that.

Hi @ramit-wandb, thank you for the suggestion, I am not really sure how to do that but I will do some research.
Thank you,
Deivid.

Thanks Deivid! Just to help you a little with your research into dockerizing your application, you might want to get started with Dockers Getting started Guide on building containerized applications
I'm assuming you are using NVIDIA GPUs to train your Machine Learning models, so you would probably want to use NVIDIA's CUDA Base Image to build your image. Additionally, to let your container access the GPU resources within the container, you would want to look into NVIDIA's Container Toolkit to install the correct runtime.
All the best and happy training!
Ramit

authors: deividbotina-alv ramit-wandb

repo: wandb/wandb


text:
[CLI]: Can't resume run with correct id

Describe the bug

I'm trying to resume a run that finished successfully. I got the ID from the W&B Web UI and have my training mechanism set up to resume from a checkpoint (so it will spit out correct step numbers, etc). What I want is to log more data to that same run as though it had never stopped.
So I do this:

run = wandb.init(id="g9jgwkjp", resume="must")

But then I get this:

wandb: ERROR resume='must' but run (g9jgwkjp) doesn't exist
Traceback (most recent call last):
  File "/home/davidg/.pycharm_helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 204, in <module>
  File "/home/davidg/.virtualenvs/learning/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1144, in init
    run = wi.init()
  File "/home/davidg/.virtualenvs/learning/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 773, in init
    raise error
wandb.errors.UsageError: resume='must' but run (g9jgwkjp) doesn't exist

Additional Files
This ID does exist, I can confirm by looking in the web UI:

Environment
WandB version: 0.14.0
OS: Windows 11
Python version: 3.10
Additional Context
Perhaps I misunderstand what resume is supposed to do, the docs are a bit unclear. It seems like the assumption is that I'd only ever like to resume a crashed run. But what if I finished a run for a few epochs, and now I want to run it for a few epochs more?
I also tried resume="allow" and that doesn't seem to do what I expect.

In case my intent isn't clear, I've managed to continue training and just log a second run to W&B with the same name, and set a very similar colour so that it looks like one run on the charts:

What I'm trying to achieve is all the same logged values, but in one run instead of two. Is that possible?

Hi @davidgilbertson thanks for reporting this issue. Could you please also provide the entity and project too? It will attempt to use the default values, and it won't find this specific run. Would the following work for you?
run = wandb.init(entity="gilbertson-david", project="rich-vocab-2d", id="g9jgwkjp", resume="must")

Thanks Thanos, I'll try that next time. May I suggest updating the docs to reflect this, currently they state:

when you resume (if you want to be sure that it is resuming, you do wandb.init(id=run_id, resume="must").

Hey @thanos-wandb, I just ran into the very same issue and wasted some time figuring this out myself before I found your conversation. May I suggest to give the docs update request a bumb? :) It's such an easy fix but not yet in the docs as far as I can tell?

Hi @JohannesTheo sorry to hear about the trouble this issue caused you. I've just checked the internal ticket I created for the Docs update request, and it appears that the pull request (PR) has not been merged yet; it's currently under review. The Docs team worked with our SDK team to rewrite this part and they will overhaul the entire 'resuming run' section.

All good :) Great to hear it's going to be in the docs soon.

I also feel the way runs, sweeps, projects, and switching back and forth between IDs and names gets confusing rapidly. The web interface makes it quite difficult to find a run/sweep's ID (I have to copy it from the URL often), and uses names instead; yet the code examples use the ID. Does a run belong to a sweep? Is it just loosely associated with one? If I'm logged in and have access to the API (Which is nice without having to add code to do this explicitly), then needing to specify the username half of the time seems to defeat the reason for being logged in already.
If the docs use the run ID, and I am opening it via a unique ID, I would expect that to be sufficient to get a handle on it. If it expects a path-like name, that needs separators, then this would be as expected. The examples seem to flip between the two - if I open it from the API, I'm expected to give a string separated by slashes. I think it would make more sense to switch to using UIDs for all objects (sweep, run, etc.) that the server-side can look up using only that ID, and then otherwise refer to these objects by their name/path, which at times may or may not be easier for end users.

Hi @stellarpower thank you for your feedback, much appreciated. Would it be please possible to create a new issue, so that the team can log it as a feature request and track it separately? The current thread is related to a run resuming issue which we will highlight in our Docs soon.

authors: JohannesTheo davidgilbertson stellarpower thanos-wandb


text:
chore(weave): Extend simple tab border line

Extends the border bottom line of the tabs to meet the edges
Before

After

Before (bottom), After (Top)

Closing, design does not approve

source: gharchive/pull-request

repo: wandb/weave


text:
Resolve type checker bugs

The type checker right now is not bug-free. The main bug is that it crashes on recursion.
Some visual bugs:

some functions like scanl visually have a signature (a -> a -> a) -> a -> [a] -> [a] when there's no wire connected (to fill in a type variable) and they are uncurried, though their actual signature is (b->a->b)->b->[a]->[b]
some functions show a number as type variable instead of a letter (this number also increases with every change, so look into that)
when currying/uncurrying a function for which the output is already attached to another node, this node may become invalid (shown with a red line and a warning sign), but it will not become visually valid again after moving the knot back.
connecting an output with a second wire to a tree only visually updates 1 occurence of that type. For example, connect a True and an (==) to an (||) as inputs, then connect the True to the (==) as input –the type will read Bool only at the argument where the True is connected.

While recursion is not supported now,  crashing should never happen.
Bug 1: is caused by the type variable implementation, and is fixed in my improved_types branch.
Bug 2: the number thing is solved by pull request #169, however the fast increasing typevar IDs shows that the UI is invoking the typechecker way too often.
Bug 3: looks like a true UI bug.
Bug 4: seems to be solved in the improved_types branch, but as soon as wires are disconnected it goes visually wrong again because not everything updates right.
Reproducing any of these bugs without using the UI in the unit tests would be appreciated.

Any bugs left with the now fully rewritten typechecker, except for error connections as mentioned in #183?
Did you find anything else with wrong types, @Flumble ?

All the bugs are fixed except for a case resulting from scenario 4:

connect a True and an (==) to an (||) as inputs, then connect the True to the (==) as input

Viskell currently crashes (probably endless recursion, judging from the stack overflow) in an attempt to check this program in which a connection is made higher up in an expression tree (after which it is no longer a tree but a DAG). Note that this doesn't hold in general: for example, (type) manipulating a list and then zipping it with the original list works fine.

Fixed in #221.

authors: Flumble comonoidial

repo: wandernauta/viskell

license: mit

license_source: bigquery


text:
start working on the markdown pages routing

Continue to focus work on the markdown routing system.
In order to activate many internal sealed within the asp.net core mvc, I need to port them over here.

I would close the book for today. but there is still a lot of work need to be done.

repo: wangkanai/wangkanai


text:
iPhone X press_coefficient是多少哇？

哪位大佬知道iPhone X的跳跃系数，我这边不准确呀

自己调一调啦，根据截图判断一下，短了就调大，长了就调小


text:
animation not worked properly when i put recyclerview inside nested scrollview

animation not worked properly when i put recyclerview inside nested scrollview
i think in nested scrollview all the items bind immediatley how to solve it

any solutions?

repo: wasabeef/recyclerview-animators


text:
make process names of capability providers friendlier for sysadmins

Capability providers run as an executable whose name is derived from the machine architecture and os, such as "x86_64-linux", which is the name given when the par file is created. One problem with this is that all provider processes would appear to have the same name in tools such as ps, top, netdata, default grafana dashboards, etc.
If one process is problematic, such as using too much memory or cpu, a sysadmin needs to be able to identify which one.
If the sysadmin were to view the full path to the executable, (which can be done with different ps flags) they can get the provider's public key, which can be looked up to figure out the provider name, but it would be much more intuitive to have top display "Httpserver". The link name should be included in the display name, to help the sysadmin distinguish between instances of the same provider.
Some ideas for resolving:

the host could save the exe as a different file name, composed of a server name part plus a link name ("Httpserver_default"). The name part could come from the --name parameter that was passed to the wash par create command.
Alternately, the provider can change its own process name on startup, using a system call. This is a stable and common operation, but it's specific to linux. The process name is limited to 15 visible characters (16 with null terminator), so it might be truncated, as in httpserver_defa.

Because of the character limitation, if the file rename solution is used, the host should only use the first 10 characters of the --name parameter, to make sure there is room to display a portion of the link name.
@autodidaddict thoughts?  Adding the change to provider_main would be straightforward, but it would be limited to linux, and only automatic if people use our libraries to implement their provider.

Why not use the instance ID? This is information passed to the provider and managed by the host. That would let you correlate metrics and events with the process running on the machine.

For metrics and events, we should definitely be logging instance id.
If the installation has a metrics server, an operator should be able to figure out all the data they need. If we log process id with the instance id, then the operator can also go back to correlate it with what's in top.
If you meant that the process name could be changed to the instance id also (or at least its first 15 chars) that would be useful to an operator if wash or the web ui has an easy way to lookup an instance-id.
I might want to dedicate one or two prefix characters of the process name, like 'V' or 'WC' so that it doesn't look completely random, and it would be easy to filter on in tools like top or grafana.

How about naming the executable something like cap-[instance id] ?

sounds good to me

Implementing this and noticed that there's an unintended consequence - by naming the extracted bytes the same as the instance ID, we have two issues:

the bytes from the PAR will never be cached, we will always extract from the PAR now because no filename duplication happens
because no filename duplication happens, we now leave cap-{guid} files lying all over the /tmp/wasmcloudcache/Vxxx directories, which means we now have to clean that up ourselves.

Come to think of it, I'm not even convinced that we need to expose the executable filename in the all_providers() function. It can be something left for developers (and possibly washboard) to use when examining a single ProviderModule pid.

to have top display "Httpserver".

In a production environment, no one is going to be inside a box (container) running top on that machine. Some central authority will alert that a process on a managed machine/container is running away and consuming too many resources, and that authority may even automatically kill/restart that process.
The more I look at the consequences of using instance IDs in the executable filenames the more I dislike the price to be paid for it.
We could use a filename like wasmcloud_httpserver_default which is just a normalization of the contract ID and link name combo.

It sounds to me like storing the executable path would help tools that support interrogating the runtime (anything with control interface access) but tools like top will still report the x86-linux name as the executable. I'm good with adding the executable path for now and then modifying the naming scheme in the future if the need arises (as it won't be a breaking changes).
edit because of your most recent comment: I like wasmcloud_httpserver_default. That wouldn't result in any caching changes as we have both of those pieces of information when we start the provider, so we could lookup the filename right?

Right, renaming to wasmcloud_httpserver_default wouldn't break the existing caching mechanism, because it's still unique to that particular provider (this executable is stored in /tmp/wasmcloudcache/Vxxx/{revision}

I considered suggesting cutting off the wasmcloud_ prefix, but that would mean that the executable implementing evilcorp:httpserver would have the same executable name and could cause unnecessary confusion

authors: autodidaddict brooksmtownsend stevelr

repo: wasmCloud/wasmCloud


text:
wip: fix(host): propagate traces through components

Feature or Problem
This PR adds a trace_ctx parameter to a component's Handler which is set as invocations are received and used as invocations are sent in order to propagate the current trace context throughout a component's lifecycle.
This needs more testing, and is only really set at the moment for outgoing-http, but I believe this is the right methodology in general. I may end up storing a different data type or using the injector differently but this did work to connect a trace.
Related Issues

Release Information

Consumer Impact

Testing

Unit Test(s)

Acceptance or Integration

Manual Verification

Traces reconnected (on trace logging so VERY verbose) but not quite at the right level. Fine tuning instrumentation...

After merging in Victor's PR, I get this:

So I'll make sure to cut a new version of the http-client/http-server with the updated instrumentation verbosity


text:
RSS-Feed

Hallo,
ich faende es toll, wenn ich die Timeline auch per RSS-Feed abonnieren koennte, so dass ich garnichts mehr verpasse.
Noch besser faende ich fuer mich einen E-Mail Newsletter, jedoch waere dass vielleicht etwas zu viel.
Vielen Danke fuer dein Projekt.

Hey @fossdd,
danke für den Vorschlag! Ich habe beides zur Roadmap hinzugefügt.
Einen RSS-Feed werde ich zeitnah umsetzen und anbieten - unterscheidet sich ja vom Prinzip her nicht wirklich stark von dem JSON, welches wir ohnehin schon anbieten.
Bezüglich E-Mail-Newsletter: Wenn es die Zeit zulässt. Für mich vorstellbar wäre hier eine wöchentliche E-Mail, sofern es Neuigkeiten gibt; ich halte das für eine gute Idee, an die ich gar nicht gedacht hatte.
Danke für Deinen Vorschlag / Deine Ideen!

authors: fossdd wasserbombe

repo: wasserbombe/lucafail-timeline


text:
Auto renew - add r.raise_for_status() after r = requests...

I have added the following after every instance of r = requests.xxx():
if r.status_code != requests.codes.ok:
    r.raise_for_status()

I feel this is the bare minimum: it might be better to include r.text in the raise_for_status.

Sorry, just realised I've got two commits here.  Can you get rid of the 'More logging is needed' commit at your end?  Or do I have to rebase or something?

If you look at the 'extraneous' self.locations = [], you can see that it inappropriately wipes the array the first time the access token expires, and headers() calls _basic_login().  In my case, I was unable to call client.locations[0].status() and got an IndexError (as len(client.locations) == 0) after 30 mins.
Similarly, I think anything like:
self.access_token
self.access_token_expires
self.account_info
self.locations

which are set after the results of a r = requests.xxx should be None (or similar) immediately before the request is made so that if the request does fail, they do not contain stale data.  This is especially true for access_token and access_token_expires, but less so for account_info and locations (you would expect them to be more stable).
Nonetheless, Honeywell's api must, unfortunately, be polled, and I take the view that every time the access token times out is a good time to do a refresh (e.g. residential address, zone names, and/or schedules may have changed in the last 30-60 minutes).  On that basis, I think this is the right approach.

And here is an example of it working:
2018-06-23 09:11:01 INFO (SyncWorker_7) [custom_components.evohome] Calling client v2 API [1 request(s)]: client.locations[0].status()...
2018-06-23 09:11:02 ERROR (MainThread) [homeassistant.helpers.entity] Update for climate._my_home fails
Traceback (most recent call last):
  File "/srv/hass/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 197, in async_update_ha_state
    yield from self.async_device_update()
  File "/srv/hass/lib/python3.6/site-packages/homeassistant/helpers/entity.py", line 320, in async_device_update
    yield from self.hass.async_add_job(self.update)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/.homeassistant/custom_components/evohome.py", line 669, in update
    _updateStateData(self.client, self.hass.data[DATA_EVOHOME])
  File "/root/.homeassistant/custom_components/evohome.py", line 202, in _updateStateData
    = _returnTempsAndModes(evo_client)
  File "/root/.homeassistant/custom_components/evohome.py", line 255, in _returnTempsAndModes
    ec2_status = client.locations[0].status()  # get latest modes/temps
  File "/srv/hass/lib/python3.6/site-packages/evohomeclient2/location.py", line 25, in status
    r.raise_for_status()
  File "/srv/hass/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://tccna.honeywell.com/WebAPI/emea/api/v1/location/2738909/status?includeTemperatureControlSystems=True

repo: watchforstock/evohome-client


text:
希望添加查看有中精及以上作品曾经选票详情的功能

对于中精及以上等级的作品，野营不支持查看过去的选票详情（如初精选票详情） 不过可以通过请求API的方式获取，希望该插件能通过请求API获取信息并显示到页面上，让用户方便、清晰地查看。
API地址为https://aerfaying.com/WebApi/Projects/<ProjectId>/GetEvaluates（若没有请求荷载，可以获取各级精华选票；需要有效cookie）

感谢建议，已添加


text:
Migrate C SPTI to C#

Feature Branch
Current feature branch for this issue: not created yet.
Progress

[ ] Create a task list for this issue.

Background
This issue was originally at watfordjc/LTO-Encryption-SPTI#5
LTO-Encryption-SPTI is currently a linear Win32 C application.
LTO-Encryption-Manager has now reached the point where it needs to use SPTI.
The goal of this issue is to implement the most basic functions as library functions, starting with opening a file descriptor for the tape drive, using a Win32 API to obtain the serial number and WWN/LUN, and then closing the file descriptor.
The Data Backups and Archiving Project contains this loosely related note:

Improve design of LTO-Encryption-SPTI
At the moment there is a lot of repetitive code, redundant code leftover from the template, and the code isn't that clean.
There is also a lot of debugging code and output and very little data being retained after it is queried.
The data in the SCSI VPDs, for example, are mostly burned into the drive's persistent non-volatile memory at the time of manufacture or during a firmware update.
Translate the SCSI pages into structured variables that can be referenced rather than doing the byte and bit stuff inline.

Data Redesign
The LTO-Encryption-SPTI application at the moment is mostly a load of constants and structs and manual parsing. Part of the reason for that is that the data is received unformatted.
Having decided to P/Invoke CreateFile() directly in LTO-Encryption-Manager rather than creating a wrapper function in a C library, I am now considering the feasibility of doing things mostly in C# rather than using a C library. The SPTI sample is C-based, and LTO-Encryption-SPTI is written in C, but almost everything is converting to structs and parsing. Sometimes the parsing is on fields of a structure that can't be converted to a struct.
In order to enable encryption, the following are needed via P/Invoke and SPTI at a minimum:

[x] A HANDLE to the drive.
[ ] The WWN/LUN for the drive (Device Identifiers page).
[ ] Whether or not the drive's encryption is supported (Supported Security Protocol, Data Encryption Management Capabilities, Data Encryption Capabilities, and Supported Key Formats pages).
[ ] Current encryption status (Data Encryption Status page).
[ ] Drive's public key for key wrapping (Device Server Key Wrapping Public Key page).
[ ] Tape barcode, whether the tape is formatted LTFS, whether the tape is encrypted (MAM attributes).
[ ] First block encryption status KAD (Next Block Encryption Status KAD).

The first item, a HANDLE to the drive, is obtained by getting the drive path via CIM or an alternative method, and then (if necessary) appending a hash/octothorpe and the GUID for a tape drive (GUID_DEVINTERFACE_TAPE): #{53F5630B-B6BF-11D0-94F2-00A0C91EFB8B}, and using that full path with CreateFile().
Step one of obtaining the WWN/LUN (or anything else) requires calling DeviceIoControl() 4 times to check we can communicate with the drive, although the only thing I'm checking is that SrbType is of the newer/extended SCSI block variety STORAGE_REQUEST_BLOCK:

Obtain a STORAGE_DESCRIPTOR_HEADER (parsed for the size of a STORAGE_ADAPTER_DESCRIPTOR).
Obtain a STORAGE_ADAPTER_DESCRIPTOR (parsed for the AlignmentMask and SrbType values).
Obtain a STORAGE_DESCRIPTOR_HEADER (parsed for the size of a STORAGE_DEVICE_DESCRIPTOR).
Obtain a STORAGE_DEVICE_DESCRIPTOR (parsed for the BusType and SerialNumber).

Next, the process of sending/getting data to/from the drive is similar for most pages. SPTI wraps the SRB and sends it to the drive. Let's look at the C code:
if (srbType == SRB_TYPE_STORAGE_REQUEST_BLOCK)
{
	int pageCode;

	/*
	* CDB: Inquiry, Device Identifiers VPD page
	*/
	length = ResetSrbIn(psptwb_ex, SCSIOP_INQUIRY);
	if (length == 0) { goto Cleanup; }
	psptwb_ex->spt.Cdb[1] = CDB_INQUIRY_EVPD;
	psptwb_ex->spt.Cdb[2] = VPD_DEVICE_IDENTIFIERS;
	status = SendSrb(fileHandle, psptwb_ex, length, &returned);

	if (CheckStatus(fileHandle, psptwb_ex, status, returned, length))
	{
		pageCode = psptwb_ex->ucDataBuf[1];
		if (pageCode == VPD_DEVICE_IDENTIFIERS)
		{
			ParseDeviceIdentifiers((PVPD_IDENTIFICATION_PAGE)psptwb_ex->ucDataBuf, &logicalUnitIdentifierLength, &logicalUnitIdentifier);
		}
	}
...
}

I'm going to need one of those psptwb_ex variables. That would be a PSCSI_PASS_THROUGH_WITH_BUFFERS_EX. I'm going to need to convert some C structs to C#. After spending way to long trying to marshal between C# and C and back, and ending up with a 9 byte offset issue, I have decided I am going to do it all in C# with CsWin32 and (where not covered) Win32 P/Invoke.
Let's compare what I've currently got in terms of C# code for the Device Identifiers VPD page:
public static void GetTapeDriveIdentifiers(TapeDrive tapeDrive)
{
    IntPtr psptwb_ex = Marshal.AllocHGlobal(Marshal.SizeOf<SCSI_PASS_THROUGH_WITH_BUFFERS_EX>());
    SCSI_PASS_THROUGH_WITH_BUFFERS_EX sptwb_ex = new();
    Marshal.StructureToPtr(sptwb_ex, psptwb_ex, true);
    uint length = ResetSrbIn(psptwb_ex, Constants.SCSIOP_INQUIRY);
    sptwb_ex = Marshal.PtrToStructure<SCSI_PASS_THROUGH_WITH_BUFFERS_EX>(psptwb_ex);
    Windows.Win32.InlineArrayIndexerExtensions.ItemRef(ref sptwb_ex.spt.Cdb, 1) = Constants.CDB_INQUIRY_EVPD;
    Windows.Win32.InlineArrayIndexerExtensions.ItemRef(ref sptwb_ex.spt.Cdb, 2) = Constants.VPD_DEVICE_IDENTIFIERS;
    Marshal.StructureToPtr(sptwb_ex, psptwb_ex, true);

    uint returnedData = 0;
    Windows.Win32.System.IO.OVERLAPPED overlapped;
    bool ok;
    unsafe
    {
        ok = Windows.Win32.PInvoke.DeviceIoControl(tapeDrive.Handle,
            Windows.Win32.PInvoke.IOCTL_SCSI_PASS_THROUGH_EX,
            (void*)psptwb_ex,
            (uint)Marshal.SizeOf(sptwb_ex),
            (void*)psptwb_ex,
            length,
            &returnedData,
            &overlapped);
    }
    if (ok)
    {
        sptwb_ex = Marshal.PtrToStructure<SCSI_PASS_THROUGH_WITH_BUFFERS_EX>(psptwb_ex);
        Trace.WriteLine(Convert.ToHexString(sptwb_ex.ucDataBuf, 0, (int)sptwb_ex.spt.DataInTransferLength));
    }
    Marshal.FreeHGlobal(psptwb_ex);
}

ResetSrbIn() pretty much does what its name suggests: it resets an SRB for usage in the in/read direction. Having found the InlineArrayIndexerExtensions method (it doesn't even have any Google results) after hunting through IntelliSense for a way to access the 2nd byte of a 1 byte variable sized array (UCHAR[ANYSIZE_ARRAY] in Win32 C), I migrated ResetSrbIn() to C#.
There is one issue with migrating to C#: many constants and structs, such as the VPD page struct, do not appear to be available via CsWin32. C# also doesn't really do bit fields in structs. Piping the Trace.WriteLine(Convert.ToHexString(...)) output through xxd and hexdump, though, I can see the LUN (and HP_____Ultrium 6-SCSI__[SerialNumber], where _ is a space) in the bytes.
I need to parse the pages though, and with CsWin32 appearing to not have the page structs, I am going to have to implement all of them myself, including working out how to handle bit fields. This is one of the problems with manual parsing: having to reimplement the same thing again.
I am considering defining the pages using ASN.1+ECN, although there might not be a library or tool that can convert those definitions to C#.
The Compaq/HP OID 1.3.6.1.4.1.232.5.5.4.1 (iso.identified-organization.dod.internet.private.enterprise.compaq.cpqScsi.cpqSasComponent.cpqSasTapeDrv.cpqSasTapeDrvTable.cpqSasTapeDrvEntry) is a table entry for a SAS tape drive used with tape libraries that do SNMP and it has some MIB objects defined such as 1.3.6.1.4.1.232.5.5.4.1.1.8 (cpqSasTapeDrvSerialNumber):
cpqSasTapeDrvSerialNumber OBJECT-TYPE
	SYNTAX  DisplayString (SIZE (0..40))
	ACCESS  read-only
	STATUS  mandatory
	DESCRIPTION
		"SAS Tape Drive Serial Number.

		This is the serial number assigned to the tape drive."
::= { cpqSasTapeDrvEntry 8 }

Such objects/modules may have a use if SNMP support were desired, however getting the pages from the drive and parsing them so that such an object can be populated is something for a tape library to implement. As far as I can see, those tables in the LTO manuals are the only definitions the LTO publish.
Let's take the VPD struct from scsi.h (with comments removed), and a non-ECN translation in bullet list form:
typedef struct _VPD_IDENTIFICATION_PAGE {
    UCHAR DeviceType : 5;
    UCHAR DeviceTypeQualifier : 3;
    UCHAR PageCode;
    UCHAR Reserved;
    UCHAR PageLength;
    UCHAR Descriptors[0];
#endif
} VPD_IDENTIFICATION_PAGE, *PVPD_IDENTIFICATION_PAGE;

The first byte contains two options: a 5 bit peripheral device type in the lower bits, and a 3 bit peripheral qualifier in the upper bits. The values of these fields are from an enum/list (e.g. DeviceType = 0x05 means CD Device).
The page code, defining what the rest of the data is. For VPD_IDENTIFICATION_PAGE that would be 0x83.
A reserved/padding byte.
The page length (total length minus this byte and the 3 bytes above).
The page (variable length sequence of Descriptors).

Using the ASN.1 example RLC-RADIO-HANDOVER-COMPLETE (from ETSI TS 101 761-2 V1.3.1, AKA the Hiperlan specification) as a template (the specification has ASN.1 throughout and transfer syntax tables in Annex A), the ASN.1 for VPD-Identification-Page arguments would be something like this:
Device-Identification-Page-Arg ::= SEQUENCE {
	device-type 				Device-Type
	device-type-qualifier			Device-Type-Qualifier
	page-code				Page-Code
	page-length				Page-Length
	device-identification-descriptor-list	Device-Identification-Descriptor-List
}

Adding in the reserved/padding, the initial encoding structure would be something like this:
#Device-Identification-Page-Arg-struct ::= #CONCATENATION {
	device-type 				#Device-Type
	device-type-qualifier			#Device-Type-Qualifier
	page-code				#Page-Code
	aux-reserved-1				#PAD
	page-length				#Page-Length
	device-identification-descriptor-list	#Device-Identification-Descriptor-List
}

[TODO...]

I can't seem to get SPIN working. Keep getting Medium not present errors.

Ubuntu
This is a side-note on installing tape and LTFS software in Ubuntu, because testing transfer rates when reading/writing from/to LTFS volumes in Windows revealed abysmal speeds. Using two operating systems isn't ideal, however enabling LTO encryption on a drive can leave that key installed on the drive through multiple reboots (i.e. not setting the CKOD bit to 1).
This is a fresh (non-live desktop) install of Ubuntu 20.04.4 on a 64 GB flash drive, installed by clicking on install from within a live session of Ubuntu 20.04.4 on a different USB flash drive, installed using rufus in Windows 10.
This is an 8 core/16 thread machine, so I use 12 threads -j 12 when I call make during installation of ltfs.
A note on formatting in this comment, given markdown's limitations:

typed command at a $ bash prompt in gnome-terminal

output from the command

Existing Software Information
Kernal and Operating System Versions

uname -a

Linux john-desktop 5.13.0-39-generic #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

Installed Packages/Software

comm -23 <(apt-mark showmanual | sort -u) <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort -u)

git
google-chrome-stable
gparted
grub2-common
keepass2
nvidia-driver-510
uuid
veracrypt

Installation
Cloning the Repository

cd /opt
sudo mkdir ltfs
sudo chown john:john ltfs
git clone 'https://github.com/LinearTapeFileSystem/ltfs.git'
cd ltfs
git log -n1 --oneline

98e8854 (HEAD -> master, origin/master, origin/HEAD) Fix filehandle corruption on LP64 (#345)

Installing Dependencies
sudo apt install automake icu-devtools libfuse-dev libsnmp-dev libtool libxml2-dev mt-st python3-pyxattr uuid-dev
There is one more thing needed: /usr/bin/icu-config

stat /usr/bin/icu-config
stat: cannot stat '/usr/bin/icu-config': No such file or directory

Building and Installing

sudo cp .github/workflows/icu-config /usr/bin/icu-config
sudo chmod +x /usr/bin/icu-config
./autogen.sh
./configure
make -j12
sudo ldconfig
sudo make install
sudo ldconfig

Testing Drive Detection

sudo mkdir /mnt/ltfs
sudo chown john:john /mnt/ltfs
sudo ltfs -o device_list

98df LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
98df LTFS14058I LTFS Format Specification version 2.4.0.
98df LTFS14104I Launched by "ltfs -o device_list".
98df LTFS14105I This binary is built for Linux (x86_64).
98df LTFS14106I GCC version is 9.4.0.
98df LTFS17087I Kernel version: Linux version 5.13.0-39-generic (buildd@lcy02-amd64-080) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 i386.
98df LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
98df LTFS17089I Distribution: NAME="Ubuntu".
98df LTFS17085I Plugin: Loading "sg" tape backend.
Tape Device list:.
Device Name = /dev/sg0 (11.0.0.0), Vendor ID = HP      , Product ID = Ultrium 6-SCSI  , Serial Number = [REDACTED], Product Name =[Ultrium 6-SCSI].

sudo ltfs /mnt/ltfs/ -o devname=/dev/sg0 -o sync_type=unmount

98e7 LTFS14000I LTFS starting, LTFS version 2.5.0.0 (Prelim), log level 2.
98e7 LTFS14058I LTFS Format Specification version 2.4.0.
98e7 LTFS14104I Launched by "ltfs /mnt/ltfs/ -o devname=/dev/sg0 -o sync_type=unmount".
98e7 LTFS14105I This binary is built for Linux (x86_64).
98e7 LTFS14106I GCC version is 9.4.0.
98e7 LTFS17087I Kernel version: Linux version 5.13.0-39-generic (buildd@lcy02-amd64-080) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #44~20.04.1-Ubuntu SMP Thu Mar 24 16:43:35 UTC 2022 i386.
98e7 LTFS17089I Distribution: DISTRIB_ID=Ubuntu.
98e7 LTFS17089I Distribution: NAME="Ubuntu".
98e7 LTFS14064I Sync type is "unmount".
98e7 LTFS17085I Plugin: Loading "sg" tape backend.
98e7 LTFS17085I Plugin: Loading "unified" iosched backend.
98e7 LTFS14095I Set the tape device write-anywhere mode to avoid cartridge ejection.
98e7 LTFS30209I Opening a device through sg-ibmtape driver (/dev/sg0).
98e7 LTFS30250I Opened the SCSI tape device 11.0.0.0 (/dev/sg0).
98e7 LTFS30207I Vendor ID is HP      .
98e7 LTFS30208I Product ID is Ultrium 6-SCSI  .
98e7 LTFS30214I Firmware revision is 35GD.
98e7 LTFS30215I Drive serial is [REDACTED].
98e7 LTFS30285I The reserved buffer size of /dev/sg0 is 1048576.
98e7 LTFS30294I Setting up timeout values from RSOC.
98e7 LTFS17160I Maximum device block size is 1048576.
98e7 LTFS11330I Loading cartridge.
98e7 LTFS11332I Load successful.
98e7 LTFS17157I Changing the drive setting to write-anywhere mode.
98e7 LTFS11005I Mounting the volume from device.
98e7 LTFS17192W Cannot read: medium is encrypted.
98e7 LTFS12049E Cannot read: backend call failed (-21600).
98e7 LTFS11174E Cannot read ANSI label: read failed (-21600).
98e7 LTFS11170E Failed to read label (-21600) from partition 0.
98e7 LTFS11009E Cannot read volume: failed to read partition labels.
98e7 LTFS14013E Cannot mount the volume from device.
98e7 LTFS30205I MODESELECT (0x55) returns -20500.
98e7 LTFS30263I MODESELECT returns Invalid Field in Parameter List (-20500) /dev/sg0.

Reading MAM/CM Attributes
[... incomplete comment - anticipated power loss ...]

Commit 945cc8c5 adds SPTI tape functions to LTO-Encryption-Manager.

repo: watfordjc/LTO-Encryption-Manager


text:
Updated the flash function

Now the caller can control the color, number of flashes, and duration between flashes

https://github.com/watir/watir#note-on-patchespull-requests
Suggests I should write tests - happy to do that but dont know where these are or how they are executed

Thank you for pull request!

Suggests I should write tests - happy to do that but dont know where these are or how they are executed

I think this is fine assuming you've added new example in documentation which is automatically tested.

All ammendments suggested completed and remerged

Merged in c30132eb46f570fb50d70c96de9e977ba9e4772c, thank you for contribution!

repo: watir/watir


text:
Update package.json

Updated Deployment Tracker client dependency to pick up latest version. Delivers #77

Codecov Report

Merging #78 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #78   +/-   ##
=======================================
  Coverage   29.31%   29.31%
=======================================
  Files          13       13
  Lines         382      382
  Branches       56       56
=======================================
  Hits          112      112
  Misses        270      270

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6d9d26e...4ae1968. Read the comment docs.

authors: codecov-io ptitzler


text:
Update README.md

Remove the 2.1.1 versions.
And mention clearly that the code given is an example.

CC: @hanwavefront , @snicoll

repo: wavefrontHQ/wavefront-spring-boot


text:
[Feature]: Allow for default command line application

Feature description
It would be nice to have the ability to set which command line application launches when opening a new terminal.
On windows, Windows Command Prompt, PowerShell 5, PowerShell 7, WSL, Python, etc
Linux, vim, nvim, etc
Implementation Suggestion
Some kind of default settings, either via wsh editconfig or otherwise.
PowerShell 7 also does not support the wsh editconfig command - I know there's still users out there that use PS5 (weirdos) but I think lately PS7 has been the new "standard"
Anything else?
I'll get another feature added for PS7 support.

Same as #1435, check out https://docs.waveterm.dev/faq#on-windows-how-can-i-use-git-bash-as-my-default-shell

You can also create custom widgets which you can assign to each of these commands: https://docs.waveterm.dev/customwidgets

authors: Lizardbutt esimkowitz

repo: wavetermdev/waveterm


text:
swhkd does not process hotkeys of newly connected devices

Version Information:

Linux arch-dktp-alexis 5.19.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 31 Aug 2022 22:09:40 +0000 x86_64 GNU/Linux
swhkd 1.2.1 (installed from the swhkd-git AUR package)

Describe the bug:
swhkd does not register hotkeys from a newly connected keyboard (or disconnected then reconnected).
Expected behavior:
swhkd should be able to register hotkeys on a newly connected keyboard.
Actual behavior:
swhkd does not grab newly connected devices and thus cannot handle their keypresses events.
To Reproduce:
Start swhkd. Connect a keyboard (or disconnect yours before reconnecting it). Hotkeys will not work.
Additional information:
This can probably be solved using udev, with either the udev crate or tokio-udev crate.
My attempts at implementing this (the same content is also in the [wip] commits of each branch):

Using the tokio-udev crate (udev_async branch):

This does not work as-is. Unfortunately, sometimes udev events get 'stuck' and only processed after another batch of events comes in. In practice, this translates to e.g. a batch of 'Add' events (after connecting a device) only passed to the application loop after a batch of 'Remove' events (after disconnecting the same device for instance).

Using the udev crate iterators wrapped in a tokio stream (udev_sync branch):

This does not work as-is, as the MonitorSocket Iterator may return None when no udev events are available. tokio stops listening after the first None received from the stream, which means that events are not listened to at all.

Using the udev crate with a busy loop (udev_sync_busy branch):

At the cost of an always busy thread, this does receive all udev events in a timely manner.

The busy loop is the only one which is working, but also the worst performance-wise. See this one more as a proof of concept.

MWE for each implementation
Each exemple prints events from the 'input' subsystem as they are received. Each log is prefixed by a basic timestamp to highlight the difference between the actual physical event and the time it is passed to the application.
Busy loop
use std::error::Error;
use std::time::{SystemTime, UNIX_EPOCH};
use tokio::select;
use tokio_stream::StreamExt;
use udev::MonitorBuilder;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let mut udev = tokio_stream::iter(MonitorBuilder::new()?.match_subsystem("input")?.listen()?);
    loop {
        select! {
            udev_item = udev.next() => {
                if udev_item.is_none() {
                    continue;
                }
                let event = udev_item.unwrap();
                println!("[{:?}] '{:?}' event - devnode: {:?}",
                         SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs(),
                         event.event_type(),
                         event.devnode());
            }
        }
    }
}

On my computer, when disconnecting and reconnecting my keyboard, this gives:

[1663146489] 'Remove' event - devnode: Some("/dev/input/event3")
[1663146489] 'Remove' event - devnode: None
[1663146489] 'Remove' event - devnode: Some("/dev/input/event5")
[1663146489] 'Remove' event - devnode: None
[1663146489] 'Remove' event - devnode: Some("/dev/input/event6")
[1663146489] 'Remove' event - devnode: None
[1663146489] 'Remove' event - devnode: Some("/dev/input/event7")
[1663146489] 'Remove' event - devnode: None
[1663146489] 'Remove' event - devnode: Some("/dev/input/event8")
[1663146489] 'Remove' event - devnode: None
[1663146491] 'Add' event - devnode: None
[1663146491] 'Add' event - devnode: None
[1663146491] 'Add' event - devnode: None
[1663146491] 'Add' event - devnode: None
[1663146491] 'Add' event - devnode: None
[1663146491] 'Add' event - devnode: Some("/dev/input/event5")
[1663146491] 'Add' event - devnode: Some("/dev/input/event7")
[1663146491] 'Add' event - devnode: Some("/dev/input/event6")
[1663146491] 'Add' event - devnode: Some("/dev/input/event8")
[1663146491] 'Add' event - devnode: Some("/dev/input/event3")

Sync loop
Replace the inner loop with:
        select! {
            Some(event) = udev.next() => {
                println!("[{:?}] '{:?}' event - devnode: {:?}",
                         SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs(),
                         event.event_type(),
                         event.devnode());
            }
            else => {}
        }

This does work in this example, but events are ignored when trying the same thing in swhkd. I think this is because the stream is ignored by tokio on the first None it returns.
Async
use std::error::Error;
use std::time::{SystemTime, UNIX_EPOCH};
use tokio::select;
use tokio_stream::StreamExt;
use tokio_udev::{AsyncMonitorSocket, MonitorBuilder};

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let mut udev = AsyncMonitorSocket::new(MonitorBuilder::new()?.match_subsystem("input")?.listen()?)?;
    loop {
        select! {
            Some(Ok(event)) = udev.next() => {
                println!("[{:?}] '{:?}' event - devnode: {:?}",
                         SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs(),
                         event.event_type(),
                         event.devnode());
            }
        }
    }
}

For two disconnections/reconnections, this gives me (late events in bold):

[1663146903] 'Remove' event - devnode: Some("/dev/input/event3")
[1663146903] 'Remove' event - devnode: None
[1663146903] 'Remove' event - devnode: Some("/dev/input/event5")
[1663146903] 'Remove' event - devnode: None
[1663146903] 'Remove' event - devnode: Some("/dev/input/event6")
[1663146903] 'Remove' event - devnode: None
[1663146903] 'Remove' event - devnode: Some("/dev/input/event7")
[1663146903] 'Remove' event - devnode: Some("/dev/input/event8")
[1663146903] 'Remove' event - devnode: None
[1663146919] 'Remove' event - devnode: None
[1663146919] 'Add' event - devnode: None
[1663146919] 'Add' event - devnode: None
[1663146919] 'Add' event - devnode: None
[1663146919] 'Add' event - devnode: None
[1663146919] 'Add' event - devnode: None
[1663146919] 'Add' event - devnode: Some("/dev/input/event6")
[1663146919] 'Add' event - devnode: Some("/dev/input/event8")
[1663146919] 'Add' event - devnode: Some("/dev/input/event5")
[1663146919] 'Add' event - devnode: Some("/dev/input/event7")
[1663146925] 'Add' event - devnode: Some("/dev/input/event3")
[1663146925] 'Remove' event - devnode: Some("/dev/input/event3")
[1663146925] 'Remove' event - devnode: None
[1663146925] 'Remove' event - devnode: Some("/dev/input/event5")
[1663146925] 'Remove' event - devnode: None
[1663146925] 'Remove' event - devnode: Some("/dev/input/event6")
[1663146925] 'Remove' event - devnode: None
[1663146925] 'Remove' event - devnode: Some("/dev/input/event7")
[1663146925] 'Remove' event - devnode: None
[1663146925] 'Remove' event - devnode: Some("/dev/input/event8")
[1663146932] 'Remove' event - devnode: None
[1663146932] 'Add' event - devnode: None
[1663146932] 'Add' event - devnode: None
[1663146932] 'Add' event - devnode: None
[1663146932] 'Add' event - devnode: None
[1663146932] 'Add' event - devnode: None
[1663146932] 'Add' event - devnode: Some("/dev/input/event6")
[1663146932] 'Add' event - devnode: Some("/dev/input/event8")
[1663146932] 'Add' event - devnode: Some("/dev/input/event5")

Please note that here, the input which is the actual keyboard is /dev/input/event3 but it is passed to the application only after it has been removed again.
I do not understand why this happens. The most I was able to understand was that the stream sometimes got 'stuck' on Poll::Pending as if no new events were available (but there actually was).
Running udevadm monitor -s input at the same time shows events in real time with no lag, as the busy loop does. I was not able to track this issue further for now.

I looked into solving my issue with udev, but if anyone has any other idea, I can also take a shot at it !

Wow! Impressive bug report and you have my respect for putting in the time to not only investigate the issue but also to propose possible changes ❤.
The implementations look interesting, I'm not too familiar with udev but I just had one question, can udev detect the type of device that we're dealing with? this can help another bug where mice are picked up as keyboards sometimes.

Apart from that, this looks good, I'll review it in depth once I'm done with relocating!

I did some digging and found out that we don't need to create a new MonitorSocket from MonitorBuilder::new()?....listen(). We can create this builder once globally and then just poll the socket. Although this too is inefficient, it's...not as bad 🤣.

can udev detect the type of device that we're dealing with? this can help another bug where mice are picked up as keyboards sometimes.

Maybe?
Take a look at https://wiki.ubuntu.com/X/InputConfiguration#Device_classification: udev classifies devices when detected. With some basic testing on my side, it seems that my keyboard is indeed detected as ID_INPUT_KEYBOARD and my mouse as ID_INPUT_MOUSE... and also ID_INPUT_KEYOARD. My mouse is connected through a Logitech receiver which works for both their mice and keyboards, so it advertises for both (even though I do not have one of their keyboards). This may be better than the current keyboard detection, but it may still yield false positives.
A basic implementation could be:
// or tokio_udev
let mut udev_scan = udev::Enumerator::new()?;
udev_scan.match_subsystem("input")?;
udev_scan.match_property("ID_INPUT_KEYBOARD", "1")?;
let keyboard_devices = udev_scan.scan_devices()?
    .map(|dev| dev.devnode().map(|path| path.to_owned()))
    .flatten()
    .map(evdev::Device::open)
    .flatten()
    .collect::<Vec<_>>();

This works with some basic testing. I could implement a PR for that if you'd like!
To be honest, I think it would be better to fix #131 rather than detecting keyboards: if swhkd could properly re-emit events from all kinds of devices, there would be no need to only grab keyboards, right?

I did some digging and found out that we don't need to create a new MonitorSocket from MonitorBuilder::new()?....listen(). We can create this builder once globally and then just poll the socket. Although this too is inefficient, it's...not as bad :rofl:.

Is it that bad? Isn't the socket only created once anyway? What does it change to create first the builder?

I did some digging and found out that we don't need to create a new MonitorSocket from MonitorBuilder::new()?....listen(). We can create this builder once globally and then just poll the socket. Although this too is inefficient, it's...not as bad :rofl:.

Is it that bad? Isn't the socket only created once anyway? What does it change to create first the builder?

It's probably not too different but why perform extra computation when we can just poll the socket descriptor?

To be honest, I think it would be better to fix #131 rather than detecting keyboards: if swhkd could properly re-emit events from all kinds of devices, there would be no need to only grab keyboards, right?

Okay! it's cool that udev can detect these things for me, intruiging.
However I think you're correct, we can drop this task from udev entirely and just implement good mouse support too. I have attempted this in an open pr but failed eventually, would you like to try your hand at it? I'll help as much as I can.
Also I've been meaning to ask, would you like to join us as a maintainer? I've noticed that you have great debugging skills along with good problem solving skills.

To be honest, I think it would be better to fix #131 rather than detecting keyboards: if swhkd could properly re-emit events from all kinds of devices, there would be no need to only grab keyboards, right?

Okay! it's cool that udev can detect these things for me, intruiging. However I think you're correct, we can drop this task from udev entirely and just implement good mouse support too. I have attempted this in an open pr but failed eventually, would you like to try your hand at it? I'll help as much as I can.

Actually, I have already tried to work on this! I will report my findings in your PR.

Also I've been meaning to ask, would you like to join us as a maintainer? I've noticed that you have great debugging skills along with good problem solving skills.

That'd be great, I would love to be part of the team!
I am not exactly sure what that entails, however. Would it be possible to discuss this more? We can talk here or on Discord if you prefer.

That'd be great, I would love to be part of the team! I am not exactly sure what that entails, however. Would it be possible to discuss this more? We can talk here or on Discord if you prefer.

Sure, what's your username ? Ps: we are very laid back, there's no set responsibilities but let's talk over discord.

It is the same handle as my GitHub profile: ajanon

It is the same handle as my GitHub profile: ajanon

And the tag?

Oh, right, sorry: ajanon#5057

I have time this weekend to work on the udev problem! I'll go ahead and start on a patch asap. PS: Cleared up some unnecessary messages.

I have reported this issue to tokio-udev upstream: https://github.com/jeandudey/tokio-udev/issues/17
Hopefully they can help us!

Thanks! I feel the delayed events might be a bug with upstream.

https://app.bountysource.com/issues/118215631-swhkd-does-not-process-hotkeys-of-newly-connected-devices
If anyone is interested!

The latest updates in the tokio_udev repo make it possible to detect added/removed devices.
This is implemented in the udev_async branch.
I will be waiting for a release of tokio_udev before creating a PR for it.

When running from the main branch swhkd doesn't give any error but it doesn't seem to accept keyboard events any longer. When running from the udev_async branch I get the same error as @Shinyzenith .
[2023-06-03T15:31:04Z INFO  swhkd] Device 'Gaming keyboard' removed
[2023-06-03T15:31:04Z INFO  swhkd] Device 'Gaming keyboard' removed
[2023-06-03T15:31:04Z INFO  swhkd] Device 'Gaming keyboard' removed
thread 'main' panicked at 'removal index (is 3) should be < len (is 1)', swhkd/src/daemon.rs:318:49
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted

Is this still an issue?
I wanted to look into it because of the bounty but I can't reproduce the issue with the udev_async head. I can plug and unplug my USB keyboard freely and swhkd seems to cope just fine.
(Incidentally I am doing this on a setup where no commands are actually run, on any hotkey swhkd prints "failed to send command through ipc". swhkd can't find the socket created by swhkd for reasons unclear to me. But that can't be the reason for the error not happening to me, can it?)

I have detailed the steps to a crash issue two comments above. Can you reproduce that?

I have detailed the steps to a crash issue two comments above. Can you reproduce that?

I can connect and disconnect the USB keyboard, and pressing super in both states, on both keyboards (obvs on the external only when connected), makes swhkd log the key. Using any combinations involving super also are logged fine (though the commands not executed, as I mentioned above).
The only combination in my config is a Super + l. I don't think the configured combinations should be relevant to the location of the crash either, however.

Here are the steps I found to reliably reproduce the issue. I have an external keyboard connected to my laptop. Upon disconnecting my external keyboard and attempting to press just a modifier key ( in this case, Super ) It causes swhkd to crash. It seems to be crashing on the keyboard_stream_map match.

Am I missing something?

Fixed the socket issue with commands, it was just polkit being weird like always. still can't reproduce the crash.
Wow, that's some main(). Looking at it, I might have some theories, even if I can't reproduce it personally. Is it alright if I make some educated-guess changes and ask any of you (who suffer from the crash) to test?

Still don't face the issue, but I came up with some theories and made some fixes:
https://github.com/ConcurrentCrab/swhkd/tree/udev_async
https://github.com/waycrate/swhkd/pull/216
@Shinyzenith and others: can you test and see if these fixes work for you?

Fixed the socket issue with commands, it was just polkit being weird like always. still can't reproduce the crash.
Wow, that's some main(). Looking at it, I might have some theories, even if I can't reproduce it personally. Is it alright if I make some educated-guess changes and ask any of you (who suffer from the crash) to test?

Sorry for the late reply. I'm more than happy to do some more testing. Yeah the main() needs cleanup...a lot of it.

I'll review the PR and get that going.

I have merged the pr sent by @ConcurrentCrab. Just to be sure @CluelessTechnologist would you mind testing the latest HEAD of udev branch once more? If all is good to go, I'll merge.

I have merged the pr sent by @ConcurrentCrab. Just to be sure @CluelessTechnologist would you mind testing the latest HEAD of udev branch once more? If all is good to go, I'll merge.

Sure. Just tested the latest commit from udev branch. Happy to report that it's still working:
[2023-07-19T22:49:04Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event2' removed
[2023-07-19T22:49:04Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event3' removed
[2023-07-19T22:49:04Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event4' removed
[2023-07-19T22:49:04Z INFO  swhkd] Device 'Logitech Gaming Mouse G502 Keyboard' at '/dev/input/event1' removed
[2023-07-19T22:49:11Z ERROR swhkd] Could not open evdev device at /dev/input/mouse0: Inappropriate ioctl for device (os error 25)
[2023-07-19T22:49:11Z DEBUG swhkd] Keyboard: Gaming keyboard
[2023-07-19T22:49:11Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event2' added.
[2023-07-19T22:49:11Z DEBUG swhkd] Keyboard: Gaming keyboard
[2023-07-19T22:49:11Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event4' added.
[2023-07-19T22:49:11Z DEBUG swhkd] Keyboard: Gaming keyboard
[2023-07-19T22:49:11Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event3' added.
[2023-07-19T22:49:11Z DEBUG swhkd] Keyboard: Logitech Gaming Mouse G502 Keyboard
[2023-07-19T22:49:11Z INFO  swhkd] Device 'Logitech Gaming Mouse G502 Keyboard' at '/dev/input/event1' added.
[2023-07-19T22:49:11Z TRACE swhkd] Other: Logitech Gaming Mouse G502

@CluelessTechnologist I made some minor changes a force  pushed. Would you mind testing the branch head one last time? My apologies. After that it'll be ready for merge.

@CluelessTechnologist I made some minor changes and force pushed. Would you mind testing the branch head one last time? My apologies. After that it'll be ready for merge.

Okay I pulled the latest commit and recompiled and reinstalled. Still seems to be working:
[2023-07-21T13:44:11Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event2' removed
[2023-07-21T13:44:11Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event3' removed
[2023-07-21T13:44:11Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event4' removed
[2023-07-21T13:44:11Z INFO  swhkd] Device 'Logitech Gaming Mouse G502 Keyboard' at '/dev/input/event1' removed
[2023-07-21T13:44:18Z ERROR swhkd] Could not open evdev device at /dev/input/mouse0: Inappropriate ioctl for device (os error 25)
[2023-07-21T13:44:18Z DEBUG swhkd] Keyboard: Gaming keyboard
[2023-07-21T13:44:18Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event2' added.
[2023-07-21T13:44:18Z DEBUG swhkd] Keyboard: Gaming keyboard
[2023-07-21T13:44:18Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event4' added.
[2023-07-21T13:44:18Z DEBUG swhkd] Keyboard: Logitech Gaming Mouse G502 Keyboard
[2023-07-21T13:44:18Z INFO  swhkd] Device 'Logitech Gaming Mouse G502 Keyboard' at '/dev/input/event1' added.
[2023-07-21T13:44:18Z DEBUG swhkd] Keyboard: Gaming keyboard
[2023-07-21T13:44:18Z INFO  swhkd] Device 'Gaming keyboard' at '/dev/input/event3' added.
[2023-07-21T13:44:18Z TRACE swhkd] Other: Logitech Gaming Mouse G502

Nice, seems to be working now. Great job!

Awesome! Happy to be of help to y'all, and thanks for the bounty!
@Shinyzenith I can't collect the bounty on bountysource.com until this issue is marked as closed, so I'd appreciate it if you could close it and test miscelleneous changes afterwards :P

@CluelessTechnologist I am not sure how you wish to split the bounty but I'll list out the people who have helped us achieve the goal:
@ajanon For their excellent udev async patch and reporting upstream on fixing the epoll error.
@jeandudey For fixing the epoll error.
@ConcurrentCrab For fixing index out of bound error due to incorrect state management.

@ConcurrentCrab were you able to claim the bounty?

I thought it best not to take any actions on my side yet in case you had actually intended to split it.

Thank you for the clarification!

@ConcurrentCrab were you able to claim the bounty?

I thought it best not to take any actions on my side yet in case you had actually intended to split it.

Unfortunately there's no way to split it. The one who creates the PR is eligible for the bounty. So you can go ahead and claim it since you were the one that created the PR.

@Shinyzenith I released the fixes for tokio-udev but who fixed it was @sjoerdsimons, so I don't actually deserve any bounty but him

@sjoerdsimons here's the bounty for tokio-udev: https://app.bountysource.com/issues/121368069-events-were-blocked-while-poll-next

Unfortunately there's no way to split it. The one who creates the PR is eligible for the bounty. So you can go ahead and claim it since you were the one that created the PR.

To be completely fair, alexis created the initial fix, just in a branch. I opened the PR. I obviously don't want any part of the bounty, I'm just stating the facts.
The next PR fixing the index out of bounds issue was from ConcurrentCrab.

@ConcurrentCrab were you able to claim the bounty?

I thought it best not to take any actions on my side yet in case you had actually intended to split it.

Unfortunately there's no way to split it. The one who creates the PR is eligible for the bounty. So you can go ahead and claim it since you were the one that created the PR.

@Shinyzenith I released the fixes for tokio-udev but who fixed it was @sjoerdsimons, so I don't actually deserve any bounty but him

@sjoerdsimons here's the bounty for tokio-udev: https://app.bountysource.com/issues/121368069-events-were-blocked-while-poll-next

I have no need for a bounty; I'm happy for @jeandudey to pick it up as thanks for maintaining tokio-udev or for the funding to be re-used for another issue that seems worthwhile.

So a kinda update regarding bountysource: I had requested a cashout of a pair of bounties I did a couple weeks ago, and have been patiently waiting for the past few weeks for it to process, which I found weird as their FAQ claimed it should be a week at max. So I tried to look up some stuff and I kinda got my answer: https://github.com/bountysource/core/issues
So it goes without saying: please stop giving more money to bountysource, or displaying their services on your projects. I think it;s pretty safe to say they're not going to pay any of it out now, even if they even exist anymore on any level.
As for money that already exists on their platform, I saw somebody suggest in the issues asking paypal to chargeback any transactions that have happened less than 6 months ago; since bountysource is unresponsive on all levels, paypal should automatically resolve in favour of customers.
@CluelessTechnologist Of course, That is what I'd recommend you do for the bounties you've posted for your issues as well. And, if I may ask this favour of you, I would also request you to paypal me for this bounty (only once the chargeback claim is successful, of course), because I sort of did actually need the pocket money. This would also allow you to split it with @ajanon as they wish.

Sadly I was warned about BountySource before but there isn't that many sites that offer bounties with escrow. Rysolv is one that I have used before. I was debating switching to it but decided against it which was my mistake I guess.
I'm really sorry about this @ConcurrentCrab. Please send me a DM with your PayPal email on Matrix (@cluelesstechnologist:matrix.org) or email me. (email in my GitHub profile)
Another option is to enable Github Sponsors on your profile.

@CluelessTechnologist I sent you an e-mail (from the same address as my commits).

Very unfortunate situation. I hope bountysource returns you your capital @CluelessTechnologist.

authors: CluelessTechnologist ConcurrentCrab Shinyzenith ajanon sjoerdsimons

repo: waycrate/swhkd


text:
Clarify forking process

While running through the process, I hit a couple snags. Hopefully this is an improvement, feedback welcome.

🚨 Hey @nphilbrook, looks like there was an issue processing your fork request!   Please ensure you've committed a properly-formatted YAML (.yml) file to the requests/fork/ directory.

authors: lhasa-ospo nphilbrook

repo: wayfair/ospo-automation


text:
Meta extension: Empty line in YAML is considered as End of Document

For:
PyYAML==3.11
This fails:
doc = """---
First name: John
Last name: Doe

email: john@doe.com
---

Hello World!
"""

import markdown
md = markdown.Markdown(extensions=[markdown.extensions.meta.MetaExtension(yaml=True)])
assert md.convert(doc) == '
Hello World
'

I understand that YAML can have blank lines.

Looking at meta.py, line 80, it appears this is deliberate?
if line.strip() == '' or have_yaml and YAML_END_RE.match(line):
    break # blank line or end of YAML header - done

I am very much inclined to reject this. I was hesitant to add support for YAML to begin with. Previously, we didn't even support the YAML style deliminators. At that time, a blank line was the only way to determine the end of the meta data and the start of the document.  As you note, the implementation is still very much dependent on that blank line. And i intend to continue to support meta-data without YAML style deliminators for the foreseeable future. However, as stated in the documentation, this feature (meta-data) was inspired by MultiMarkdown which also includes support for YAML style deliminators (but not YAML data). Therefore, in the last release, we added support for the deliminators. However, it is the blank line that really ends the data section (the deliminators are just ignored as long as they are on the first or last line). Yes, this means that a blank line gets precedence over the closing deliminator in determining the end of the data section. This was an intentional design decision which gives us full conformity with MultiMarkdown's implementation.
In the process of adding the above mentioned support, someone requested that we also support YAML data. I was hesitant, but saw the value in users being able to have more control over the structure of their data so I included it as an optional feature. That said, I am more inclined to remove it that to expand it. It seems to me that if you want full-featured YAML support, a third party implementation would be better suited to your needs.

I kinda feel like YAML support could be its own plugin...maybe even should.  When people see YAML support, they are going to think they can use the traditional YAML frontmatter (which is what they really want).  But the truth is you can't, which I think is fine.
I kind of thought YAML was already implemented as 3rd party extension listed in the wiki https://github.com/teoric/python-markdown-yaml-meta-data, so I was kind of surprised to see it added officially.  I haven't used it the 3rd party or the official because I strip out YAML via a wrapper around Markdown to use as the data as actual HTML meta data, or use special keywords to set settings or dynamically control which extensions get loaded for a specific file.

@facelessuser in retrospect I agree. I suspect I will be removing the support for YAML data. I would have never implemented it myself, but when the pull-request came in with it already implemented, I accepted it without thinking it through fully. At the time, it didn't occur to me that YAML permits blank lines and that this would not offer full support. Taking a step back, it is now clear to me that the existing meta-data extension and YAML support are two very different things which should not exist together in the same extension.

I haven't used the 3rd party or the official because I strip out YAML via a wrapper around Markdown to use as the data as actual HTML meta data, or use special keywords to set settings or dynamically control which extensions get loaded for a specific file.

Personally, this is how I think it should be done. I'm always surprised how many people want to use the extension and continue to insist when I suggest they use a wrapper instead. I suppose it is because no one wants to reinvent the wheel. However, if someone wrote a good implementation which was available as a third party package (perhaps via PyPI), I might be inclined to deprecate the extension altogether.
As an aside, I wrote the original meta-data extension many-many years ago as an experiment. At the time I was a relatively user of Python-Markdown only. However, when I announced the meta-data extension (which I believe was the third I had written) it caught the attention of the maintainer of Python-Markdown (Yuri at the time) and he asked me to join the project. At the time, no extensions were not being shipped with Python-Markdown and when we decided to start doing that, a number of my extensions were all added in at once. I was never particularly happy that meta-data was included because once i built it and started using it, I realized that a wrapper would make for a much more useful tool. I just never developed the need or found the time to write it myself.

Perhaps I should explain why I think this would be better as a wrapper than an extension. Consider two projects which make heavy use of the existing Meta-Data extension. mkdocs and Pelican. Both of those projects have a list of default extensions that their users will get out-of-the-box without changing any settings. In each instance, Meta-Data is one of those extensions (and it is required for things to work in each of those tools). However, each project offers a setting to their users where the user can define their own list of extensions. Naturally, that setting can be used to remove/replace the default list. However, what happens when the user removes the meta-data extension? Then the list needs to be checked to ensure that that extension is included and if not, insert it before passing the list on to Markdown. Otherwise things will break. Wouldn't it make more sense for those projects to just run the input file through some code which extracted the meta-data first, then pass the rest of the document on to Markdown without needing to juggle the extension list. And remember, the list of extensions can be class instances or string names. You can't just do a simple if not 'meta' in extensions. You need to loop through the list and check the type of each item. It is true that mkdocs is simpler as their settings file is YAML (however PyYAML does allow python objects to be specified in YAML files so I wouldn't rule the possibility out), but Pelican uses a Python file for settings. You can totally import an extension and create an instance to add to the list of extensions (which is great for the user to configure each extension how s\he desires). The point is, I would much rather see these projects doing something like this:
data, md = get_data(source)
html = markdown.markdown(md, **get_md_kwargs(config, data))
Template(body=html, context=get_context(config, data))

While that is mostly pseudo code which leaves the implementation details out, I think the general concept is a much better way of working with meta data. As an alternate example, the above functions could all be methods on a class. The important point is that the meta-data is extracted before passing the document on to Markdown.

With stuff like PyYAML, getting the frontmatter and parsing it is trivial.  Something like this can be added to a wrapper real easy.  Heck you could use whatever content you wanted, JSON, YAML, <insert some obscure format>.
import re
import yaml

def get_frontmatter(string):
    """ Get frontmatter from string """
    frontmatter = {}

    if string.startswith("---"):
        m = re.search(r'^(---(.*?)---[ \t]*\r?\n)', string, re.DOTALL)
        if m:
            try:
                frontmatter = yaml.load(m.group(2))
            except:
                pass
            string = string[m.end(1):]

    return frontmatter, string

I struggled to find a reason that required Python Markdown to parse my frontmatter, when I realized there wasn't really any reason to have Python Markdown do it for me (as it did absolutely nothing with it once it stripped it out), I realized I should just parse it out myself and treat it exactly as it was, just extra data that Python Markdown doesn't care about.  It worked out better this way because quite often, I wanted to do something with the info before I handed the actual text to Python Markdown.

Right. It occurs to me that when I first created the meta-data extension, JSON and YAML were, at most, in their very early stages (definitely not popular or standard library material) and so the benefit of having a parser of a key/value data format which was easy to write and required no additional dependencies was very attractive to users (the only standard library option was INI files). Today, given the great strides that have been made in python package management, third party dependencies are mostly a non-issue and the more fully developed data formats should be more desirable (PYAML's c dependency is optional so that's not  a blocker either). In fact, the maintainers of those projects are better equipped to maintain that sort of thing than a Markdown project would be. That being the case, I think maybe meta-data will be deprecated in the near future. We should leave data parsing to data parsers.

To be clear, if I deprecate meta-data, the extension would be broken out into its own package which would get no further development by me. If other's wanted to use it or fork it, they certainly could, although, as stated above, there are better options.

@tomchristie or @d0ugal (from mkdocs) and @getpelican (from Pelican), do you guys have any input on the future of Python-Markdown's Meta-Data extension. I'd like to hear from you (presumably Python-Markdown's biggest users now that Django removed the markup contrib app) before doing anything drastic like deprecating an extension that both projects rely heavily on. See the earlier comments in this discussion for the details, but the current idea is to remove the broken YAML support immediately (in 2.6.1 as a bug-fix) as it has not even been a week and then start the deprecation process for the entire extension with the next major release which would be months out at the earliest.
Oh and look for a plan for a 3.0 published sometime in the near future. I'm ruminating on it now and will start to put something together soon. I don't expect much more in the 2.x series (3.0 might be next or if not, after 2.7).

Could something like
if not have_yaml and line.strip() == '' or have_yaml and YAML_END_RE.match(line):

be used for removingfixing broken YAML support?

@kernc that could work, but I'm not sure we should. One thing is certain, I'm not interested in maintaining YAML support long-term. So every time I get a bug report from here on out, I'll wish I had ripped it out now. I'm not sure why people are so insistent that the built-in extensions must support their desired feature. Why can't this be in a third party extension? Especially now that the built-in extensions get no preferential treatment.

@waylan I've no problem with it being moved into a third party package - certainly sounds like the right approach for your project, and it's probably not much work for us - if you wanted to handle opening a ticket on MkDocs pointing us at the right place etc for the new package, then I can't see any objection.

+1, I agree with @tomchristie.

For anyone who is interested, I threw together a package that parses meta-data as I would want it to work. It is a standalone package (not an extension) and it can work with any lightweight markup language (not just Markdown). I haven't written the YAML implementation yet, but the MultiMarkdown style meta-data is feature complete. You can even define types for specific keys. Any feedback is welcome (please send feedback through that project not here). Thanks.
https://github.com/waylan/docdata

authors: alixedi d0ugal facelessuser kernc tomchristie waylan

repo: waylan/Python-Markdown


text:
chore(master): release 1.2.40

:robot: I have created a release beep boop
1.2.40 (2023-08-24)
Miscellaneous

deps: update dependency wayofdev/cs-fixer-config to v1.2.46 (80dbcaf)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/wayofdev/laravel-symfony-serializer/releases/tag/v1.2.40 :sunflower:

repo: wayofdev/laravel-symfony-serializer


text:
Add tools to assemble packages

Description
This PR adds tooling to assemble packages (include plugins and production configuration).
Issues Resolved
Closes #76
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[ ] GitHub issue/PR created in OpenSearch documentation repo for the required public documentation changes (#[Issue/PR number])

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Superseded by https://github.com/wazuh/wazuh-indexer/pull/85

repo: wazuh/wazuh-indexer


text:
generated get calls include body by default

So I was generating code to work with the EVE Online api and was getting bad requests returned by the server.
Dug into it and it was because the generated code included empty but non-null bodies in all requests and the server was rejecting them. Presumably their server is just being more strict than the HTTP spec which allows for GET to have a body that would be ignored.
Is it worth adding an option to codegen that would tell the templates to not include an empty body on verbs like GET/DELETE? is there a reason not to? I have this half-implemented (template modified, but no opt) and would be happy to submit a PR if its something desired.

I was just dealing with this today(2nd time this year). It's painful enough to troubleshoot that I hope we can get a fix.

@duffman03 accordingly to what you mention in the edition added to your comment, this issue should be marked as closed?

Actually my edit shows that swagger codegen is actually out of spec now. Previously it was tolerable to include empty bodies in those requests, but now its agaisnt spec.

repo: wcandillon/swagger-js-codegen


text:
BarChart X and Y axis lable does not show

BarChart X and Y axis lable does not show
graph := chart.BarChart{ 		// Title:      "Consumption Trend", 		// TitleStyle: chart.StyleShow(), 		Background: chart.Style{ 			Padding: chart.Box{ 				Top:    10, 				Left:   10, 				Right:  40, 				Bottom: 40, 			}, 		}, 		Height:   200, 		Width:    400, 		BarWidth: 50, 		XAxis: chart.Style{ 			StrokeWidth: 1, 			Show:        true, 		}, 		YAxis: chart.YAxis{ 			Name: "(Cu M)", 			NameStyle: chart.Style{ 				Show:                true, 				TextRotationDegrees: 45.0, 			}, 			Style: chart.Style{ 				StrokeWidth: 1, 				Show:        true, 			}, 			TickStyle: chart.Style{ 				TextRotationDegrees: 45.0, 			}, 			ValueFormatter: func(v interface{}) string { 				return fmt.Sprintf("%0.2f (Cu M)", v) 			}, 			Range: yrange, 		}, 		Bars: chartValues, 	}

Same for me, even doing bar_chart.go.
OS X 10.14.5 (18F132)
go version go1.13.1 darwin/amd64

I'm having this same issue. No labels on either axis.

It seems related to modules. This resolved https://github.com/wcharczuk/go-chart/issues/132 to me

authors: alexaung dylanirlbeck xn3cr0nx zackb

repo: wcharczuk/go-chart


text:
Starting point for sklearn EM algorithm

Can components be initialized using sklearn?

https://stackoverflow.com/questions/45170093/latent-dirichlet-allocation-with-prior-topic-words


text:
HTML cleanup.

Organize some HTML.

🆗

repo: weareswat/squad-share


text:
Fixed url sanitization

URL sanitation was miss behaving leading to URLs with suffix .git that go-git doesn't accept causing errors when attempting to clone the repo. It was tested by adding unit tests  for all possible combinations and making sure smoke and accept tests passed.

Hey Jose, have you manually tested the whole flow and checked the status of the GitRepository? I remember that breaking without the .git.

authors: josecordaz luizbafilho

repo: weaveworks/weave-gitops


text:
[Question]Deleting the weave-net pod

Hi All,
I have installed the weave plugin through kubectl apply, Now I want to remove the pods that is running on kube-system namespace , is there any way to uninstall the weave plugin?
I tried deleting the pods and it is coming back as it is deployed as daemonset.
Edited: removed unused parts of the template

Hi @selvakumarnms,
The way to remove the pods is via kubectl delete. For example, if you created the Weave Net pods using:
$ kubectl -n kube-system apply -f https://git.io/weave-kube-1.6

you would remove them using:
$ kubectl -n kube-system delete -f https://git.io/weave-kube-1.6

I tried to use $ kubectl -n kube-system delete -f https://git.io/weave-kube-1.6 to remove, but it hung. I wonder why it did not proceed
$ kubectl -n kube-system delete -f https://git.io/weave-kube-1.6
clusterrole "flannel" deleted
clusterrolebinding "flannel" deleted
serviceaccount "flannel" deleted
configmap "kube-flannel-cfg" deleted

@kasheemlew it's better to create your own issue than to piggy-back on an old one.
That said, it's extremely surprising that the Weave Net config would delete a flannel role or account.  Could you double-check?
To find out more about what kubectl is doing, increase the logging level, e.g. kubectl -v=8 -n kube-system delete -f https://git.io/weave-kube-1.6

Hi @selvakumarnms,
The way to remove the pods is via kubectl delete. For example, if you created the Weave Net pods using:
$ kubectl -n kube-system apply -f https://git.io/weave-kube-1.6

you would remove them using:
$ kubectl -n kube-system delete -f https://git.io/weave-kube-1.6

Hi @selvakumarnms,
The way to remove the pods is via kubectl delete. For example, if you created the Weave Net pods using:
$ kubectl -n kube-system apply -f https://git.io/weave-kube-1.6

you would remove them using:
$ kubectl -n kube-system delete -f https://git.io/weave-kube-1.6

thanks, it work for me.

Assuming you've installed the weave package using:
export VAL=$(kubectl version | base64 | tr -d '\n')
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$VAL"

You can remove it using the same config file.
kubectl delete -f "https://cloud.weave.works/k8s/net?k8s-version=$VAL"

authors: Winterhart bboreham kasheemlew marccarre selvakumarnms wen73

repo: weaveworks/weave


text:
weave reset does not remove net-plugin v2 weavedata.db

What you expected to happen?
After running weave reset --force, /var/lib/weave/weavedata.db would be removed.
What happened?
/var/lib/weave/weavedata.db still exists on the host.
How to reproduce it?

Install net-plugin v2.
Disable and remove the plugin.
Run weave reset --force

Anything else we need to know?
This is a similar issue that was fixed with PR #2610.
Versions:

$ weave version
weave script 2.0.4
weave 2.0.4
$ docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:23:31 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:19:04 2017
 OS/Arch:      linux/amd64
 Experimental: false
$ uname -a
Linux NUC 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Logs:
NA
Network:
NA

Thanks for the report.

FYI, I think I have a fix for this.  Similar to have removal is handled for weave-kube.

Fixed by #3114

authors: brb dtshepherd