text:
How to use system.NewCreateAccountWithSeedInstruction properly?

I am trying to replicate this solana example hello world program in .ts using go. But I am stuck on using system.NewCreateAccountWithSeedInstruction. I got this error Transaction simulation failed: Error processing Instruction 0: invalid instruction data when executing confirm.SendAndConfirmTransaction using system.NewCreateAccountWithSeedInstruction instruction:

2021/12/24 11:57:11 (*jsonrpc.RPCError)(0xc0001f5890)({
 Code: (int) -32002,
 Message: (string) (len=87) "Transaction simulation failed: Error processing Instruction 0: invalid instruction data",
 Data: (map[string]interface {}) (len=3) {
  (string) (len=8) "accounts": (interface {}) <nil>,
  (string) (len=3) "err": (map[string]interface {}) (len=1) {
   (string) (len=16) "InstructionError": ([]interface {}) (len=2 cap=2) {
    (json.Number) (len=1) "0",
    (string) (len=22) "InvalidInstructionData"
   }
  },
  (string) (len=4) "logs": ([]interface {}) (len=2 cap=2) {
   (string) (len=51) "Program 11111111111111111111111111111111 invoke [1]",
   (string) (len=73) "Program 11111111111111111111111111111111 failed: invalid instruction data"
  }
 }
})

And here is my code in go:
func main() {
	// var
	ctx := context.Background()
	rpcCli := rpc.New(rpc.LocalNet_RPC)
	wsCli, _ := ws.Connect(ctx, rpc.LocalNet_WS)
	payerPrivateKey, _ := solana.PrivateKeyFromBase58("5LRLfrUP22VtiNaPGAEgHPucoJmG8ejmomMVmpn4fkXjexYsT7RQGfGuMePG5PKvecZxMGrqa6EP2RmYcm7TYQvX") // don't worry, I only use this private key in my local computer
	payerAccount, _ := solana.WalletFromPrivateKeyBase58(payerPrivateKey.String())
	programID := "4sCcZNQR8vfWckyi5L9KdptdaiLxdiMjVgKQay7HxzmK"
	programPubKey, _ := solana.PublicKeyFromBase58(programID)

	// create new subaccount (seed account)
	newSubAccount, err := solana.CreateWithSeed(
		payerAccount.PublicKey(),
		"hello",
		programPubKey,
	)
	if err != nil {
		log.Fatalln("Failed to CreateWithSeed")
	}

	instruction, err := system.NewCreateAccountWithSeedInstruction(
		payerAccount.PublicKey(),
		"hello",
		918720,
		4,
		programPubKey,
		payerAccount.PublicKey(),
		newSubAccount,
		payerAccount.PublicKey(),
	).ValidateAndBuild()
	if err != nil {
		log.Fatalln("Failed building NewCreateAccountInstruction instruction", err)
	}

	instructionData, _ := instruction.Data()
	fmt.Printf("------> instructionData length: %d bytes, data: %v\n", len(instructionData), instructionData)

	instructions := []solana.Instruction{instruction}
	signers := []solana.PrivateKey{payerAccount.PrivateKey}
	opts := []solana.TransactionOption{solana.TransactionPayer(payerAccount.PublicKey())}

	// create recent blockhash
	recent, err := rpcCli.GetRecentBlockhash(ctx, rpc.CommitmentFinalized)
	if err != nil {
		log.Fatalln(err)
	}

	// create transaction
	tx, err := solana.NewTransaction(
		instructions,
		recent.Value.Blockhash,
		opts...,
	)
	if err != nil {
		log.Fatalln(err)
	}

	// sign transaction
	_, err = tx.Sign(
		func(key solana.PublicKey) *solana.PrivateKey {
			for _, signer := range signers {
				if signer.PublicKey().Equals(key) {
					return &signer
				}
			}

			return nil
		},
	)
	if err != nil {
		log.Fatalln(err)
	}
	tx.EncodeTree(text.NewTreeEncoder(os.Stdout, "Transfer SOL"))

	// send and confirm transaction
	_, err = confirm.SendAndConfirmTransaction(
		ctx,
		rpcCli,
		wsCli,
		tx,
	)
	if err != nil {
		log.Fatalln(err)
	}
}

I tried to print the instruction.Data() in go and got this
------> instructionData length: 90 bytes, data: [3 0 0 0 204 95 77 127 148 25 135 127 89 146 22 90 233 80 113 3 70 176 165 222 81 200 100 223 117 165 155 44 53 225 124 20 5 104 101 108 108 111 192 4 14 0 0 0 0 0 4 0 0 0 0 0 0 0 57 111 59 111 183 248 249 251 128 174 206 0 81 22 3 173 244 104 15 249 239 112 33 255 66 169 29 66 7 106 231 230]

And here is the instruction data on the .ts from the solana helloworld example for comparison
console.log(instruction.data.buffer)
ArrayBuffer {
  [Uint8Contents]: <03 00 00 00 cc 5f 4d 7f 94 19 87 7f 59 92 16 5a e9 50 71 03 46 b0 a5 de 51 c8 64 df 75 a5 9b 2c 35 e1 7c 14 05 00 00 00 00 00 00 00 68 65 6c 6c 6f c0 04 0e 00 00 00 00 00 04 00 00 00 00 00 00 00 39 6f 3b 6f b7 f8 f9 fb 80 ae ce 00 51 16 03 ad f4 68 0f f9 ef 70 21 ff 42 a9 1d 42 07 6a e7 e6>,
  byteLength: 97
}

And here is the conversion from hex to 0-255
3 0 0 0 204 95 77 127 148 25 135 127 89 146 22 90 233 80 113 3 70 176 165 222 81 200 100 223 117 165 155 44 53 225 124 20 5 **0 0 0 0 0 0 0** 104 101 108 108 111 192 4 14 0 0 0 0 0 4 0 0 0 0 0 0 0 57 111 59 111 183 248 249 251 128 174 206 0 81 22 3 173 244 104 15 249 239 112 33 255 66 169 29 66 7 106 231 230

I see that there are 7 bytes difference (the one that I bold) in 0 0 0 0 0 0 0. What do I do wrong? Am I missing something? Is there any example on how to use this instruction? Thanks!

Hi
Try go get github.com/gagliardetto/solana-go@843d863a52562739a20dbfba59e082c55365dd4a
The error was wrong string length encoding: https://github.com/gagliardetto/solana-go/commit/843d863a52562739a20dbfba59e082c55365dd4a
The issue should be fixed in that commit (not yet in a release), but here's the breakdown of what the instruction encoded data contains:
solana-go:
(instruction index as uint32, 4 bytes) 3 0 0 0
(base pubkey, 32 bytes) 204 95 77 127 148 25 135 127 89 146 22 90 233 80 113 3 70 176 165 222 81 200 100 223 117 165 155 44 53 225 124 20
(seed string length, which should be uint64, 8 bytes) 5
(the seed string bytes, in this case 5 bytes) 104 101 108 108 111
(lamports uint64, 8 bytes) 192 4 14 0 0 0 0 0
(space uint64, 8 bytes) 4 0 0 0 0 0 0 0
(owner pubkey, 32 bytes) 57 111 59 111 183 248 249 251 128 174 206 0 81 22 3 173 244 104 15 249 239 112 33 255 66 169 29 66 7 106 231 230
typescript:
3 0 0 0
204 95 77 127 148 25 135 127 89 146 22 90 233 80 113 3 70 176 165 222 81 200 100 223 117 165 155 44 53 225 124 20
5 0 0 0 0 0 0 0
104 101 108 108 111
192 4 14 0 0 0 0 0
4 0 0 0 0 0 0 0
57 111 59 111 183 248 249 251 128 174 206 0 81 22 3 173 244 104 15 249 239 112 33 255 66 169 29 66 7 106 231 230

Side note:
You can debug instructions using "github.com/gagliardetto/utilz".NewWriteByWrite
package main

import (
	"fmt"
	"log"

	bin "github.com/gagliardetto/binary"
	"github.com/gagliardetto/solana-go"
	"github.com/gagliardetto/solana-go/programs/system"
	"github.com/gagliardetto/utilz"
)

func main() {
	payerPrivateKey := solana.MustPrivateKeyFromBase58("5LRLfrUP22VtiNaPGAEgHPucoJmG8ejmomMVmpn4fkXjexYsT7RQGfGuMePG5PKvecZxMGrqa6EP2RmYcm7TYQvX") // don't worry, I only use this private key in my local computer
	payerAccount, _ := solana.WalletFromPrivateKeyBase58(payerPrivateKey.String())
	programID := "4sCcZNQR8vfWckyi5L9KdptdaiLxdiMjVgKQay7HxzmK"
	programPubKey := solana.MustPublicKeyFromBase58(programID)

	// create new subaccount (seed account)
	newSubAccount, err := solana.CreateWithSeed(
		payerAccount.PublicKey(),
		"hello",
		programPubKey,
	)
	if err != nil {
		log.Fatalln("Failed to CreateWithSeed")
	}

	instruction := system.NewCreateAccountWithSeedInstruction(
		payerAccount.PublicKey(),
		"hello",
		918720,
		4,
		programPubKey,
		payerAccount.PublicKey(),
		newSubAccount,
		payerAccount.PublicKey(),
	)

	buffer := utilz.NewWriteByWrite("CreateAccountWithSeed")

	err = bin.NewBinEncoder(buffer).Encode(instruction)
	if err != nil {
		panic(err)
	}

	fmt.Println(buffer.String())
}

which will print something like this:
CreateAccountWithSeed:
- 0: [204](1)
- 1: [95](1)
- 2: [77](1)
- 3: [127](1)
- 4: [148](1)
- 5: [25](1)
- 6: [135](1)
- 7: [127](1)
- 8: [89](1)
- 9: [146](1)
- 10: [22](1)
- 11: [90](1)
- 12: [233](1)
- 13: [80](1)
- 14: [113](1)
- 15: [3](1)
- 16: [70](1)
- 17: [176](1)
- 18: [165](1)
- 19: [222](1)
- 20: [81](1)
- 21: [200](1)
- 22: [100](1)
- 23: [223](1)
- 24: [117](1)
- 25: [165](1)
- 26: [155](1)
- 27: [44](1)
- 28: [53](1)
- 29: [225](1)
- 30: [124](1)
- 31: [20](1)
- 32: [5, 0, 0, 0, 0, 0, 0, 0](8)
- 33: [104, 101, 108, 108, 111](5)
- 34: [192, 4, 14, 0, 0, 0, 0, 0](8)
- 35: [4, 0, 0, 0, 0, 0, 0, 0](8)
- 36: [57](1)
- 37: [111](1)
- 38: [59](1)
- 39: [111](1)
- 40: [183](1)
- 41: [248](1)
- 42: [249](1)
- 43: [251](1)
- 44: [128](1)
- 45: [174](1)
- 46: [206](1)
- 47: [0](1)
- 48: [81](1)
- 49: [22](1)
- 50: [3](1)
- 51: [173](1)
- 52: [244](1)
- 53: [104](1)
- 54: [15](1)
- 55: [249](1)
- 56: [239](1)
- 57: [112](1)
- 58: [33](1)
- 59: [255](1)
- 60: [66](1)
- 61: [169](1)
- 62: [29](1)
- 63: [66](1)
- 64: [7](1)
- 65: [106](1)
- 66: [231](1)
- 67: [230](1)

Nice, I just tried it and now it works!
Btw, earlier, I try to debug it by decode the instruction data that I got from typescript. Using something like this:
		data := []byte{3, 0, 0, 0, 204, 95, 77, 127, 148, 25, 135, 127, 89, 146, 22, 90, 233, 80, 113, 3, 70, 176, 165, 222, 81, 200, 100, 223, 117, 165, 155, 44, 53, 225, 124, 20, 5, 0, 0, 0, 0, 0, 0, 0, 104, 101, 108, 108, 111, 192, 4, 14, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 57, 111, 59, 111, 183, 248, 249, 251, 128, 174, 206, 0, 81, 22, 3, 173, 244, 104, 15, 249, 239, 112, 33, 255, 66, 169, 29, 66, 7, 106, 231, 230}
		in, err := system.DecodeInstruction([]*solana.AccountMeta{
			&solana.AccountMeta{payerAccount.PublicKey(), true, true},
			&solana.AccountMeta{newSubAccount, true, false},
			&solana.AccountMeta{payerAccount.PublicKey(), true, true},
		}, data)

Then, in this function func (dec *Decoder) decodeBin(rv reflect.Value, opt *option) (err error) {, I see traceEnabled var. Is there any way to activate traceEnabled?

Btw, earlier, I try to debug it by decode the instruction data that I got from typescript. Using something like this:

You can see more examples of how to decode a (whole) transaction from here: https://github.com/gagliardetto/solana-go/issues/15 and https://github.com/gagliardetto/solana-go/issues/19
Also here: https://github.com/gagliardetto/solana-go#parsedecode-an-instruction-from-a-transaction

Then, deep inside system.DecodeInstruction, I found this function func (dec *Decoder) decodeBin(rv reflect.Value, opt *option) (err error) {. There, I see traceEnabled var. Is there any way to activate traceEnabled?

That's some legacy code I need to fix (doesn't really work right now).

authors: gagliardetto pikomonde

repo: gagliardetto/solana-go


text:
[Audit_RV]Suggested Code Improvements

There are a few suggestions to improve the current implementation with regards to more efficient computation, better precision and consistent code style.
Detailed Description

[ ] Storage access is better to be in one place. The following implementation is recommended to move <HubAssetImbalance<T>>::get() into its function body, similar to functions like update_tvl().
https://github.com/galacticcouncil/HydraDX-node/blob/20c8ac6079b8b3d6f16a9f15e9e43353ea6b9abc/pallets/omnipool/src/lib.rs#L553

Context
To have cleaner and more consistent code style.
Audit by Runtime Verification

fixed in #450

source: gharchive/issue

repo: galacticcouncil/HydraDX-node

license_type: permissive

license_source: github-api


text:
Job directories failed to be deleted for failed jobs

I see this error on the Galaxy logs everytime there is a failed job:
galaxy.jobs.runners.kubernetes DEBUG 2022-12-13 12:57:25,734 [pN:job_handler_0,p:8,tN:KubernetesRunner.work_thread-0] (513/gxy-galaxy-dev-skl7v) Terminated at user's request
galaxy.objectstore CRITICAL 2022-12-13 12:57:25,823 [pN:job_handler_0,p:8,tN:KubernetesRunner.work_thread-0] None delete error [Errno 13] Permission denied: 'fontlist-v330.json'

I suspect this is stopping the setup from deleting the job directory.

What is fontlist-v330.json? Do you know where on the filesystem it's located? This looks like an object-store configuration issue maybe? I don't recall having seen this before.

Not sure really... I do see this on the job directories:
galaxy@galaxy-dev-job-0-5cdc9d777f-v6q4q:/galaxy/server/database/jobs_directory/_cleared_contents/000/627/20221214-164444$ ls -ltr
total 72
-rw-r--r-- 1 galaxy root    0 Dec 14 16:44 galaxy_627.o
drwxr-sr-x 2 galaxy root 4096 Dec 14 16:44 inputs
-rwxr-xr-x 1 galaxy root 1109 Dec 14 16:44 tool_script.sh
-rwxr-xr-x 1 galaxy root 5900 Dec 14 16:44 galaxy_627.sh
-rw-r--r-- 1 root   root    0 Dec 14 16:44 memory_statement.log
-rw-r--r-- 1 root   root    2 Dec 14 16:44 __instrument_core_galaxy_slots
-rw-r--r-- 1 root   root    5 Dec 14 16:44 __instrument_core_galaxy_memory_mb
-rw-r--r-- 1 root   root   11 Dec 14 16:44 __instrument_core_epoch_start
drwxr-sr-x 2 root   root 4096 Dec 14 16:44 configs
drwxr-sr-x 2 root   root 4096 Dec 14 16:44 _working
drwxr-sr-x 2 root   root 4096 Dec 14 16:44 _outputs
drwxr-sr-x 2 root   root 4096 Dec 14 16:44 _configs
drwxr-sr-x 2 galaxy root 4096 Dec 14 16:44 outputs
drwxr-sr-x 2 galaxy root 4096 Dec 14 16:44 working
drwxr-sr-x 4 root   root 4096 Dec 14 16:44 home
-rw-r--r-- 1 root   root    4 Dec 14 16:44 galaxy_627.ec
-rw-r--r-- 1 root   root   11 Dec 14 16:44 __instrument_core_epoch_end
drwxr-sr-x 2 root   root 4096 Dec 14 16:44 tmp
-rw-r--r-- 1 galaxy root   62 Dec 14 16:44 galaxy_627.e

I'm guessing that some files owned by root might give some process issues when trying to delete? or this is well controlled?

I don't seem to find that fontlist-v330.json file there within at least.

ahhh, found them:
rm: cannot remove '000/623/20221214-163640/home/.cache/matplotlib/fontlist-v330.json': Permission denied
rm: cannot remove '000/623/20221214-163640/home/.config/matplotlib': Permission denied

they are:
galaxy@galaxy-dev-job-0-5cdc9d777f-v6q4q:/galaxy/server/database/jobs_directory/_cleared_contents$ ls -l 000/623/20221214-163640/home/.cache/matplotlib/fontlist-v330.json
-rw-r--r-- 1 root root 24399 Dec 14 16:36 000/623/20221214-163640/home/.cache/matplotlib/fontlist-v330.json

so how does the tool get to write those files as root? are we missing to pass the 101 user to the jobs for some reason?
so I'm guessing these might be specific to some of the scanpy tools that we use, but I guess they shouldn't be written as root anyways, right:

Final runner setup looks like this in the job_conf.yml:
runners:
  k8s:
    k8s_cleanup_job: never
    k8s_extra_job_envs:
      HDF5_USE_FILE_LOCKING: "FALSE"
    k8s_fs_group_id: "101"
    k8s_galaxy_instance_id: 'galaxy-dev'
    k8s_interactivetools_ingress_annotations: |

      nginx.ingress.kubernetes.io/proxy-body-size: 10G
      nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    k8s_interactivetools_use_ssl: true
    k8s_job_ttl_secs_after_finished: 600
    k8s_namespace: 'default'
    k8s_persistent_volume_claims: |-
      vol-nfs:/galaxy/server/database,vol-nfs/cvmfsclone:/cvmfs/cloud.galaxyproject.org
    k8s_pod_priority_class: 'galaxy-dev-job-priority'
    k8s_pull_policy: IfNotPresent
    k8s_supplemental_group_id: "101"
    k8s_use_service_account: true
    load: galaxy.jobs.runners.kubernetes:KubernetesJobRunner

I think that k8s_cleanup_job this is only for k8s API Job object, not the working directory within the file system.

NFS mount inside the Galaxy job handler looks like this:
10.43.9.166:/export/pvc-4372166d-a268-4763-832b-2a5c1ac66330 on /galaxy/server/database type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.43.9.166,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.43.9.166)

do we need some root squash for the job container perhaps setting 101 on the exports (I'm guessing this is done in the StorageClass setting perhaps)?

Yes ,this sure looks like an issue with that the tool is doing. While most biocontainers tools do run as root, this particular tool appears to be creating files without adding the fs_group_id as an owner. As a result, it's a root only file which galaxy can't clean up.
What if you reconfigure this particular tool in TPV, and force the user id to 101?
Something like:
tools:
  .*scanpy.*:
    params:
       k8s_run_as_user_id: 101

Using k8s_run_as_user_id on the runner sorts the problem. I cannot yet do it on tpv because of the resubmissions issue, hopefully when that works and I can live only on tpv destinations, that could work. Thanks!

repo: galaxyproject/galaxy-helm

license: MIT


text:
Data fetched at docker run to decrease container size

At image build, docker build -t containername -f topics/topicname/docker/Dockerfile ., only the data-library.yaml files are merged, the way it used to, but the data itself is not downloaded.
When you use the normal run command  docker run -p "8080:80" containername the container will run without the example data.
When the command  docker run -p "8080:80" containername -d is used, the corresponding data libraries of that topic will be downloaded and accessible within the training environment.
This reduces the size of the images enormously, enabling them to be uploaded to docker hub or quay.io

if this idea will be implemented, we should also change planemo to generate the new docker files with the new
ADD bin/datalibrarydownload.sh /datalibrarydownload.sh
ENTRYPOINT ["/datalibrarydownload.sh"]

lines

@bgruening What do you think of this implementation? Or would you still prefer to incorporate this function inside the galaxy-stable base image.

@bgruening I checked the building of several topics, and it works like a charm. Only concern: when I wanted to build the containers in windows, I had some line ending problems in the bash scripts. (although on github LF is used ...  ). But that is a windows thingy.. I think it can be merged. Biggest container now is 28 gig for transcriptomics.

@bgruening Done !

Merci!

@bedroesb you might want to inform Frederik about this. He probably wants to put this in some fancy GDoc for ELIXIR.

source: gharchive/pull-request

authors: bedroesb bgruening

repo: galaxyproject/training-material


text:
Ease + doc the use of gh-pages on forks

To fix #567 and make it easy deployment on gh-pages

With #634 and on-the-fly deployment for every PR, I think this PR is not useful anymore


text:
color options for showPass icon and help text for the input

Galio is a great UI library, I love it so far! But there is a scenario in my app where i don't want to use the the default black color of showPass icon and help text in the input(I want these in white).

I have been looking for the props of input component in the documentation but i couldn't find anything on that in the documentation. So I wasn't able to find a way to achieve this using Galio.
If there is a way please let me know. Otherwise it would be nice to provide more props to control the styling of these elements.

You can do like this:
                <Input
                  ...
                  help={
                    this.state.available ? (
                      <Text style={{ color: "orange" }}>
                        {this.state.message}
                      </Text>
                    ) : (
                      <Text style={{ color: "red" }}>{this.state.message}</Text>
                    )
                  }
                  bottomHelp
                 ...

@EtainLove  any solution for the icon color?

It is easy. You need to set iconColor to white
Here is how I did:
<Input
 password
 viewPass
 color="white"
 iconColor="white"
 ...
/>

awesome! it worked thanks.

repo: galio-org/galio


text:
Test of Unterminated comment is not valid

Test of Unterminated comment is not valid since block comments do not have to be terminated PHP.

This test causes error in PHP as expected https://ato.pxeger.com/run?1=m724IKNgwYKlpSVpuhY3zWzsgVyulNTknMSiVI3ikqLM5JL4ksqC1GJbQ01rLn0thYzUnJx8hZKM1KJURUVFLi4rTS4uiG6oITDDAA
Or do you mean something else?

In w3school php compiler it was just a warning.

W3school is probably using PHP 7 or older it seems.

authors: galloj xpavlic


text:
No longer works on new comfy..?

Hey there. Recently I've made an XL model engine and it worked, however, after updating my comfyui to the latest version, I encounter this error regardless of the model base.
TensorRTEngineOriginModelPatcherWrapper_UnetPatch.patch_model() got an unexpected keyword argument 'patch_weights'
Simpliest workflow, nothing custom.

(Full Error Message, just in case)
Error occurred when executing KSampler:
TensorRTEngineOriginModelPatcherWrapper_UnetPatch.patch_model() got an unexpected keyword argument 'patch_weights'
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1369, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1339, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 267, in motion_sample
return orig_comfy_sample(model, noise, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\sample.py", line 93, in sample
real_model, positive_copy, negative_copy, noise_mask, models = prepare_sampling(model, noise.shape, positive, negative, noise_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\sample.py", line 86, in prepare_sampling
comfy.model_management.load_models_gpu([model] + models, model.memory_required([noise_shape[0] * 2] + list(noise_shape[1:])) + inference_memory)
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\model_management.py", line 443, in load_models_gpu
cur_loaded_model = loaded_model.model_load(lowvram_model_memory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\model_management.py", line 303, in model_load
raise e
File "C:\ComfyUI_BLYAT\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\model_management.py", line 299, in model_load
self.real_model = self.model.patch_model(device_to=patch_model_to, patch_weights=load_weights)


text:
New Update can't make

root@filter:~/XDP-Firewall# make
mkdir -p build/
clang -O2 -c -o build/config.o src/config.c
clang -O2 -c -o build/cmdline.o src/cmdline.c
make -C modules/xdp-tools
make[1]: Entering directory '/root/XDP-Firewall/modules/xdp-tools'
sh configure
Found clang binary 'clang' with version 11 (from 'Ubuntu clang version 11.1.0-++20211011094159+1fdec59bffc1-1~exp1~20211011214622.5')
bpftool doesn't support skeleton generation
make[1]: *** [Makefile:49: config.mk] Error 1
make[1]: Leaving directory '/root/XDP-Firewall/modules/xdp-tools'
make: *** [Makefile:75: libxdp] Error 2
root@filter:~/XDP-Firewall# make libxdp
make -C modules/xdp-tools
make[1]: Entering directory '/root/XDP-Firewall/modules/xdp-tools'
sh configure
Found clang binary 'clang' with version 11 (from 'Ubuntu clang version 11.1.0-++20211011094159+1fdec59bffc1-1~exp1~20211011214622.5')
bpftool doesn't support skeleton generation
make[1]: *** [Makefile:49: config.mk] Error 1
make[1]: Leaving directory '/root/XDP-Firewall/modules/xdp-tools'
make: *** [Makefile:75: libxdp] Error 2
root@filter:~/XDP-Firewall#

Hey! What Linux distro are you using when building the program and can you give the output of the following?
# Kernel version
uname -r

# Release file
cat /etc/*-release

I've ran into this error before, but I don't remember what I did to fix it and I'm pretty sure it's related to LibXDP (missing a dependency).
If you want something quickly, you can try checking out the commit before moving to LibXDP which is detailed here. There haven't been many changes to the XDP code itself, so you should be running the same program with the exception of using LibXDP to load the BPF/XDP program.

root@filter:~# uname -r
5.4.0-139-generic
root@filter:~# cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

See https://github.com/xdp-project/xdp-tools/issues/370

After a full reinstall of the VM, I was able to get away with installing kernel image 5.15.0-86-generic and Clang 12. Clang 10 wasn't enough, but it's possible Clang 11 may work (not tested).
I didn't need to build bpftool from source as well, so I installed the kernel 5.15.0-86-generic tools as suggested in @pettai's linked issue :smile:
Here are the commands I performed from the start of the vanilla VM:
# Update and upgrade packages
sudo apt update
sudo apt upgrade

# Install kernel 5.15.0-86-generic and its tools for bpftool
sudo apt install -y linux-image-5.15.0-86-generic linux-headers-5.15.0-86-generic linux-tools-5.15.0-86-generic

# Reboot to load new kernel
sudo reboot

# Confirm kernel version
uname -r

# Install dependencies for XDP Firewall
sudo apt install -y libconfig-dev llvm clang libelf-dev build-essential
sudo apt install -y libpcap-dev m4 gcc-multilib

# Install Clang 12
sudo apt install -y clang-12

# Update primary clang binaries to use version 12
sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-12 100
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-12 100

# Confirm Clang version
clang -v

# Replace LLVM/LLC 10 with version 14 (otherwise an error occurs when building).
# Note - I'm not sure what to use with `update-alternatives` since `llc` and `llvm` doesn't work with it. So instead, I manually remove the old symbolic link and create a new one pointed towards LLVM 12.
sudo rm -f /usr/bin/llc
sudo ln -s /usr/lib/llvm-12/bin/llc /usr/bin/llc

# Confirm LLVM/LLC version
llc --version

# Clone XDP Firewall and its sub-modules
git clone --recursive https://github.com/gamemann/XDP-Firewall.git

# Change directory
cd XDP-Firewall

# Make and install LibXDP
make libxdp

# Make XDP Firewall
make

# Install Firewall
sudo make install

# Edit /etc/xdpfw/xdpfw.conf if needed (e.g. using different interface name)

# Run Firewall
sudo xdpfw

When building, I do receive a warning, but I believe it's harmless.
clang -lconfig -lelf -lz -I modules/xdp-tools/lib/libbpf/src -I /usr/include -I /usr/local/include -o build/xdpfw modules/xdp-tools/lib/libbpf/src/staticobjs/bpf_prog_linfo.o modules/xdp-tools/lib/libbpf/src/staticobjs/bpf.o modules/xdp-tools/lib/libbpf/src/staticobjs/btf_dump.o modules/xdp-tools/lib/libbpf/src/staticobjs/btf.o modules/xdp-tools/lib/libbpf/src/staticobjs/gen_loader.o   modules/xdp-tools/lib/libbpf/src/staticobjs/hashmap.o modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf_errno.o modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf_probes.o  modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf.o modules/xdp-tools/lib/libbpf/src/staticobjs/linker.o modules/xdp-tools/lib/libbpf/src/staticobjs/netlink.o  modules/xdp-tools/lib/libbpf/src/staticobjs/nlattr.o modules/xdp-tools/lib/libbpf/src/staticobjs/relo_core.o modules/xdp-tools/lib/libbpf/src/staticobjs/ringbuf.o modules/xdp-tools/lib/libbpf/src/staticobjs/str_error.o modules/xdp-tools/lib/libbpf/src/staticobjs/strset.o modules/xdp-tools/lib/libbpf/src/staticobjs/usdt.o modules/xdp-tools/lib/libxdp/sharedobjs/xsk.o modules/xdp-tools/lib/libxdp/sharedobjs/libxdp.o build/config.o build/cmdline.o src/xdpfw.c
In file included from src/xdpfw.c:20:
modules/xdp-tools/lib/libbpf/src/libbpf.h:70:54: warning: declaration of 'enum bpf_link_type' will not be visible outside of this function [-Wvisibility]
LIBBPF_API const char *libbpf_bpf_link_type_str(enum bpf_link_type t);
                                                     ^
1 warning generated.

I hope this helps and if you have any questions or run into other issues, please let me know!

~/XDP-Firewall# make && sudo make install
mkdir -p build/
clang -O2 -c -o build/config.o src/config.c
clang -O2 -c -o build/cmdline.o src/cmdline.c
make -C libbpf/src
make[1]: Entering directory '/root/XDP-Firewall/libbpf/src'
make[1]: Leaving directory '/root/XDP-Firewall/libbpf/src'
mkdir -p build/
clang -lconfig -lelf -lz -I libbpf/src -o build/xdpfw libbpf/src/staticobjs/bpf_prog_linfo.o libbpf/src/staticobjs/bpf.o libbpf/src/staticobjs/btf_dump.o libbpf/src/staticobjs/btf.o libbpf/src/staticobjs/hashmap.o libbpf/src/staticobjs/libbpf_errno.o libbpf/src/staticobjs/libbpf_probes.o libbpf/src/staticobjs/libbpf.o libbpf/src/staticobjs/netlink.o libbpf/src/staticobjs/nlattr.o libbpf/src/staticobjs/str_error.o  libbpf/src/staticobjs/xsk.o build/config.o build/cmdline.o src/xdpfw.c
clang: error: no such file or directory: 'libbpf/src/staticobjs/xsk.o'
make: *** [Makefile:33: xdpfw] Error 1
any ideas ? its the b54c466
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
with pve installed

@GermanKingYT Hey! It appears you have mismatched Clang and LLVM versions. Can you provide the output of the following commands please?
llc --version
clang --version

Can you also provide the Linux distro you're using and its release version/kernel? You can do that by providing the outputs of the following commands.
cat /etc/*-release

uname -a

I believe it'll be best if we started a separate issue related to this issue as well since it looks like the error you received is unrelated to the initial issue's error. I plan on closing this one soon.

i use old kernel,
clang 11
and update xdp-tools 1.4.1
but
clang -lconfig -lelf -lz -I modules/xdp-tools/lib/libbpf/src -I /usr/include -I /usr/local/include -o build/xdpfw modules/xdp-tools/lib/libbpf/src/staticobjs/bpf_prog_linfo.o modules/xdp-tools/lib/libbpf/src/staticobjs/bpf.o modules/xdp-tools/lib/libbpf/src/staticobjs/btf_dump.o modules/xdp-tools/lib/libbpf/src/staticobjs/btf.o modules/xdp-tools/lib/libbpf/src/staticobjs/gen_loader.o   modules/xdp-tools/lib/libbpf/src/staticobjs/hashmap.o modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf_errno.o modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf_probes.o  modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf.o modules/xdp-tools/lib/libbpf/src/staticobjs/linker.o modules/xdp-tools/lib/libbpf/src/staticobjs/netlink.o  modules/xdp-tools/lib/libbpf/src/staticobjs/nlattr.o modules/xdp-tools/lib/libbpf/src/staticobjs/relo_core.o modules/xdp-tools/lib/libbpf/src/staticobjs/ringbuf.o modules/xdp-tools/lib/libbpf/src/staticobjs/str_error.o modules/xdp-tools/lib/libbpf/src/staticobjs/strset.o modules/xdp-tools/lib/libbpf/src/staticobjs/usdt.o modules/xdp-tools/lib/libxdp/sharedobjs/xsk.o modules/xdp-tools/lib/libxdp/sharedobjs/libxdp.o build/config.o build/cmdline.o src/xdpfw.c
In file included from src/xdpfw.c:19:
modules/xdp-tools/lib/libbpf/src/bpf.h:471:60: warning: declaration of 'struct bpf_link_info' will not be visible outside of this function [-Wvisibility]
LIBBPF_API int bpf_link_get_info_by_fd(int link_fd, struct bpf_link_info *info, __u32 *info_len);
^
In file included from src/xdpfw.c:20:
modules/xdp-tools/lib/libbpf/src/libbpf.h:70:54: warning: declaration of 'enum bpf_link_type' will not be visible outside of this function [-Wvisibility]
LIBBPF_API const char *libbpf_bpf_link_type_str(enum bpf_link_type t);
^
2 warnings generated.
/usr/bin/ld: modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf.o: in function elf_find_func_offset_from_archive': /root/XDP-Firewall/modules/xdp-tools/lib/libbpf/src/libbpf.c:10893: undefined reference to zip_archive_open'
/usr/bin/ld: /root/XDP-Firewall/modules/xdp-tools/lib/libbpf/src/libbpf.c:10900: undefined reference to zip_archive_find_entry' /usr/bin/ld: /root/XDP-Firewall/modules/xdp-tools/lib/libbpf/src/libbpf.c:10934: undefined reference to zip_archive_close'
/usr/bin/ld: /root/XDP-Firewall/modules/xdp-tools/lib/libbpf/src/libbpf.c:10934: undefined reference to zip_archive_close' /usr/bin/ld: /root/XDP-Firewall/modules/xdp-tools/lib/libbpf/src/libbpf.c:10934: undefined reference to zip_archive_close'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:59: xdpfw] Error 1
how to fix? i can't change kernel but some interface issue

@namecloudz The errors you're receiving are related to ZIP functions and references missing in the version of LibBPF you're using. The latest version of LibBPF appears to compile a new zip.c file into a shared/static object while this firewall currently uses an outdated version of LibBPF that does not. Since we still use raw LibBPF functionality in the XDP Firewall loader (which I should definitely look into eliminating when I have the time so that we only rely on xdp-tools independently), we manually link the LibBPF's static objects here. The new ZIP object is not linked here which is causing the error you're receiving most likely.
There are two options that will most likely resolve this issue.
The first option is downgrading the xdp-tools submodule to commit 81ea740 and the libbpf submodule (within the xdp-tools submodule) to commit a032540 (if you reinitialize the XDP firewall repository with the --recursive flag via git clone --recursive https://github.com/gamemann/XDP-Firewall.git or use git submodule update --init in a vanilla repository , it should automatically checkout these commits that compile properly for XDP Firewall.
The second option is to edit the Makefile and add $(LIBBPFSRC)/staticobjs/zip.o and any other new static objects to the LIBBPFOBJS variable here so that they are linked.
I hope this helps!

I've updated the project's xdp-tools submodule to its latest commit, cae9c91, which included linking the zip static object from libbpf. Try pulling the recent changes and building again. libbpf is now on commit fbd60db within the xdp-tools submodule.
I tested these changes in a dev environment and they appear to be working fine. The GitHub workflow I made also succeeded in building the project.

thank you,
and i will try test in 22.04
libbpf: elf: skipping unrecognized data section(7) .xdp_run_config
libbpf: elf: skipping unrecognized data section(8) xdp_metadata
libbpf: elf: skipping unrecognized data section(23) .eh_frame
libbpf: elf: skipping relo section(24) .rel.eh_frame for section(23) .eh_frame
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: prog 'xdp_pass': BPF program load failed: Invalid argument
libbpf: prog 'xdp_pass': failed to load: -22
libbpf: failed to load object '/usr/local/lib/bpf/xdp-dispatcher.o'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
Loaded XDP program on mode DRV/native.

@namecloudz You're welcome! It looks like it's working from the output you gave. Can you confirm this? If so, would it be alright to close this issue?

Edit I just relooked at xdp-tools's code and wanted to note that we won't be able to rely on xdp-tools separately since xdp-tools doesn't include the BPF functionality the XDP Firewall loader requires. So we'll have to still link libbpf in the XDP Firewall loader separately for now, but we do use the LibBPF library that is cloned inside of the xdp-tools submodule. This doesn't impact the solutions above and they should still work fine, I just wanted to clarify this in the case anybody wonders why we can't rely on xdp-tools for all BPF functionality in the project.

What you mean is that XDP Firewall can't be built nor use the packages version of xdp-tools at run-time?

Edit I just relooked at xdp-tools's code and wanted to note that we won't be able to rely on xdp-tools separately since xdp-tools doesn't include the BPF functionality the XDP Firewall loader requires. So we'll have to still link libbpf in the XDP Firewall loader separately for now, but we do use the LibBPF library that is cloned inside of the xdp-tools submodule. This doesn't impact the solutions above and they should still work fine, I just wanted to clarify this in the case anybody wonders why we can't rely on xdp-tools for all BPF functionality in the project.

What you mean is that XDP Firewall can't be built nor use the packages version of xdp-tools at run-time?

The firewall does use xdp-tools, but I was initially hoping that we could eliminate the need of using LibBPF directly in the xdpfw.c file and replacing it with functionality from xdp-tools. However, xdp-tools doesn't provide the BPF functionality we'd need from what I've seen (e.g. adding/deleting entries from a BPF map). Therefore, we still need to link the LibBPF objects here in this firewall's Makefile.
For example, if we didn't need to link the LibBPF objects directly in our Makefile, the errors from this reply probably wouldn't have occurred since xdp-tools would have handled linking the LibBPF objects which would have included the newer zip.o object file.
I hope this clarifies what I meant, but if not, feel free to open a new issue (this one was closed, so I don't think you can reply)! 😄

authors: GermanKingYT gamemann namecloudz pettai

repo: gamemann/XDP-Firewall


text:
🛑 Friends API Endpoint is down

In 2356303, Friends API Endpoint (https://friends.roblox.com/v1/metadata) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Friends API Endpoint is back up in bb86e3d.


text:
🛑 Games API Endpoint is down

In ae90e84, Games API Endpoint (https://games.roblox.com) was down:

HTTP code: 503
Response time: 730 ms

Resolved: Games API Endpoint is back up in a4e1d14 after 25 minutes.


text:
Support for TeamDrive ?

hello it would be great if uploading to teamdrive instead of personal drive

I'm sorry, but I don't have a teamdrive, only a personal drive, so I can't test it.

u can easly get on for free
https://td.msgsuite.workers.dev/

authors: gaowanliang xVoldx

repo: gaowanliang/DownloadBot


text:
Persist secrets required by extension controllers so that they can be used during Control Plane Migration

What would you like to be added:
Extend the ShootState, ShootState Sync Controller and the Status of Extension CRs so that secret resources required by extension controllers can be persisted in the ShootState and migrated from the source seed to the destination seed.
Why is this needed:
Currently extension controllers can save their state in the status.state field of the corresponding extension resource. This is synced by the ShootState Sync Controller to the ShootState and used to restore the state of the extension controller on the destination seed during control plane migration. However, if the extension controller has to persist a secret, it cannot be directly saved to the status.state as it contains confidential information.
Proposal:
To handle such cases an additional field is added to the status of extension CRs similar to Pod Volumes - status.Secrets (if it is required to handle other resources similarly then we could change the name to e.g. status.Resources)

The ShootState Sync Controller will read the new field, copy the data of the corresponding secrets and add them to the ShootState resource.
The control plane migration's restoration phase has an additional step which gets all such secrets from the ShootState and deploys them on the Seed in the Shoot's namespace.

Here's an example with the Extension CR used by the certificate extension controller
apiVersion: extensions.gardener.cloud/v1alpha1
kind: Extension
metadata:
  name: "extension-certificate-service"
  namespace: shoot--project--abc
spec:
  type: shoot-cert-service
  providerConfig:
    apiVersion: shoot-cert-service.extensions.config.gardener.cloud/v1alpha1
    kind: Configuration
    issuerName: gardener
    restrictIssuer: true # restrict issuer to any sub-domain of shoot.spec.dns.domain (default)
    acme:
      email: john.doe@example.com
      server: https://acme-v02.api.letsencrypt.org/directory
      privateKeyRef:
        name: private-key
status:
  observedGeneration: 2
  state:
    # some data
    privateKeySecretRef: my-lets-encrypt-private-key
  secretReferences:
  - name: private-key
     secretName: my-lets-encrypt-private-key

Thanks for opening the issue!
We should be in line with https://github.com/gardener/gardener/issues/2321#issuecomment-628417974

/assign @stoyanr

The example should actually look as follows (in sync with #2321):
apiVersion: extensions.gardener.cloud/v1alpha1
kind: Extension
metadata:
  name: "extension-certificate-service"
  namespace: shoot--project--abc
spec:
  type: shoot-cert-service
status:
  observedGeneration: 2
  state:
    # some data
    privateKeyRef: private-key
  resources:
  - name: private-key
    resourceRef:
      apiVersion: v1
      kind: Secret
      name: my-secret

authors: plkokanov rfranzke stoyanr

repo: gardener/gardener


text:
Allow multiple AZs in OpenStack w/o dedicated subnets

What this PR does / why we need it:
It allows to specify multiple availability zones for OpenStack environment without specifying multiple worker CIDRs.
Which issue(s) this PR fixes:
Fixes https://github.com/gardener/gardener-extensions/issues/184
Special notes for your reviewer:
Release note:

tested this change in my openstack environment. shoot nodes were successfully created in different AZs.

Found one bug. If the gardener contains the shoots with the multiple AZs, e.g.
...
    openstack:
      networks:
        nodes: 198.18.1.0/24
        pods: 100.94.0.0/16
        services: 100.73.0.0/16
        workers:
        - 198.18.1.0/24
      workers:
      - name: cpu-worker
        machineType: m1.xlarge
        maxSurge: 50%
        maxUnavailable: 0
        autoScalerMin: 3
        autoScalerMax: 5
      zones: ["region1a","region1b"]
...

then dashboard drives crazy. It doesn't allow to create secrets, it doesn't allow to create clusters, and it doesn't list machineTypes.

Dashboard has to be fixed for this PR.

dashboard issue is created https://github.com/gardener/dashboard/issues/432
thanks @kayrus for reporting

authors: kayrus petersutter


text:
Add missing links to reconcilers in documentation

How to categorize this PR?

/area documentation
/kind enhancement
What this PR does / why we need it:
Add missing links to reconcilers in gardener-controller-manager and gardenlet documentation. Similar to https://github.com/gardener/gardener/blob/master/docs/concepts/operator.md#controllers.
Release note:

NONE

/test pull-gardener-e2e-kind-operator

authors: acumino rfranzke


text:
local-path provisioner with support for volume resizing

How to categorize this PR?

/area auto-scaling
/area control-plane
/kind enhancement
/kind poc
What this PR does / why we need it:
This PR adds support for running the local dev environment with local-path provisioner which supports volume resizing.
The default StorageClass (named standard) in the local setup currently defaults to hostPath (this is what rancher.io/local-path provisioner defaults to), and hostPath does not expose metrics via kubelet.
In order to expose the various kubelet_volume_stats_* metrics via kubelet in the local setup this PR also configures the default StorageClass to use local volume instead of hostPath, since localVolume exposes metrics.
This configuration is done on the StorageClass object provided by rancher/local-path provisioner by configuring the defaultVolumeType: local setting.
Additionally the hack/kind-up.sh script has been modified to support a new option called --with-lpp-resize-support, which when set to true will deploy and configure the local path provisioner with volume resize support. The PVC resize support in rancher/local-path-provisioner is currently under review and is scheduled for next release around May, 2024. Until this feature is merged we will use the the fork providing this feature, and later once merged into the upstream project we can drop the code from kind-up.sh.
Why is all this needed?
The end goal is to integrate PVC auto{scaler,resizer} into Gardener, which would allow PVCs to be auto-resized when they reach a certain threshold. The primary target during the first iteration of this integration are the observability components (Vali, Plutono and Prometheus), but later this could be leveraged by other PVCs as well.
In order to enable PVC auto-resizer we first need to start collecting metrics about the utilization of the PVCs. By enabling these metrics in the local dev setup we would be able to easily test the existing PVC auto-scalers and test their integration with Gardener.
Which issue(s) this PR fixes:
Part of https://github.com/gardener/gardener/issues/4173
Special notes for your reviewer:
Running make kind-up will create a local kind cluster just like before, with the only exception that now the default StorageClass (named standard) will default to local instead of hostPath.
In order to create a local dev environment with local-path provisioner, which supports volume resizing execute this command instead.
DEV_SETUP_WITH_LPP_RESIZE_SUPPORT=true make kind-up

Release note:

/assign

/assign

Thanks for the PR!
I had the following question. It is not very clear for a reviewer why the PR is in draft state. Will this PR be in draft state until https://github.com/rancher/local-path-provisioner/pull/350 is merged and released in the local-path-provisioner and we adopt the new local-path-provisioner version? If that's the case, we could extract the part that fixes the volume metrics in the local setup to a dedicated PR and proceed with it because that part does not depend on the upstream local-path-provisioner release.
FYI @istvanballok (as I recall discussing with you that volume metrics don't work in local setup).

Thanks for the PR!
I had the following question. It is not very clear for a reviewer why the PR is in draft state. Will this PR be in draft state until rancher/local-path-provisioner#350 is merged and released in the local-path-provisioner and we adopt the new local-path-provisioner version? If that's the case, we could extract the part that fixes the volume metrics in the local setup to a dedicated PR and proceed with it because that part does not depend on the upstream local-path-provisioner release.
FYI @istvanballok (as I recall discussing with you that volume metrics don't work in local setup).

Hey @ialidzhikov ,
I left it in draft, because I wanted to test a few more things with the local setup, before calling it ready :)
Just finished with the tests and things look good from my side, so removing the draft label.
Here's also a screenshot of the kubelet_volume_stats_available_bytes metric from garden/prometheus-cache service.

Thanks!

Please hold this one for now. I just realized that I need to add kustomize to tools.

Added kustomize tool dependency. Should be good now.
cc: @ialidzhikov @nickytd @istvanballok

PR rebased.

/approve

authors: dnaeon ialidzhikov plkokanov


text:
Specify nonces for feePayer and zkApp to allow multiple tx in a block

Need to track these offline, but can then sign and broadcast the transactions and include multiple in a block e.g. https://github.com/garethtdavies/zkAppPool/blob/main/batch-signed-tx.ts#L96

Only need the feePayer as zkApp incrementNonce is False.

As the index is checking against an on-chain state variable, this is limited to 1 per block.

Added fee payer nonce via #22

Hopefully closed by #23

repo: garethtdavies/zkAppPool


text:
Cross compile to Scala.js

Thanks for this great library.
I kindly request to publish it cross-compiled to Scala.js so that it can be used in hybrid (Scala/Scala.js) projects.
Thank you!

Sorry for delay in getting to this.  I initially missed the issue.
I have republished the 0.6.1-SNAPSHOT with cross compile versions for JVM and Scala.js.
Let me know if it doesn't work.
Thanks.

HI @garyKeorkunian , sorry I couldn't test until now.
It seems to working just fine, thank you very much!

authors: garyKeorkunian rpiaggio

repo: garyKeorkunian/squants

license_source: bigquery


text:
Change for Support bam-file/bai-file list as input

Little changes in .wdl and .json to support the use of a file list for .bam- and bai-input files instead of an array.

actual GATK-Docker container version added

Hi Selonka,
Thanks for contributing to improve the wdl, I’ll pass on the suggestion to the developers.

Hi Selonka,
This PR will be closed, if you would like see the changes added to the workflow please post here:
https://github.com/broadinstitute/gatk/tree/master/scripts/cnv_wdl/somatic

authors: Selonka bshifaw


text:
Proxy configuration lost while redirect process

Hi,
I'm stuck with a simulation building an order on internal web application (internal network) and ending with final step using paypal express-checkout sand box (external network behind http proxy).
The typical case to reproduce the problem is simple :

configure protocol with right proxy settings to access to external network
exec scenario request to internal server returning redirect (302) response to external network ( .disablefollowRedirect not called)
=> connexion timeout is raised

When the request is splitted using .disableFollowRedirect + location header capture + exec of redirection from location capture : all works well.
In AsyncHandlerActor class, processResponse function and inner redirectRequest function :
the originalRequest.getProxyServer used to build redirect request is null if originalRequest did not use proxy configuration previously.
Environment:

Windows 7 64 bits
Maven project (archetype gatling)
Eclipse IDE
Gatling 2.1.7 and Gatling 2.2.0-M2

Tests cases
Gatling Test :
val httpProtocol = http
		.baseURL("http://localhost:8080")
		.inferHtmlResources(BlackList(""".*\.js""", """.*\.css""", """.*\.gif""", """.*\.jpeg""", """.*\.jpg""", """.*\.ico""", """.*\.woff""", """.*\.(t|o)tf""", """.*\.png"""), WhiteList())
		.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
		.acceptEncodingHeader("gzip, deflate")
		.acceptLanguageHeader("fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3")
		//.connectionHeader("keep-alive")
		.userAgentHeader("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0")
    .proxy(Proxy("proxy", 8080, 8080))
    .noProxyFor("localhost")
    .disableWarmUp

  // Bug case (non proxyfied request + internal proxyfied redirect : ko)
	val scn = scenario("Request_to local server (without proxy) with redirect to paypal through proxy (followredirect -> bug)")
		.exec(http("Request_to local server with redirect to paypal followRedirect=true")
        .get("/redirect")
        .check(status.is(200)))

  // splitted to skip bug (non proxyfied request + proxyfied redirect : ok)
  val scn2 = scenario("Request_to local server (without proxy) with redirect to paypal through proxy (disableFollowRedirect -> manual -> ok)")
    .exec(http("Request_to local server with redirect to paypal disableFollowRedirect=true")
            .get("/redirect")
            .disableFollowRedirect
            .check(header("Location").saveAs("redirect"),
                status is 302
            )
          )
    .exec(http("Redirect simulation to paypal")
          .get("${redirect}")
          .check(status is 200)
    )

  // simple check on paypal redirect (proxyfied request + proxyfied redirect : ok)
  val scn3 = scenario("Request_to paypal through proxy with redirect to paypal ")
    .exec(http("Request_to paypal followRedirect=true")
        .get("https://www.sandbox.paypal.com/")
        .check(status.is(200)))

	setUp(scn.inject(atOnceUsers(1)).protocols(httpProtocol),
        scn2.inject(atOnceUsers(1)).protocols(httpProtocol),
        scn3.inject(atOnceUsers(1)).protocols(httpProtocol)
        )

Simple springboot webapp simulation
package com.boulanger.simpleserver;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;

@Controller
@EnableAutoConfiguration
public class Application {

	@RequestMapping("/")
	@ResponseBody
	String home() {
		return "Hello World!";
	}

	@RequestMapping(value = "/redirect", method = RequestMethod.GET)
	public String redirect() {
	      return "redirect:https://www.sandbox.paypal.com/";
	}

	public static void main(String[] args) throws Exception {
		SpringApplication.run(Application.class, args);
	}

}

Laurent

Hi,
Don't you want to try to contribute the fix? You're almost there, it's just a matter of using the http conf that's passed in the tx :)

Hi Stephane,
I am a beginner with Scala ( and its environment ) and the architecture of
your project, and totally new to SBT and with very little time to work on
this problem.
I would like to fix it ( good starting point to learn scala ), but no idea
when .
Laurent
2015-08-21 8:07 GMT+02:00 Stephane Landelle notifications@github.com:

Hi,
Don't you want to try to contribute the fix? You're almost there, it's
just a matter of using the http conf that's passed in the tx :)
—
Reply to this email directly or view it on GitHub
https://github.com/gatling/gatling/issues/2791#issuecomment-133298400.

Rule is:

use original request proxy if it was defined and redirect targets the same host
otherwise, use protocol proxy
in any case, honor proxy exceptions

authors: lolodesbois slandelle

repo: gatling/gatling


text:
No logs after running the first simulation from list

Gatling version: 3.2.0
Hi! I have a case when I need to filter simulation by custom attributes (custom annotation with specific value, a list of names of simulation to launch in args, etc.), so I wrote a code which scans classpath, making a subset of simulations with predicate. And after that I wrote another piece of code, which launches simulations one by one:
def runMultipleSimulations(): Unit = {       simulations.filter(simulation => {           val doesCurrentSimulationInArgs = args.contains(simulation.getCanonicalName)           logger.debug(s"Simulation ${simulation.getCanonicalName} is in args $doesCurrentSimulationInArgs")           doesCurrentSimulationInArgs}).foreach(launchSimulation)  }
def launchSimulation(simulation: Class[Simulation]): Unit = {
  val props = new GatlingPropertiesBuilder
  props.simulationClass(simulation.getCanonicalName)
  println(s"##teamcity[testStarted name='${simulation.getSimpleName}']")
  Gatling.fromMap(props.build) match {
    case 0 => println(s"##teamcity[message text='Simulation is successful' status='NORMAL']")
    case 1 => println(s"##teamcity[testIgnored name='${simulation.getSimpleName}' message='Simulation has been ignored because of invalid parameters']")
    case 2 => println(s"##teamcity[testFailed name='${simulation.getSimpleName}' message='Simulation has been failed by assertion']")
  }
  println(s"##teamcity[testFinished name='${simulation.getSimpleName}']")
}

All seems OK - simulations run sequentially, reports are generated, but with an issue - all subsequent simulations but first brings to STDOUT only output from io.gatling.core.stats.writer.ConsoleSummary  object, logs from StrictLogging trait's logger  are not passed to STDOUT at all.
I've investigated it a bit and found something suspicious in io.gatling.app.Gatling in private method private[app] def start(overrides: ConfigOverrides, selectedSimulationClass: SelectedSimulationClass) with very confusing piece of code:
} finally {       val factory = LoggerFactory.getILoggerFactory       try {         factory.getClass.getMethod("stop").invoke(factory)       } catch {         case _: NoSuchMethodException => //Fail silently if a logging provider other than LogBack is used.         case NonFatal(ex)             => logger.warn("Logback failed to shutdown.", ex)       }     }
What is the sense to do away with the Logback logging engine after running a single simulation? io.gatling.app.Gatling methods fromArgs/fromMap are designed as one-shot actions?

Gatling’s architecture expects 1 fresh JVM per simulation so we make sure to free resources and always start with fresh ones.

authors: Tikani slandelle


text:
Can't find DART7.0.0 under Ubuntu 22.04.3

......
Looking for gz-plugin2 - found

 Looking for gz-utils2 -- found version 2.2.0
 Looking for gz-utils2 - found

 Looking for EIGEN3 - found

 Looking for sdformat13 -- found version 13.6.0
 Searching for dependencies of sdformat13
 Checking for module 'tinyxml2'
   Found tinyxml2, version 9.0.0
 Looking for gz-math7 -- found version 7.3.0
 Looking for gz-utils2 -- found version 2.2.0
 Looking for sdformat13 - found

 Could NOT find DART (missing: DART_DIR)
 Looking for DART - not found

 Looking for GzBullet - found

 -------------------------------------------

 CMake Warning at /usr/local/share/cmake/gz-cmake3/cmake3/GzConfigureBuild.cmake:68 (message):
    CONFIGURATION WARNINGS:
    -- Skipping component [dartsim]: Missing dependency [DART] (Components: collision-bullet, collision-ode, utils, utils-urdf).
       ^~~~~ Set SKIP_dartsim=true in cmake to suppress this warning.

 Call Stack (most recent call first):
   CMakeLists.txt:106 (gz_configure_build)

 -- Searching for host SSE information
......

Desired behavior
Expecting gz-physics to build with DART
Alternatives considered
All cmake files that can be found under /usr/local/share/dart7/cmake .
➜  cmake pwd
/usr/local/share/dart7/cmake
➜  cmake ls
DART7Config.cmake                          dart_commonTargets.cmake                  DARTFindassimp.cmake  DARTFindNLOPT.cmake           DARTFindurdfdom.cmake          dart_mathTargets.cmake                  dart_simulationComponent.cmake        FindODE.cmake
DART7ConfigVersion.cmake                   dart_commonTargets-release.cmake          DARTFindBullet.cmake  DARTFindoctomap.cmake         dart_guiComponent.cmake        dart_mathTargets-release.cmake          dart_simulationTargets.cmake          Findtinyxml2.cmake
dart7-external-odelcpsolver.cmake          dart_dynamicsComponent.cmake              DARTFindccd.cmake     DARTFindODE.cmake             dart_guiTargets.cmake          dart_optimizationComponent.cmake        dart_simulationTargets-release.cmake
dart7-external-odelcpsolver-release.cmake  dart_dynamicsTargets.cmake                DARTFindEigen3.cmake  DARTFindOpenCL.cmake          dart_guiTargets-release.cmake  dart_optimizationTargets.cmake          Findassimp.cmake
dart_collisionComponent.cmake              dart_dynamicsTargets-release.cmake        DARTFindfcl.cmake     DARTFindOpenGL.cmake          dart_ioComponent.cmake         dart_optimizationTargets-release.cmake  Findccd.cmake
dart_collisionTargets.cmake                dart_external-imguiComponent.cmake        DARTFindfmt.cmake     DARTFindOpenSceneGraph.cmake  dart_ioTargets.cmake           dart_physicsComponent.cmake             Findfcl.cmake
dart_collisionTargets-release.cmake        dart_external-imguiTargets.cmake          DARTFindGLUT.cmake    DARTFindspdlog.cmake          dart_ioTargets-release.cmake   dart_physicsTargets.cmake               FindIPOPT.cmake
dart_commonComponent.cmake                 dart_external-imguiTargets-release.cmake  DARTFindIPOPT.cmake   DARTFindtinyxml2.cmake        dart_mathComponent.cmake       dart_physicsTargets-release.cmake       FindNLOPT.cmake
➜  cmake

Implementation suggestion
cmake issue??
Additional context
What's in current CMakeLists.txt
......
# Find dartsim for the dartsim plugin wrapper
gz_find_package(DART
  COMPONENTS
    collision-bullet
    collision-ode
    utils
    utils-urdf
  CONFIG
  VERSION 6.9
  REQUIRED_BY dartsim
  PKGCONFIG dart
  PKGCONFIG_VER_COMPARISON >=)
......

Can anybody help please???

I think that we are looking for DART with the 6 major version because there hasn't been an official tagged release of the 7.0.0 versions yet.  It should work with anything in the 6.x.x track, which is up to 6.13 these days (https://github.com/dartsim/dart/releases)

where and how to speicify/define GZ_SIM_PHYSICS_ENGINE_PATH ???

This is an environment variable.  You can set it with
export GZ_SIM_PHYSICS_ENGINE_PATH=/path/you/want

repo: gazebosim/gz-physics


text:
Fix Windows warnings

Signed-off-by: Ian Chen ichen@osrfoundation.org
🦟 Bug fix
Summary
Fixing new Windows warnings that popped up recently.
Checklist

[x] Signed all commits for DCO
[ ] Added tests
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if the library has them)
[ ] codecheck passed (See contributing)
[ ] All tests passed (See test coverage)
[ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

this didn't fix the warnings. Looks like the warnings are coming from ign-transport

repo: gazebosim/gz-sensors


text:
🛑 Images - GBIF Portugal is down

In 58f4161, Images - GBIF Portugal (https://imagens.gbif.pt/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Images - GBIF Portugal is back up in 4e33ee9 after 1 hour, 44 minutes.

repo: gbifatlas/status


text:
cpus in shortest_path() does not catch all cores.

Contributing guidelines

[X] I understand the contributing guidelines

Documentation

[X] My problem is not addressed by the documentation or examples

Existing issues

[X] My problem does not appear in an existing issue

What operating system and Python version are you using?
Centos 7, Python 10
What OSMnx version are you using?
1.5.0
Environment packages and versions
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
affine                    2.4.0              pyhd8ed1ab_0    conda-forge
anyio                     3.7.0              pyhd8ed1ab_1    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h5764c6d_3    conda-forge
asttokens                 2.2.1                    pypi_0    pypi
attrs                     23.1.0             pyh71513ae_1    conda-forge
aws-c-auth                0.6.28               h435f46f_9    conda-forge
aws-c-cal                 0.6.0                h93469e0_0    conda-forge
aws-c-common              0.8.23               hd590300_0    conda-forge
aws-c-compression         0.2.17               h862ab75_1    conda-forge
aws-c-event-stream        0.3.0                h1e03375_8    conda-forge
aws-c-http                0.7.10               h9ab9c9b_2    conda-forge
aws-c-io                  0.13.27              h3870b5a_0    conda-forge
aws-c-mqtt                0.8.13               hd4f18eb_5    conda-forge
aws-c-s3                  0.3.11               hfaf190a_2    conda-forge
aws-c-sdkutils            0.1.11               h862ab75_1    conda-forge
aws-checksums             0.1.16               h862ab75_1    conda-forge
aws-crt-cpp               0.20.2              h42d1850_14    conda-forge
aws-sdk-cpp               1.10.57             he0ca162_15    conda-forge
babel                     2.12.1             pyhd8ed1ab_1    conda-forge
backcall                  0.2.0                    pypi_0    pypi
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
blosc                     1.21.4               h0f2a231_0    conda-forge
bokeh                     3.2.0              pyhd8ed1ab_0    conda-forge
boost-cpp                 1.78.0               h6582d0a_3    conda-forge
branca                    0.6.0              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotli-python             1.0.9           py310hd8f1fbe_9    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
cairo                     1.16.0            hbbf8b49_1016    conda-forge
certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
cfitsio                   4.2.0                hd9d235c_0    conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
click                     8.1.3           unix_pyhd8ed1ab_2    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.7.2              pyhd8ed1ab_1    conda-forge
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
comm                      0.1.3                    pypi_0    pypi
contourpy                 1.1.0           py310hd41b1e2_0    conda-forge
curl                      8.1.2                h409715c_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.0          py310h5764c6d_1    conda-forge
dask                      2023.6.1           pyhd8ed1ab_0    conda-forge
dask-core                 2023.6.1           pyhd8ed1ab_0    conda-forge
dask-geopandas            0.3.1              pyhd8ed1ab_0    conda-forge
debugpy                   1.6.7                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
dill                      0.3.6              pyhd8ed1ab_1    conda-forge
distributed               2023.6.1           pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.2              pyhd8ed1ab_0    conda-forge
executing                 1.2.0                    pypi_0    pypi
expat                     2.5.0                hcb278e6_1    conda-forge
fiona                     1.9.4           py310h111440e_0    conda-forge
flit-core                 3.9.0              pyhd8ed1ab_0    conda-forge
folium                    0.14.0             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.40.0          py310h2372a71_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
freexl                    1.0.6                h166bdaf_1    conda-forge
fsspec                    2023.6.0           pyh1a96a4e_0    conda-forge
gdal                      3.7.0           py310h65bb550_3    conda-forge
geographiclib             1.52               pyhd8ed1ab_0    conda-forge
geopandas                 0.13.2             pyhd8ed1ab_1    conda-forge
geopandas-base            0.13.2             pyha770c72_1    conda-forge
geopy                     2.3.0              pyhd8ed1ab_0    conda-forge
geos                      3.11.2               hcb278e6_0    conda-forge
geotiff                   1.7.1                h22adcc9_9    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
hdf4                      4.2.15               h501b40f_6    conda-forge
hdf5                      1.14.1          nompi_h4f84152_100    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.7.0              pyha770c72_0    conda-forge
importlib_metadata        6.7.0                hd8ed1ab_0    conda-forge
importlib_resources       5.12.0             pyhd8ed1ab_0    conda-forge
ipykernel                 6.23.3                   pypi_0    pypi
ipython                   8.14.0                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                8.0.6                    pypi_0    pypi
jedi                      0.18.2                   pypi_0    pypi
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.3.0              pyhd8ed1ab_1    conda-forge
json-c                    0.16                 hc379101_0    conda-forge
json5                     0.9.14             pyhd8ed1ab_0    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0                    pypi_0    pypi
jupyter-console           6.6.3                    pypi_0    pypi
jupyter_client            7.4.1              pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.1           py310hff52083_0    conda-forge
jupyter_server            1.23.6             pyhd8ed1ab_0    conda-forge
jupyterlab-widgets        3.0.7                    pypi_0    pypi
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.23.0             pyhd8ed1ab_0    conda-forge
kealib                    1.5.1                h3e6883b_4    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4           py310hbf28c38_1    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lcms2                     2.15                 haa2dc70_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230125.3      cxx17_h59595ed_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libarchive                3.6.2                h039dbb9_1    conda-forge
libarrow                  12.0.1           h32ed672_2_cpu    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.1.2                h409715c_0    conda-forge
libdeflate                1.18                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgdal                   3.7.0                h4a547c6_3    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libglib                   2.76.3               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libgoogle-cloud           2.12.0               h840a212_1    conda-forge
libgrpc                   1.56.0               h3905398_2    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.5.1              h0b41bf4_0    conda-forge
libkml                    1.3.0             h37653c0_1015    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
libnetcdf                 4.9.2           nompi_h78c856c_106    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.3                 hbcd7760_1    conda-forge
libprotobuf               4.23.3               hd1fb520_0    conda-forge
librttopo                 1.1.0               h0d5128d_13    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libspatialindex           1.9.3                h9c3ff4c_4    conda-forge
libspatialite             5.0.1               hca56755_27    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libthrift                 0.18.1               h8fd135c_2    conda-forge
libtiff                   4.5.1                h8b53f26_0    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.1                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzip                    1.9.2                hc929e4a_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.3.2           py310h0cfdcf0_0    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mapclassify               2.5.0              pyhd8ed1ab_1    conda-forge
markupsafe                2.1.3           py310h2372a71_0    conda-forge
matplotlib-base           3.7.1           py310he60537e_0    conda-forge
matplotlib-inline         0.1.6                    pypi_0    pypi
mistune                   3.0.0              pyhd8ed1ab_0    conda-forge
movingpandas              0.16.0             pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
multiprocess              0.70.14         py310h5764c6d_3    conda-forge
munch                     4.0.0              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nbclassic                 1.0.0                    pypi_0    pypi
nbclient                  0.7.4              pyhd8ed1ab_0    conda-forge
nbconvert                 7.6.0              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.6.0              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.6.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.9.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
networkx                  3.1                pyhd8ed1ab_0    conda-forge
notebook                  6.5.4                    pypi_0    pypi
notebook-shim             0.2.3                    pypi_0    pypi
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
numpy                     1.25.0          py310ha4c1d20_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.1.1                hd590300_1    conda-forge
orc                       1.9.0                h385abfd_1    conda-forge
osmnx                     1.5.0              pyhd8ed1ab_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    2.0.3           py310h7cbd5c2_0    conda-forge
pandoc                    3.1.3                h32600fe_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3                    pypi_0    pypi
partd                     1.4.0              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    10.0.0          py310h582fbeb_0    conda-forge
pip                       23.1.2             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
platformdirs              3.8.0              pyhd8ed1ab_0    conda-forge
pooch                     1.7.0              pyha770c72_3    conda-forge
poppler                   23.05.0              hd18248d_1    conda-forge
poppler-data              0.4.12               hd8ed1ab_0    conda-forge
postgresql                15.3                 hd458b1d_1    conda-forge
proj                      9.2.1                ha643af7_0    conda-forge
prometheus_client         0.17.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.38                   pypi_0    pypi
psutil                    5.9.5           py310h1fa729e_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure-eval                 0.2.2                    pypi_0    pypi
pyarrow                   12.0.1          py310h0576679_2_cpu    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygeos                    0.14            py310h056c13c_1    conda-forge
pygments                  2.15.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.0              pyhd8ed1ab_0    conda-forge
pyproj                    3.6.0           py310h24ef57a_1    conda-forge
pyrsistent                0.19.3          py310h1fa729e_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.12         hd12c33a_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.17.1             pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py310h5764c6d_5    conda-forge
pyzmq                     25.1.0          py310h5bbb5d0_0    conda-forge
qtconsole                 5.4.3                    pypi_0    pypi
qtpy                      2.3.1                    pypi_0    pypi
rasterio                  1.3.8           py310hd227816_0    conda-forge
rdma-core                 28.9                 h59595ed_1    conda-forge
re2                       2023.03.02           h8c504da_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
rtree                     1.0.1           py310hbdcdc62_1    conda-forge
s2n                       1.3.46               h06160fa_0    conda-forge
scikit-learn              1.3.0           py310hf7d194e_0    conda-forge
scipy                     1.11.1          py310ha4c1d20_0    conda-forge
send2trash                1.8.2              pyh41d4057_0    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
shapely                   2.0.1           py310h056c13c_1    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
snuggs                    1.4.7                      py_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.42.0               h2c6b66d_0    conda-forge
stack-data                0.6.2                    pypi_0    pypi
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
terminado                 0.17.1             pyh41d4057_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tiledb                    2.13.2               hd532e3d_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.3.2           py310h2372a71_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.7.1                hd8ed1ab_0    conda-forge
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzcode                    2023c                h0b41bf4_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucx                       1.14.1               hf587318_2    conda-forge
unicodedata2              15.0.0          py310h5764c6d_0    conda-forge
urllib3                   2.0.3              pyhd8ed1ab_1    conda-forge
voila                     0.4.1              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.6                    pypi_0    pypi
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.6.1              pyhd8ed1ab_0    conda-forge
websockets                11.0.3          py310h2372a71_0    conda-forge
wheel                     0.40.0             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.7                    pypi_0    pypi
xerces-c                  3.2.4                h8d71039_2    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xyzservices               2023.5.0           pyhd8ed1ab_1    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zict                      3.0.0              pyhd8ed1ab_0    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.2                h3eb15da_6    conda-forge

How did you install OSMnx?
Conda and conda-forge
Problem description
Hello.
I am trying to run the function shortest_path() using 16 cores that I have available in my virtual machine.
I don't know why the function is it not using all the cores I have available. I am sure I have 16 cores in my machine checking in the next way.
import multiprocessing as mp
cpus = mp.cpu_count()
Giving back 16.
Then, I am using a list of 5000 origins and 5000 destinations, a total of 5000 routes as outcome. I have tested in the next ways
1
ox.shortest_path(graph,                           orig=closest_origin_node_list,                           dest=closest_target_node_list,                           weight='length',                          cpus=1)
Processing time 5.33 minutes, htop shows only 1 core running. All correct.
2
ox.shortest_path(graph,                           orig=closest_origin_node_list,                           dest=closest_target_node_list,                           weight='length',                          cpus=None)
Processing time 5.33 minutes, htop shows only 1 core running. This is incorrect, I should see 16 cores running.
3
ox.shortest_path(graph,                           orig=closest_origin_node_list,                           dest=closest_target_node_list,                           weight='length',                          cpus=16)
Processing time 5.33 minutes, htop shows only 1 core running. This is incorrect, I should see 16 cores running.
Am I doing something wrong? Is there something wrong with the code?
Complete minimal reproducible example
# ---- 1) geom as destination

destin_geom = get_geom_destination()

# ---- 2) Closest destination node

closest_target_node = ox.nearest_nodes(G=graph,
                                       X=destin_geom.x,
                                       Y=destin_geom.y)

# ----- 3) Closest origins node

closest_origin_node_list = [ox.nearest_nodes(G=graph,
                                           X=origin_geom.x,
                                           Y=origin_geom.y)
                       for origin_geom in all_origins_list]

# -------- 4) Fetch graph nodes

graph_nodes = ox.graph_to_gdfs(graph, edges=False)

# --------5) Shortest Path ---------

# len origins and destinations must be equal
closest_target_node_list = [closest_target_node] * len(closest_origin_node)

# run
routes = ox.shortest_path(graph,
                         orig=closest_origin_node_list,
                         dest=closest_target_node_list,
                         weight='length',
                         cpus=16)

# ---- 6) Get path as geom GDF

all_routes_gdf = gpd.GeoDataFrame()

n=0
for each_route in routes:
    n=n+1

    print(f'Processing route {n}/5000')

    # ----- create nodes path to GDF

    route_gdf = nodes_to_path(graph_nodes, each_route)

    # ----- Compute distance

    route_distance_gdf = compute_distance(route_gdf)

    all_routes_gdf = pd.concat([all_routes_gdf, route_distance_gdf])

What kind of virtual machine is this? Does it use core affinity or anything like SLURM that affects the cores per task?

Hi, I solved it.
I run the function
ox.shortest_path(graph,  orig=closest_origin_node_list,  dest=closest_target_node_list,  weight='length', cpus=None)
in an independent cell in the Jupyter Notebook and it worked. In total, 19277 routes from 15mins (1 core) to 2 mins (16 core)
The process that took the longest was ox.nearest_nodes() like 40 mins (1 core). I did parallelization with multiprocessing and it took 4 mins.
I recommend to implement paralellization also in the function ox.nearest_nodes()
Thank you for the reply :)

I recommend to implement paralellization also in the function ox.nearest_nodes()

There are no loops in the nearest_nodes function to parallelize.

repo: gboeing/osmnx

license: mit


text:
Include src directory in package

So it would be possible to use "Go to implementation" and view ts source file instead of js file

@ptol please describe the required steps or send a PR.
Relevant https://github.com/gcanti/fp-ts/issues/223

authors: gcanti ptol

repo: gcanti/fp-ts


text:
Bug report: node; SyntaxError: Unexpected identifier crypto-api

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior or a link to the recipe / input used to cause the bug:
git clone https://github.com/gchq/cyberchef
cd cyberchef
npm install
grunt node

test.js:
var chef = require("cyberchef/build/node/CyberChef.js")
console.log(chef)
chef.default.bake("test", [{"op":"To Hex","args":["Space"]}]).then(r => { console.log(r) })

node test.js
cyberchef/node_modules/crypto-api/src/crypto-api.mjs:3
import Has160 from "./hasher/has160";
       ^^^^^^

SyntaxError: Unexpected identifier
    at Module._compile (internal/modules/cjs/loader.js:760:23)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:827:10)
    at Module.load (internal/modules/cjs/loader.js:685:32)
    at Function.Module._load (internal/modules/cjs/loader.js:620:12)
    at Module.require (internal/modules/cjs/loader.js:723:19)
    at require (internal/modules/cjs/helpers.js:14:16)
    at eval (webpack://CyberChef/external_%22crypto-api/src/crypto-api.mjs%22?:1:18)
    at Object.crypto-api/src/crypto-api.mjs (cyberchef/build/node/CyberChef.js:5377:1)
    at __webpack_require__ (cyberchef/build/node/CyberChef.js:41:30)
    at eval (webpack://CyberChef/./src/core/lib/Hash.mjs?:4:87)

Expected behavior
test to hex gets executed
Desktop (if relevant, please complete the following information):

OS: linux
Browser node
Version v11.15.0

@shellliii please try with v9.0.5

Note a couple of things -

require("cyberchef") should now be enough, rather than explicit path (there is no 'build' version for node now)
bake is not async

This code worked for me n v11.15.0 in a new dir:
npm init (accept all defaults)
npm install --save cyberchef
vim test.js (with the following contents)
const chef = require("cyberchef")
const d = chef.bake("test", [{"op":"To Hex","args":["Space"]}])
console.log(d)

node test.js

Closing for now. Please reopen if you have any issues.

I downgrade to node 10,and it works,the reason is here #596

repo: gchq/CyberChef


text:
gh-1469 - added sample elements for split points operation

There is a bit of extra complexity in the handler as I have attempted to make an abstract class that is generic so that it is easy for different Stores to implement it. The operation SampleElementsForSplitPoints returns a List where T is the split point type. For Accumulo and HBase this is a String, but other stores may use a different type. For Accumulo and HBase the split points are Base64 encoded.

Codecov Report

Merging #1517 into develop will decrease coverage by 0.07%.
The diff coverage is 57.02%.

@@              Coverage Diff              @@
##             develop    #1517      +/-   ##
=============================================
- Coverage      66.21%   66.14%   -0.08%
- Complexity      1768     1780      +12
=============================================
  Files            731      735       +4
  Lines          24232    24353     +121
  Branches        3129     3140      +11
=============================================
+ Hits           16046    16108      +62
- Misses          6920     6974      +54
- Partials        1266     1271       +5

Impacted Files
Coverage Δ
Complexity Δ

...fer/operation/serialisation/TypeReferenceImpl.java
56.75% <0%> (-1.58%)
1 <0> (ø)

...r/AbstractSampleElementsForSplitPointsHandler.java
0% <0%> (ø)
0 <0> (?)

...java/uk/gov/gchq/gaffer/hbasestore/HBaseStore.java
64.81% <100%> (+0.66%)
0 <0> (ø)
:arrow_down:

.../core/AbstractCoreKeyAccumuloElementConverter.java
82.47% <100%> (+0.3%)
0 <0> (ø)
:arrow_down:

...k/gov/gchq/gaffer/accumulostore/AccumuloStore.java
67.53% <100%> (+0.21%)
0 <0> (ø)
:arrow_down:

...n/handler/SampleElementsForSplitPointsHandler.java
77.27% <77.27%> (ø)
0 <0> (?)

...n/handler/SampleElementsForSplitPointsHandler.java
84.21% <84.21%> (ø)
0 <0> (?)

...r/operation/impl/SampleElementsForSplitPoints.java
90.9% <90.9%> (ø)
13 <13> (?)

.../gov/gchq/gaffer/store/schema/SchemaOptimiser.java
75.26% <0%> (-6.46%)
0% <0%> (ø)

...onutil/iterable/LimitedInMemorySortedIterable.java
47.22% <0%> (-2.78%)
15% <0%> (-1%)

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 27056a0...82e07e0. Read the comment docs.

I've made a couple of minor comments.
Main comment is how are the list of split points applied to a table? There is a SplitStore operation but that takes a path to a file of split points, whereas the new operation added here outputs a list of split points. Is the intention that the split points will be written to file and then read by the SplitStore operation? Should we add the ability to split a store directly from a list of split points?

@gaffer01 - added made your suggested changes. I've deprecated SplitStore and created SplitStoreFromFile and SplitStoreFromIterable. The SplitStoreFromFile reads the file locally or in HDFS then delegates to SplitStoreFromIterable.

repo: gchq/Gaffer


text:
Make retrieval of config.json async

Move store instantiation to index.js

Not doing

repo: gchq/stroom-auth


text:
DeDupe for Rules not working

created a Batch Search rule with the following Query, ran it every minute for 3 minutes, I received multiple firings
from Dual

eval Grp = 'TEST'

group by Grp

select Grp

see https://github.com/gchq/stroom/issues/4280

repo: gchq/stroom


text:
🛑 Phone Services is down

In a88ea15, Phone Services ($PHONE_SERVICES) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Phone Services is back up in cbe74f5 after 11 minutes.

repo: gcommeuneidee/upptime