text:
IREE Runtime - UNAVAILABLE; VK_ERROR_INITIALIZATION_FAILED error

What happened?
On trying to pass the IR through iree-run-module, I get the following error :-
C:\A\iree\runtime\src\iree\hal\drivers\vulkan\native_executable.cc:157: UNAVAILABLE; VK_ERROR_INITIALIZATION_FAILED; while invoking native function hal.executable.create; while calling import;
[ 1]   native hal.executable.create:0 -
[ 0] bytecode module.__init:268 .\dispatch\module_forward_dispatch_28_vulkan_spirv_fb.mlir:2:3

This takes place for --iree-vulkan-target-triple=rdna2-unknown-windows
Steps to reproduce your issue
Download module_forward_dispatch_28_vulkan_spirv_fb.mlir.
Step 1.
.\iree-compile.exe module_forward_dispatch_28_vulkan_spirv_fb.mlir --iree-hal-target-backends=vulkan --iree-vulkan-target-triple=rdna2-unknown-windows -o test_28vmfb

Step 2.
.iree-run-module.exe --module=test_28.vmfb --device=vulkan --function=forward_dispatch_28_matmul_4096x512x512

What component(s) does this issue relate to?
Compiler, Runtime
Version information
No response
Additional context
No response

This used to work at some point and stopped working in the recent past. We have pinned to an earlier version to workaround.

ew, using a temporary buffer is really unfortunate - one shouldn't be needed here for a simple bias add.

So it was working until about a week ago and we hit the issue. We have pinned torch-mlir to an older version to avoid this issue.  Is the issue in higher level in torch-mlir or in flow dialect ?

This is because we have a transposed linalg.generic fused together with linalg.matmul:
%10 = linalg.matmul
  ins(%4, %5 : tensor<4096x512xf16>, tensor<512x512xf16>)
  outs(%9 : tensor<4096x512xf16>) -> tensor<4096x512xf16>
%11 = linalg.generic {
  indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d1)>,
                   affine_map<(d0, d1) -> (d1, d0)>],
  iterator_types = ["parallel", "parallel"]
} ins(%10, %6 : tensor<4096x512xf16>, tensor<512xf16>) outs(%7 : tensor<512x4096xf16>) {
^bb0(%in: f16, %in_0: f16, %out: f16):
  %12 = arith.addf %in, %in_0 : f16
  linalg.yield %12 : f16
} -> tensor<512x4096xf16>

So bufferization allocated a buffer from workgroup memory to hold the intermediate result for linalg.matmul, and that exceeds the total amount of allowed workgroup memory.
The issue is likely at a higher level. I'd be interested to know how we generate such dispatches in flow. Did you perform layout transformations and handle transposes somehow causing this?

So I used the linalg IR I get from torch-mlir to dump the dispatches using iree-compile and find the "culprit" dispatch.
As @powderluv mentioned, we have pinned to an older version of torch-mlir to avoid this issue.
There seems to be no special layout transformations or handling of transposes that we're using.

Can we confirm the IR difference between the two versions? And post both too?

The op count difference with OLD vs LATEST torch-mlir :
At torch level:
torch.aten.broadcast_to        , 126 vs 120
torch.aten.mm                  , 1 vs 4
torch.aten.view                , 81 vs 78
torch.prim.ListConstruct       , 147 vs 146

At linalg level:
linalg.batch_matmul      , 5 vs 2
linalg.generic           , 794 vs 790
linalg.matmul            , 1 vs 4
linalg.yield             , 844 vs 840
tensor.collapse_shape    , 65 vs 66
tensor.empty             , 39 vs 38
tensor.expand_shape      , 126 vs 129

Elided Linalg IR which has the issue
Elided Linalg IR which we have from the older version of torch-mlir
The main set of difference I see are the 3 "extra" matmuls we have in the newer IR (%94, %97 and %100) forming a matmul + expand_shape + generic op set.
I tried experimenting with different torch versions and found that the issue is specifically with torch==2.0.0.dev20230228.
If we keep the latest torch-mlir, but pin to older torch versions, say torch==2.0.0.dev20230220 or torch==2.0.0.dev20230227 - the issue doesn't persist.
There are just a handful of decompositions we use in our pipeline as can be seen here.
So, I tried inspecting the decompositions which we're mainly using in our pipeline and the only "relevant" delta I see between torch==2.0.0.dev20230228 and torch==2.0.0.dev20230220 is fix embedding_backward_dense.
I tried reverting the one line change to see if it has any effect, but to avail.

Thanks for the full input IR. However, I cannot reproduce the issue. With e2151d3beb14171e05fc05e4be168668c1f150dc and tools/iree-compile --iree-hal-target-backends=vulkan --iree-vulkan-target-triple=rdna2-unknown-windows --compile-to=flow running on the new Linalg IR supposed to have the issue, I see 1) all the dispatch regions containing fused matmul and elementwise do not involve transpose as discovered above; transpose linalg.generic ops are in their own dispatch regions. Please double check that this is still a problem (maybe some transformation done on your own side causing it?).

I think this bug is magically fixed. I don't know what changed. We will revert the pin and keep an eye out and report back. Thank you for investigating.

yeah there is no magic :( I was testing via the wmma pipeline which worked. This failure only  happens on the RDNA2/ SIMT pipeline. Re-opening.

@yzhang93 / @qedawkins could this be because the tunings changed or are now invalid since @antiagainst tried on top of master?

yup confirmed that the tunings we apply causes this crash at runtime.  Maybe we can have runtime checks for them ? I don't know how the verifier let this go ? Maybe we need to enhance the verifier to capture this failure too.

The verifier might not be considering fused ops when determining shared memory requirements. Also I think it only verifies named matmuls (everything else just passes straight through) but I'd have to double check the verifier to be sure (am away from desk)

yup confirmed that the tunings we apply causes this crash at runtime. Maybe we can have runtime checks for them ? I don't know how the verifier let this go ? Maybe we need to enhance the verifier to capture this failure too.

I'm sure previously both tuned and untuned model had the failure. But maybe something has changed and untuned model works fine now. I think it's the VAE model and we don't apply lowering configs on it. I'll check if the Winograd transform caused the problem.

Not sure tuning is the problem--tuning just adjusts the tile sizes and such after seeing the dispatch region. The problem is having fused transposed elementwise op in the dispatch region from the beginning, which causes bufferization to insert extra allocations in shared memory.
It would be beneficial to understand why we are forming such dispatch regions, like doing --mlir-print-ir-after-all to understand how such dispatch region is generated using Shark fork, as I cannot repro this with IREE top of the tree. (I assume its some combination of patterns causing this..) The solution would be either to avoid forming such dispatch regions (e.g., separating/having transpose in its own dispatch region), or teach bufferization to be smarter (not sure how feasible it is here).

Or actually maybe I'm not using the proper command-line option to reproduce the issue from the full model. I was using tools/iree-compile --iree-hal-target-backends=vulkan --iree-vulkan-target-triple=rdna2-unknown-windows from the issue; but that's just for the single dispatch. I recall there are preprocessing steps. Let me know.

here are our typical flags
 C:\Users\foo\AppData\Local\Temp\_MEI83882\iree\compiler\tools\..\_mlir_libs\iree-compile.exe - --iree-input-type=none --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=vulkan --iree-llvm-embedded-linker-path=C:\Users\foo\AppData\Local\Temp\_MEI83882\iree\compiler\tools\..\_mlir_libs\iree-lld.exe --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false --iree-llvm-target-cpu-features=host --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-util-zero-fill-elided-attrs -iree-vulkan-target-triple=rdna2-7900-windows

@antiagainst I just tested with the latest nightly IREE python package (https://github.com/openxla/iree/releases/tag/candidate-20230311.455) and the above dispatch still has the compilation error. I tested on my navi3 system with the following commands:
iree-run-module --module=test_28.vmfb --device=vulkan --function=forward_dispatch_28_matmul_4096x512x512```

@yzhang93 can we please run with --mlir-print-ir-after-all too

Here is the output of --mlir-print-ir-after-all

Sorry, I still cannot reproduce the issue. I used commit https://github.com/openxla/iree/commit/503d81229dcf53aa3f391866c3fa93231831b7bb (which is behind candidate-20230311.455) and the command line given in https://github.com/openxla/iree/issues/12523#issuecomment-1465059677 tools/iree-compile- --iree-input-type=none --iree-hal-target-backends=vulkan --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-util-zero-fill-elided-attrs -iree-vulkan-target-triple=rdna2-7900-windows on this input IR. This is what I get after compiling to Flow: https://gist.github.com/antiagainst/08758cde21ba798ac68d27b6016ddd67.
You can see that dispatch 28 is a transpose in its own dispatch region, not fused with linalg.matmul. Actually if you search affine_map<(d0, d1) -> (d1, d0)>, all such cases are in their own dispatch regions. OTOH, if you search linalg.matmul, all linalg.generic ops fused with them don't have the transposed cases like showed in the original report. Yes, module_forward_dispatch_28_vulkan_spirv_fb.mlir alone is problematic, but I still don't know how it's produced..

I can recreate it every time with rdna2 and choosing a size not in SHARK tank so it exports with torch-mlir. I did the following for 768x512. Let me try to dump the mlir file so we can recreate it end to end.
(shark.venv) PS C:\g\SHARK\apps\stable_diffusion\web> iree-run-module.exe --module=.\vae1_64_768_512_fp16_stable-diffusion-2-1-base_vulkan.vmfb --device=vulkan --function="forward" --input=1x4x64x64xf16
EXEC @forward
D:\a\SHARK-Runtime\SHARK-Runtime\c\runtime\src\iree\modules\hal\utils\buffer_diagnostics.c:220: INVALID_ARGUMENT; tensor shape dimension 2 mismatch; expected 96 but have 64; expected shape 1x4x96x64, actual shape 1x4x64x64; while invoking native function hal.buffer_view.assert; while calling import;
[ 1]   native hal.buffer_view.assert:0 -
[ 0] bytecode module.forward:2900 [unknown]; invoking function 'forward'

@antiagainst Could you please try this mlir?
Command lines:
iree-compile --iree-input-type=none --iree-hal-target-backends=vulkan --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-util-zero-fill-elided-attrs --iree-vulkan-target-triple=rdna2-unknown-linux --iree-preprocessing-pass-pipeline='builtin.module(func.func(iree-flow-detach-elementwise-from-named-ops,iree-preprocessing-pad-linalg-ops{pad-size=32}))' vae_1_64_512_512_fp16_stable-diffusion-2-1-base_vulkan_torch.mlir -o test.vmfb
iree-benchmark-module --module=test.vmfb --function=forward --device=vulkan --input=1x4x64x64xf16
The error output:
[ 1]   native hal.executable.create:0 -
[ 0] bytecode module.__init:2106 vae_1_64_512_512_fp16_stable-diffusion-2-1-base_vulkan_torch.mlir:665:12
      at vae_1_64_512_512_fp16_stable-diffusion-2-1-base_vulkan_torch.mlir:18:3```

ok this seems like a codegen change that is breaking AMD but not Nvidia
I compiled the same code for rdna2 and it runs ok my 4090 card but fails on my AMD card.
(shark.venv) PS C:\g\SHARK\apps\stable_diffusion\web> iree-compile --iree-input-type=none --iree-hal-target-backends=vulkan --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-util-zero-fill-elided-attrs --iree-vulkan-target-triple=rdna2-unknown-linux --iree-preprocessing-pass-pipeline='builtin.module(func.func(iree-flow-detach-elementwise-from-named-ops,iree-preprocessing-pad-linalg-ops{pad-size=32}))'

C:\Users\anush\Downloads\vae_1_64_512_512_fp16_stable-diffusion-2-1-base_vulkan_torch.mlir -o test.vmfb
(shark.venv) PS C:\g\SHARK\apps\stable_diffusion\web> iree-benchmark-module --module=test.vmfb --function=forward --device=vulkan://1 --input=1x4x64x64xf16

D:\a\SHARK-Runtime\SHARK-Runtime\c\runtime\src\iree\hal\drivers\vulkan\native_executable.cc:157: UNAVAILABLE; VK_ERROR_INITIALIZATION_FAILED; while invoking native function hal.executable.create; while calling import;
[ 1]   native hal.executable.create:0 -
[ 0] bytecode module.__init:2106 C:\Users\anush\Downloads\vae_1_64_512_512_fp16_stable-diffusion-2-1-base_vulkan_torch.mlir:665:12
      at C:\Users\anush\Downloads\vae_1_64_512_512_fp16_stable-diffusion-2-1-base_vulkan_torch.mlir:18:3

(shark.venv) PS C:\g\SHARK\apps\stable_diffusion\web> iree-benchmark-module --module=test.vmfb --function=forward --device=vulkan://0 --input=1x4x64x64xf16
2023-03-13T13:07:56-07:00
Running C:\g\shark\shark.venv\Lib\site-packages\iree\runtime\scripts\iree_benchmark_module\..\..\iree-benchmark-module
Run on (64 X 2720.07 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x32)
  L1 Instruction 32 KiB (x32)
  L2 Unified 512 KiB (x32)
  L3 Unified 16384 KiB (x8)
--------------------------------------------------------------------------------------------
Benchmark                                  Time             CPU   Iterations UserCounters...
--------------------------------------------------------------------------------------------
BM_forward/process_time/real_time        523 ms          469 ms            1 items_per_second=1.91174/s

This system may have an older driver so I am going to try with the latest one to make sure we are on the latest.

@antiagainst if this is a AMD driver issue is there a temporary way to override this dispatch 28 creation until it is fixed ?

Also should the dispatch_28 not be formed for those cards that expose less shared memory ?

How much is the shared memory usage?

it seems like the tile sizes chosen (which affects the shared memory usage) is not account for shared memory usage... So this is a backend issue.

@yzhang93 can we tune dispatch_28 for rdna2 so it doesn't crash ?

it seems like the tile sizes chosen (which affects the shared memory usage) is not account for shared memory usage... So this is a backend issue.

Backend here is the AMD vulkan driver ?

it seems like the tile sizes chosen (which affects the shared memory usage) is not account for shared memory usage... So this is a backend issue.

Backend here is the AMD vulkan driver ?

No, this should be the SPIR-V backend in IREE.

ah ok. Our generated spv

Thanks for the repro steps in https://github.com/openxla/iree/issues/12523#issuecomment-1466602501. I've seen the issue and figured out what went wrong and put up a fix in #12627.

With the above we won't fuse such cases. @powderluv or somebody else if you can help to verify this works that'd be nice.

Thank you. I can confirm it works ok
(shark.venv) PS C:\g\shark> iree-compile --iree-input-type=none --iree-hal-target-backends=vulkan --iree-stream-resource-index-bits=64 --iree-vm-target-index-bits=64 --iree-util-zero-fill-elided-attrs --iree-vulkan-target-triple=rdna2-unknown-linux --iree-preprocessing-pass-pipeline='builtin.module(func.func(iree-flow-detach-elementwise-from-named-ops,iree-preprocessing-pad-linalg-ops{pad-size=32}))' C:\Users\anush\Downloads\vae_1_64_512_512_fp16_stable-diffusion-2-1-base_vulkan_torch.mlir -o test.vmfb
(shark.venv) PS C:\g\shark> iree-benchmark-module --module=test.vmfb --function=forward --device=vulkan --input=1x4x64x64xf16
2023-03-14T00:10:14-07:00
Running C:\g\shark\shark.venv\Lib\site-packages\iree\runtime\scripts\iree_benchmark_module\..\..\iree-benchmark-module
Run on (32 X 4491.57 MHz CPU s)
CPU Caches:
 L1 Data 32 KiB (x16)
 L1 Instruction 32 KiB (x16)
 L2 Unified 1024 KiB (x16)
 L3 Unified 32768 KiB (x2)
--------------------------------------------------------------------------------------------
Benchmark                                  Time             CPU   Iterations UserCounters...
--------------------------------------------------------------------------------------------
BM_forward/process_time/real_time        792 ms        0.000 ms            1 items_per_second=1.26315/s

@antiagainst When the right fix lands would we back to original memory usage? We are carrying this locally on SHARK-Runtime and some end users on 8GB cards are running out of VRAM.

I'm not sure this one would address memory usage issues. We were never able to handle such fusion cases before; we never hit such cases previously. So it's not like we are not fusing some previously fused cases. The memory usage issue is likely different.

Silly Q, but this issue is marked "Done" in status, but open in the IREE project (https://github.com/orgs/openxla/projects/13?pane=issue&itemId=22177652). Any idea why? (I can click "Close with Comment", but I guess I don't get the difference here.) Thanks!

Ha, interesting. This is not done; so I moved it back to "In Progress". The fix in https://github.com/openxla/iree/pull/12627 is not the long term way to go. I'll spend some time to do it more proper later.

Hi - double checking on this P0 issue. More to say? Ok to close or lower priority? Thanks!

Moving this to a P1 since we have a workaround (for SHARK at least)

P1 is ok, but this is a release blocker, I think. I foolishly started talking about things in discord, but cross-posting here.
Looking at https://github.com/openxla/iree/pull/12627 it seems like we have the option to drop a feature in order to fix the bug. I would advocate for that or hiding it behind a flag so that we don't have to wait for a big rewrite to fix this issue.

P1 is ok, but this is a release blocker, I think. I foolishly started talking about things in discord, but cross-posting here.
Looking at #12627 it seems like we have the option to drop a feature in order to fix the bug. I would advocate for that or hiding it behind a flag so that we don't have to wait for a big rewrite to fix this issue.

It works on other backend, and its just a can we kicked down the road for a while....

If this is indeed a release blocker then I think we need to revert the offending feature. This looks like we are miscompiling and we have head and unstable releases in a known-broken state.

I'm coming to fix this in the proper way next. The issue is triggered by some new IR patterns from torch-mlir which we didn't see before. So it's not that we have a regression---previous releases won't support it either. I don't want to blocking releasing on my implementation; so I'm fine rolling forward the release.

Got it, then I think this is not a release blocker. Thanks for clarifying (and for fixing :slightly_smiling_face: )

authors: Abhishek-Varma GMNGeoffrey MaheshRavishankar aaron-schneider antiagainst benvanik powderluv qedawkins yzhang93

repo: openxla/iree


text:
Moves IREE Dispatch Profiler to Tools

This PR moves IREE dispatch profiler out of experimental into iree/tools. IREE dispatch profiler is a tool to verify and profile individual dispatches.
Other places that can be a place for dispatch_profiler would be:
iree/build_tools/benchmark : But it is not really a build_tool. It is used post build.
iree/python : dispatch_profiler is  is a python-based tool.
What is the most appropriate home for the dispatch profiler tool?

Tested that the workflow triggered on this PR succeeded: https://github.com/openxla/iree/actions/runs/5224458620

I'd prefer you put it there. Underneath build_tools/ the build_tools/python/

Given that the post-submit workflow has already been verified with the dispatch profiler moved to tools, I would prefer to keep the dispatch_profiler in iree/tools.

I'd prefer you put it there. Underneath build_tools/ the build_tools/python/

Given that the post-submit workflow has already been verified with the dispatch profiler moved to tools, I would prefer to keep the dispatch_profiler in iree/tools.

source: gharchive/pull-request

authors: GMNGeoffrey julianwa manishucsd

license_type: permissive

license_source: github-api


text:
[Flow] Avoid collapsing reduction dimensions on pre-formed dispatches

Rebased on top of #14433

#14433 has landed so you can rebase on top of main.

Rebased on top of main

Something is off with the rebase. You have commits that are not part of the PR

Rebase seems right now

authors: MaheshRavishankar Max191 qedawkins


text:
add new gateway version v1beta1

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind enhancement
/sig network

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

/lgtm

authors: River-sh rambohe-ch

repo: openyurtio/openyurt


text:
modify certificate related code

What type of PR is this?

Uncomment only one  /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

@YRXING Very appreciate for your new pr. i think It's a good idea to unify the csr approver and put it in yurt-controller-manager for all of openyurt components(like yurthub, yurt-tunnel.)

@YRXING please merge three commits into one commit.

authors: YRXING rambohe-ch


text:
Operator throwing error in an endless loop "too old resource version"

Bug Report

What did you do?
We are not sure of the events that led to this. It started occurring suddenly. A restart has fixed it though but the operator was non-functional by this time i.e. it was not reconciling anything

What did you expect to see?
No errors

What did you see instead? Under which circumstances?
Our operator is throwing the following in a endless loop
2024-04-19 08:59:46,858 i.f.k.c.d.i.AbstractWatchManager [ERROR] Received an error which is not a status but {"type":"ERROR","object":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"too old resource version: 31159423 (31160199)","reason":"Expired","code":410}} - will retry

Environment
Kubernetes cluster type:
EKS

$ Mention java-operator-sdk version from pom.xml file

4.8.2
$ java -version
openjdk version "21.0.2" 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-21.0.2.13.1 (build 21.0.2+13-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.2.13.1 (build 21.0.2+13-LTS, mixed mode, sharing)

$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.1-eks-b9c9ed7

Possible Solution

Additional context
Unfortunately no, this error has no logs prior to it and it just started occuring out of the blue. We are using 6.10.0 fabric8 client

actually, fabric8 6.11.0 is in the dependency tree

Classloading issues make this logic subseptiable to this problem - https://github.com/fabric8io/kubernetes-client/issues/5692
We could consider making the deserialization here to just generic instead, but more than likely the user will want to fix having more than one definition of Status in the classpath.

Hi @shawkins

but more than likely the user will want to fix having more than one definition of Status in the classpath
I'm not sure what is this Status you are referring to. Are you saying I look at my mvn dependency:tree?

this is how the deps look like
[INFO] +- io.javaoperatorsdk:operator-framework:jar:4.8.2:compile
[INFO] |  +- io.javaoperatorsdk:operator-framework-core:jar:4.8.2:compile
[INFO] |  |  \- io.fabric8:kubernetes-client:jar:6.11.0:compile
.
.
.
.
[INFO] +- io.strimzi:api:jar:0.40.0:compile
[INFO] |  +- io.fabric8:kubernetes-client-api:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-gatewayapi:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-resource:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-rbac:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-admissionregistration:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-apps:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-autoscaling:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-batch:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-certificates:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-coordination:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-discovery:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-events:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-extensions:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-flowcontrol:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-metrics:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-policy:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-scheduling:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-storageclass:jar:6.10.0:compile
[INFO] |  |  +- io.fabric8:kubernetes-model-node:jar:6.10.0:compile
[INFO] |  |  +- org.snakeyaml:snakeyaml-engine:jar:2.7:compile
[INFO] |  |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.16.0:compile
[INFO] |  +- io.fabric8:kubernetes-model-core:jar:6.10.0:compile
[INFO] |  +- io.fabric8:kubernetes-model-networking:jar:6.10.0:compile
[INFO] |  +- io.fabric8:kubernetes-model-common:jar:6.10.0:compile
[INFO] |  +- io.fabric8:kubernetes-model-apiextensions:jar:6.10.0:compile

is it better to keep the fabric8 version consistent?

@fhalde yes it is especially if you don't have a flat classloader and end up with two different Status class definitions accessible from different classloaders.

hmm, we definitely don't make use of any classloaders. is this some fabric8 internals? anyway here is what my fat jar contents look like
jar -tvf operator.jar | grep '/Status.class'
 io/javaoperatorsdk/operator/health/Status.class
 io/strimzi/api/kafka/model/kafka/Status.class
 io/fabric8/kubernetes/api/model/Status.class
 org/apache/logging/log4j/core/util/internal/Status.class
 ch/qos/logback/core/status/Status.class

@shawkins

Hi @metacosm , we were running our operator with a single version of fabric8 for a few days and today this error came up once again
here is what i could gather by attaching a debugger. the status message was unmarshalled into a GenericKubernetesResource class rather than Status. Weirdly the error stopped after while after I attached a remote debugger
If this comes up once again i'll let you know.

will close this issue, pls let us know if that happens again.

source: gharchive/issue

authors: csviri fhalde shawkins

repo: operator-framework/java-operator-sdk


text:
Add tmshort to reviewers

Testing remarks:

Reviewer Checklist

[ ] Code is properly formatted

/retest

/retest

license_source: bigquery


text:
resource update ignored when ResourceVersion do not change

Fix #335

@carllhw I'll have to close this for now. Please see #336 for why we can't add this change right now.
This is almost a duplicate of #294 as well which has more discussion on the issue with this fix.

repo: operator-framework/operator-sdk


text:
Add CI workflows for PRs and pushes

For both PRs and pushes:

golangci-lint
go-apidiff
go fmt && git diff
unit and e2e testing

For PRs:

goreleaser build, but skip push/release

For main branch pushes:

goreleaser build, push images with main tag

For tag pushes

goreleaser build, push images with versions tags, publish github release.

@joelanford I broke this out into multiple issues:

#39
#40
#41
#42
#44
#45

I think these issues covers everything, but before closing this issue in favor of all the separate issues I wanted to make sure it looked good to you as well

closing this issue as per above comment

authors: everettraven joelanford

repo: operator-framework/oria-operator


text:
Jailed instance of opnsense

Hello everyone,
If someone else is interested please join this topic.
Since FreeBSD has vnet (network virtualization), it might be helpful to have opnsense as jail instance (vimage/vnet).
So far I was able to run opnsense within jail, but with some limitation.
There are two major issues that might be fixed really quick.

No need to run /usr/bin/protect (it is not supported inside jail):
root@OPNsense:/ # /usr/bin/protect -i /usr/local/sbin/sshd
protect: procctl: Operation not permitted
What I propose here is to check two sysctl variables:
security.jail.jailed
security.jail.vnet
And if they are not 0, just run processes without protecting them.
TUN interfaces naming convention:
Since all devices in host OS (FreeBSD) are technically accessible from the jail and no way to have two jail instances with their TUN1 interfaces, they have to be named uniquely.
So, it is better to use ifconfig tun create instead ifconfig tun1 create. This will give us automatic interface number creation, even easier to use "ifconfig tun create name ovpns1"
Example is below:
root@OPNsense:/ # ifconfig tun create name ovpns10
ovpns10
These two obstacles are only obstacles I have found so far.
Let me know your thoughts on this. I can be assisting in technical or any other testing.

Hi,
So far VTNET is not enabled but I heard it will be for FreeBSD 12. Do you build your own kernel today?
For 1. that's an unfortunate issue but not major by any standard. Protection was added for low memory devices so be able to always gain access so the solution to probe security.jail.jailed sysctl is feasible. Patches welcome.
For 2. I'm not sure what yo mean. Isn't it already done this way? There is no good way to avoid clashing two OPNsense jails on a single host but that is not the goal here, is it? OpenVPN does need the device node as well as the interface name so we cannot directly create ovpns1 because we don't know which device node that is....
Cheers,
Franco

Hi Franco,
Thanks for stopping by and your interest.
First item is not a real issue, I can fix it by just replacing /usr/bin/protect with "fake" shell script which will be bypassing anything you add as a command arguments (will work perfectly).
Second part is a bit harder, but I will try to explain it here.
In my case I would like to have multiple opnsense instances on the same physical host.
Since all jails share same devfs (I mean same devices but some could be filtered), so we cannot create the same device (in my example tun1) for more that one jailed host or just host machine.
root@OPNsense:/ # ls -l /dev/tun*
crw-------  1 uucp  dialer  0x84 Sep  3 16:27 /dev/tun0
crw-------  1 uucp  dialer  0x64 Sep  3 16:11 /dev/tun1
crw-------  1 uucp  dialer  0x8a Sep  4 08:28 /dev/tun10
crw-------  1 uucp  dialer  0x8b Sep  4 08:28 /dev/tun11
crw-------  1 uucp  dialer  0x8c Sep  4 08:52 /dev/tun12
crw-------  1 uucp  dialer  0x65 Sep  3 16:11 /dev/tun2
crw-------  1 uucp  dialer  0x71 Sep  3 16:11 /dev/tun3
crw-------  1 uucp  dialer  0x72 Sep  3 16:11 /dev/tun4
crw-------  1 uucp  dialer  0x85 Sep  3 16:27 /dev/tun5
crw-------  1 uucp  dialer  0x86 Sep  3 16:27 /dev/tun6
crw-------  1 uucp  dialer  0x83 Sep  3 16:38 /dev/tun7
crw-------  1 uucp  dialer  0x88 Sep  4 08:27 /dev/tun8
crw-------  1 uucp  dialer  0x89 Sep  4 08:27 /dev/tun9
Of course this command will return such error:
Sep  3 16:37:43 OPNsense opnsense: /status_services.php: The command '/sbin/ifconfig 'tun1' name 'ovpns1'' returned exit code '1', the output was 'ifconfig: interface tun1 does not exist'
Sep  3 16:37:43 OPNsense opnsense: /status_services.php: The command '/sbin/ifconfig 'ovpns1' group openvpn' returned exit code '1', the output was 'ifconfig: interface ovpns1 does not exist'
If I do not specify the tun interface number, the system will check all tun interfaces and then chose the higher number +1
root@OPNsense:/ # ifconfig tun create
tun13
root@OPNsense:/ # ifconfig tun create
tun14
root@OPNsense:/ # ifconfig tun create
tun15

I see. The issue is that the device node is stored in the config.xml and that causes the problem. I haven't looked at how to get rid of this idiosyncrasy but I would guess it's possible with temporary files in /var/openvpn
https://github.com/opnsense/core/blob/master/src/etc/inc/plugins.inc.d/openvpn.inc#L539

Hi fichtner,
It's been a while since I started poking around in "jail" edition of the OPNsesne.
I was able to run OPNsense in a vimage jail and found some places that could be improved either as separate edition of OPNsesne (lets say additionally to Nano, Serial, VGA, it could be Jail), or can be implemented as a special check function that will tell if the instance is running under Jail.
Obstacles:

Openvpn tunnels handling.
protected sshd (protect simply doesn't work under jail).
So my temporarily solution for the first item is (until I have better one):
Instead of getting tunnel numbers by:

@@ -536,7 +536,7 @@
         $settings['dev_mode'] = "tun";
     }

-    //$devnode = "{$settings['dev_mode']}"."1067"."{$vpnid}";
+    $devnode = "{$settings['dev_mode']}{$vpnid}";

     if ($mode == "server") {
         $devname = "ovpns{$vpnid}";
@@ -545,14 +545,12 @@
     }

     if (!does_interface_exist($devname)) {
-        $devnode = exec_command('/sbin/ifconfig '.$settings['dev_mode'].' create');
-        //if (!file_exists("/dev/{$devnode}")) {
-        //    mwexecf('/sbin/ifconfig %s create', array($devnode));
-        //}
+        if (!file_exists("/dev/{$devnode}")) {
+            mwexecf('/sbin/ifconfig %s create', array($devnode));
+        }

         mwexecf('/sbin/ifconfig %s name %s', array($devnode, $devname));
         mwexecf('/sbin/ifconfig %s group openvpn', array($devname));
-        mwexecf('/sbin/ifconfig %s descr %s', array($devname, $devnode.":".$devname));
     }
@@ -563,7 +561,6 @@
     if (substr($settings['protocol'], 0, 3) == "TCP") {
         $proto = "{$proto}-{$mode}";
     }
-    $dev_mode = $settings['dev_mode'];
@@ -598,10 +595,7 @@
             break;
     }

-    //$conf .= "dev-node /dev/{$devnode}\n";
-    $myifdescr = exec_command('/sbin/ifconfig '.$devname.' | grep description | awk \'{print $2}\'');
-    $myifparam = explode(":", $myifdescr);
-    $conf .= "dev-node /dev/{$myifparam[0]}\n";
+    $conf .= "dev-node /dev/{$devnode}\n";

This "pathc" gives an ability to handle tun/tap interfaces automatically with no dependencies of strict tunnel numbers.
The code is a bit dirty, but it is just a proof of concept.
sshd can be fixed by checking whether instance is running under jail:
security.jail.vnet: 1
security.jail.jailed: 1
If so, just run /usr/local/sbin/sshd right away.
Let me know of any thoughts.

Let me know of any thoughts.

Can't say much to the code, 'coz I'm no programmer, but I really enjoy vnet jails.
I would like to have this possibility when opnsense gets on 12.0.

This issue has been automatically timed-out (after 180 days of inactivity).
For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.
If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

authors: AdSchellevis fichtner mr44er zeon-ua

repo: opnsense/core


text:
System: Configuration: Defaults: allow reboot as well as shutdown + confirmation dialog

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
[x] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[x] I have searched the existing issues and I'm convinced that mine is new.
Use case
Be able to reconfigure the system right after a configuration reset and reboot, without need to manually power on (e.g. firewall being remote, having separate connection to the first LAN port getting configured to default 192.168.1.1/24)
Current behaviour
"Shut down after changes are complete".
Proposed solution
Remove the "yes" button, instead show 2 buttons

Load defaults and poweroff
Load defaults and reboot

As an extra, add a confirmation dialog before performing the config reset.

This issue has been automatically timed-out (after 180 days of inactivity).
For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.
If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

authors: AdSchellevis igpit


text:
Fix issue with route add command

The hyphen was added in the wrong place causing the 'route add' command to error out. This was causing it to run
route add inet -'IP' 'GW'
when it should be
route add -inet 'IP ' GW'
Reference: https://github.com/opnsense/core/pull/6580#issue-1729389899

Nice catch, thank you!

authors: fichtner idanoo


text:
net/haproxy: chrooted sockets binding

Hi!
ref. https://forum.opnsense.org/index.php?topic=28007.0
looks like it require special unix sockets binding and targeting since haproxy is chrooted
pr moves sockets to /var/haproxy/sockets
Thanks!

Merged, thanks!
I cannot really test it, but it was confirmed to work on the forum post, so... :)

Thanks!)

authors: fraenki kulikov-a

repo: opnsense/plugins


text:
security/acme: Add JD Cloud API dns challenge type, closes #3315

Add new API for ACME dns challenge type:

JD Cloud

Merged, thanks!


text:
Create better support for multiple licenses

Is your feature request related to a problem? Please describe.
As a user, I want the option to enter more than one license per attribution.
Describe the solution you'd like
Create a mechanism to add and remove input fields, one for each license name. Additional licenses should not support custom license texts, only the first one does. Then, concatenate the license names on saving. Also parse concatenated license names from the input file to display them separately.
Additional context
At the moment, the user must know that licenses can be concatenated using "AND".

Related issue: #970

authors: benedikt-richter mstykow

repo: opossum-tool/OpossumUI


text:
Identity Set to blank instead of ApplicationPoolIdentity

Issue with recent patch iis@3.0.5

resolved #96

repo: opscode-cookbooks/iis


text:
Apppool properties

Jira : https://tickets.opscode.com/browse/COOK-4684
I updated the pool request from ProTip (https://github.com/opscode-cookbooks/iis/pull/41) to ad README update and some ruby cleanup.

Thanks for the cleanup @Annih .  I still feel a bit uncomfortable about passing arbitrary apppool properties through without checking them against the ones already defined.  what if a default somewhere respecifies a property that is explicitly handled as an attribute and someone tries to override the attribute but instead finds that their changes aren't taking effect.
Furthermore, the way we currently shell out, if a hash is provided and any single value in that has has & or ^ or % or something that we think might be a shell metacharacter, the apppool command is run through cmd /c which would throw of the escaping for all the other values.  It's generally inviting quite a bit of pain.
For now, if there are specific properties you want to add, add them as attributes.  I'll need to think a bit more about how best to handle arbitrary passthrough.

@ksubrama There are a couple of pull request open/closed-without-merge that all seem to have been created out of the necessity for more properties being set.  We are actually having the same issue.  There are several settings on the app pool that we need to set that are not currently supported by this recipe.  Have you been able to put additional thought to this.  I agree I don't think arbitrary passthrough is the right way, but I also don't want to maintain a separate fork just to add more property attributes.  I also don't like the idea of using the iis_config provider because that would not provide idempotence.  How hard would it be to support all possible properties?  Or maybe somehow allowing for a dynamic property/command definition.
Side note: I also like the idempotence/check that was added to pool.rb using is_new_value etc.

@nwesoccer can you give me a list of all the settings you want to be able to set on an app pool?  I'll put them all into official support, also thanks I was happy to write all the iis code that gained everyone idempotence.  It was something we were using and we are all about sharing.

@EasyAsABC123 Sure thing. The following are the properties we change from their default.  The ones I have in bold are the ones that I don't believe are currently supported by the recipe.  I generated this list using: appcmd list apppool /apppool.name:NAME -text:* and narrowed it down to only the ones IIS shows as modified from the defaults (bolded).
Ideally we'd be able to set any of the properties that this command lists.
Thanks!

[add]
startMode
[processModel]

identityType
userName
password
loadUserProfile
idleTimeout

[recycling]

disallowRotationOnConfigChange
logEventOnRecycle
[periodicRestart]

time
[schedule]

[add]

value

@nwesoccer Opened a new one for you https://github.com/opscode-cookbooks/iis/pull/123, it adds full support for the items mentioned above.  Feel free to test and make sure you are satisfied.
https://github.com/EasyAsABC123/iis/tree/feature-iis_pool_properties

Closing this - #123 should contain the properties you needed.

authors: Annih EasyAsABC123 ksubrama nwesoccer


text:
Set oc-chef-pedant to 1.0.69

Bumps pedant to the latest release, which is warning free on RSpec 2.99. A similar (but not identical, because rebase) branch passed Ci as chef-server-12-test build 484.

LGTM

:+1:

authors: danielsdeleo oferrigni sdelano

repo: opscode/opscode-omnibus


text:
[5.x] v5.0.0-RC5: Specifying kubernetes_version does not update k8s in node-pools

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version and Provider Version

terraform -v
Terraform v1.4.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.2
+ provider registry.terraform.io/hashicorp/helm v2.11.0
+ provider registry.terraform.io/hashicorp/http v3.4.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/hashicorp/time v0.9.1
+ provider registry.terraform.io/oracle/oci v4.119.0

Affected Resource(s)

oci_containerengine_node_pool
Terraform Configuration Files

module "oke" {
  source = "github.com/oracle-terraform-modules/terraform-oci-oke?ref=v5.0.0-RC5"

  kubernetes_version         = "v1.26.2"
  cluster_name               = local.name
  control_plane_is_public    = false
  cluster_type               = "enhanced"
  cluster_autoscaler_install = true

  compartment_id         = module.oke_prereqs.oke_compartment_id
  network_compartment_id = module.oke_prereqs.oke_compartment_id
  worker_compartment_id  = module.oke_prereqs.oke_compartment_id
  tenancy_id             = local.tenancy_id

  ssh_private_key_path = var.ssh_private_key_path
  ssh_public_key_path  = var.ssh_public_key_path

  home_region = local.home_region
  region      = var.region

  // node pool config
  worker_pools            = var.worker_pools
  worker_image_id         = "none"
  worker_image_type       = "platform"
  worker_image_os         = "Oracle Linux"
  worker_image_os_version = "8"

  // leaving this disabled for now, previously it didn't actually enable
  // kms secrets like it should, will have to go back and fix that if
  // we want kms encryption (we still have AES encryption).
  //cluster_kms_key_id = module.oke_prereqs.oke_kms_key_id

  create_operator       = true
  create_bastion        = true

  create_iam_resources       = true
  create_iam_operator_policy = "always"

  providers = {
    oci.home = oci.home
  }
}

Problem

I originally created a Kubernetes cluster, and the version was v1.25.4.

I then updated the Terraform to specify kubernetes_version = "v1.26.2"

I did a terraform apply

The control plane of the Kubernetes cluster was upgraded to v1.26.2

However, the Kubernetes versioin in the node pools were not updated and is still at v1.25.4

I queried the node pool:

oci ce node-pool get --node-pool-id ocid1.nodepool.oc1.region.xxxxxx

and see:
 {
        "availability-domain": "xJwU:US-ASHBURN-AD-1",
        "defined-tags": null,
        "fault-domain": "FAULT-DOMAIN-3",
        "freeform-tags": null,
        "id": "ocid1.instance.oc1.region.XXX",
        "kubernetes-version": "v1.25.4",

There needs to be a way to specify the kubernetes_version, and have it update the node pool metadata in OCI.

This line:
https://github.com/oracle-terraform-modules/terraform-oci-oke/blob/5.x/modules/workers/nodepools.tf#L105
ignores changes to the kubernetes_version for node pools.
There needs to be a way to upgrade the kubernetes_version in node pools, especially if the kubernetes_version is upgraded in the control plane.

Resolved in #830

authors: hyder rodrigc