text:
Consolidates and cleans up getTraces thrift query

This deprecates thrift calls for zipkin-web data conversion and
consolidates usage to a getTraces query. We will drop the obsolete
methods in 1.5.0, resulting in a much cleaner thrift api.
Before this change, there was a leaky abstraction between the query and
the web packages. For example, types like TraceCombo were introduced
eventhough they were only a single stage of data conversion and only
used to support one web view. The web component now handles data
conversions directly. Moreover, the web component duplicated work by
doing an id scan which implied looking up spans before looking up spans.
This also cleans up a couple questionable api concerns: adjuster and
binary annotation query.
In practice, there was only one type of adjustment, time-skew, and this
was almost always enabled. The code was confused by an idea that there
would be many types of adjustments, and in some places time-skew was
applied multiple times. This moves it to a boolean flag, just like the
http api.
The binary annotation query in thrift was a weird one. It allows the
user to send a string -> string query only. However, the signature
accepted literally the BinaryAnnotation type, which has other members,
and might not be a string. This clarifies the concern by only accepting
a map of string -> string.

+1 LGTM

belated :+1:

:+1:

authors: adriancole eirslett jamescway spencergibb


text:
The identity is not loaded without daemon and master_process being set to off in the config file

Experienced
The service is not reachable with the following error on the client side:
/mnt/c/Users/dsliwinski/Repos/sdk-golang/example$ build/curlz https://akssand-d225fb69.4b98eb49-011c-426f-9ce3-edfa150aa84e.privatelink.centralus.azmk8s.io build/dariusz-curlz.json
panic: Get "https://akssand-d225fb69.4b98eb49-011c-426f-9ce3-edfa150aa84e.privatelink.centralus.azmk8s.io": unable to dial service 'akssand-d225fb69.4b98eb49-011c-426f-9ce3-edfa150aa84e.privatelink.centralus.azmk8s.io': dial failed: service 47clQA1FrqLNXEwB8RHxEX has no terminators
goroutine 1 [running]:
main.main()
        /mnt/c/Users/dsliwinski/Repos/sdk-golang/example/curlz/curlz.go:63 +0x105
dsliwinski:/mnt/c/Users/dsliwinski/Repos/sdk-golang/example$

No logs recorded in the nginx log error file related to Ziti
Expected
Nginx process to run in daemon mode and able to reach services.
Reproduction

Build the module using Build using CMake.
Modify the existing conf file under configs/nginx.conf to comment out the following settings:
#daemon off;
#master_process off;
Run this command:
cmake-build/_deps/nginx/src/nginx/objs/nginx -c configs/nginx.conf -p .
Build curlz using Build Examples
Run Client Command:
build/curlz https://akssand-d225fb69.4b98eb49-011c-426f-9ce3-edfa150aa84e.privatelink.centralus.azmk8s.io build/dariusz-curlz.json

conf file details:
load_module cmake-build/ngx_ziti_module.so;

#error_log /dev/stderr debug;
error_log logs/error.log debug;

# development settings to keep nginx from starting in daemon mode and forking child processes
# do not use for non-dev deployments
#daemon off;
#master_process off;

thread_pool ngx_ziti_tp threads=32 max_queue=65536;

events {
    worker_connections  1024;
}

ziti myZitiInstaceNameUsedForLogging {
    identity_file /home/ziggy/azure-nginx-01.json;

    bind akssand-d225fb69.4b98eb49-011c-426f-9ce3-edfa150aa84e.privatelink.centralus.azmk8s.io {
        upstream akssand-d225fb69.4b98eb49-011c-426f-9ce3-edfa150aa84e.privatelink.centralus.azmk8s.io:443;
    }

    bind helloworld {
        upstream 10.17.1.114:5000;
    }

    bind 414523ab-69ab-47af-a99d-48467720a1c4.production.netfoundry.io {
        upstream 414523ab-69ab-47af-a99d-48467720a1c4.production.netfoundry.io:443;
    }
}

Seems to work just fine. As a warning, when daemon is not specified (results in on) or is explicitly set to on, the starting command will exit and spawn a background process. It will appear that the process exits, but it should leave a process running in the background you can see via ps -x | grep nginx,

The issue that I came across is not that I can not run the nginx in the background. I cam, but the ziti module is not working.
Here are logs when I run nginx in foreground.
2022/12/20 01:51:20 [notice] 99930#99930: using the "epoll" event method
2022/12/20 01:51:20 [notice] 99930#99930: nginx/1.23.2 (nginx-plus-r28)
2022/12/20 01:51:20 [notice] 99930#99930: built by gcc 11.2.0 (Ubuntu 11.2.0-19ubuntu1)
2022/12/20 01:51:20 [notice] 99930#99930: OS: Linux 5.15.0-1029-azure
2022/12/20 01:51:20 [notice] 99930#99930: getrlimit(RLIMIT_NOFILE): 1024:1048576
2022/12/20 01:51:20 [warn] 99930#99930: initializing block identity1
(99930)[        0.002]    INFO ziti_log_set_level set log level: root=1
2022/12/20 01:51:20 [info] 99930#99930: offloading service start for block identity1 service nginx-service-01
2022/12/20 01:51:20 [debug] 99930#99938: writing upstream
2022/12/20 01:51:20 [debug] 99930#99938: wrote 348d
2022/12/20 01:51:20 [debug] 99930#99938: writing to client
2022/12/20 01:51:20 [debug] 99930#99938: --wrote 2531d
2022/12/20 01:51:20 [debug] 99930#99938: writing upstream
2022/12/20 01:51:20 [debug] 99930#99938: wrote 1954d
2022/12/20 01:51:20 [debug] 99930#99938: writing upstream
2022/12/20 01:51:20 [debug] 99930#99938: wrote 86d
2022/12/20 01:51:20 [debug] 99930#99938: writing upstream
2022/12/20 01:51:20 [debug] 99930#99938: wrote 353d
2022/12/20 01:51:20 [debug] 99930#99938: writing to client
2022/12/20 01:51:20 [debug] 99930#99938: --wrote 55d
2022/12/20 01:51:20 [debug] 99930#99938: writing to client
2022/12/20 01:51:20 [debug] 99930#99938: --wrote 66d
2022/12/20 01:51:20 [debug] 99930#99938: writing to client
2022/12/20 01:51:20 [debug] 99930#99938: --wrote 8192d
2022/12/20 01:51:20 [debug] 99930#99938: writing to client
2022/12/20 01:51:20 [debug] 99930#99938: --wrote 1147d
2022/12/20 01:51:20 [debug] 99930#99938: writing upstream
2022/12/20 01:51:20 [debug] 99930#99938: wrote 31d
2022/12/20 01:51:20 [debug] 99930#99938: writing upstream
2022/12/20 01:51:20 [debug] 99930#99938: wrote 35d
2022/12/20 01:51:20 [debug] 99930#99938: closing, client disconnected
2022/12/20 01:51:20 [debug] 99930#99938: service client thread exited

Client kubectl call to get nodes info:
PS tmp>.\kubectl-windows-amd64.exe get nodes
NAME                                STATUS   ROLES   AGE    VERSION
aks-agentpool-15371798-vmss000000   Ready    agent   5d9h   v1.23.12
PS tmp>

It seems to work for me. I'm using this config:
load_module cmake-build-debug/ngx_ziti_module.so;

error_log /dev/stderr debug;
error_log logs/error.log debug;

# development settings to keep nginx from starting in daemon mode and forking child processes
# do not use for non-dev deployments
daemon on;
master_process on;

thread_pool ngx_ziti_tp threads=32 max_queue=65536;

events {
    worker_connections  1024;
}

ziti myZitiInstaceNameUsedForLogging {
    identity_file /home/andrew/.zi/identities/http_host.json;

    bind http-service {
        upstream localhost:7070;
    }
}

and in the error log file I can see:
2022/12/19 14:19:10 [notice] 26312#26312: using the "epoll" event method
2022/12/19 14:19:10 [notice] 26312#26312: nginx/1.23.2
2022/12/19 14:19:10 [notice] 26312#26312: built by gcc 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
2022/12/19 14:19:10 [notice] 26312#26312: OS: Linux 5.15.79.1-microsoft-standard-WSL2
2022/12/19 14:19:10 [notice] 26312#26312: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/12/19 14:19:10 [notice] 26312#26312: start worker processes
2022/12/19 14:19:10 [notice] 26312#26312: start worker process 26317
2022/12/19 14:19:10 [warn] 26317#26317: initializing block myZitiInstaceNameUsedForLogging
2022/12/19 14:19:10 [info] 26317#26317: offloading service start for block myZitiInstaceNameUsedForLogging service http-service
2022/12/19 14:19:10 [emerg] 26317#26320: connection with the upstream server failed
2022/12/19 14:19:40 [debug] 26317#26321: writing upstream
2022/12/19 14:19:40 [debug] 26317#26321: wrote 93d
2022/12/19 14:19:40 [debug] 26317#26321: writing to client
2022/12/19 14:19:40 [debug] 26317#26321: --wrote 224d
2022/12/19 14:19:40 [debug] 26317#26321: writing upstream
2022/12/19 14:19:40 [debug] 26317#26321: wrote 93d
2022/12/19 14:19:40 [debug] 26317#26321: writing to client
2022/12/19 14:19:40 [debug] 26317#26321: --wrote 224d
2022/12/19 14:19:40 [debug] 26317#26321: writing upstream
2022/12/19 14:19:40 [debug] 26317#26321: wrote 93d
2022/12/19 14:19:40 [debug] 26317#26321: writing to client
2022/12/19 14:19:40 [debug] 26317#26321: --wrote 224d
2022/12/19 14:19:40 [debug] 26317#26321: closing, upstream disconnected

The 2022/12/19 14:19:10 [emerg] 26317#26320: connection with the upstream server failed is me testing the upstream doing up/down was working in this mode.

It turns out my testing was flawed. I had daemons running in the bg that were handling the requests - no the instance I was testing. I found the issue, PR to follow.

source: gharchive/issue

authors: andrewpmartinez dariuszSki

license_type: permissive

license_source: github-api


text:
Awesome Tunneling

Useful north star material:
https://github.com/anderspitman/awesome-tunneling

closed with: https://github.com/anderspitman/awesome-tunneling/pull/82 :)

authors: dovholuknf michaelquigley

repo: openziti/zrok


text:
Creating ClusterCatalog hangs for a while, without any signal for what's going on.

I created a ClusterCatalog with a private image from redhat.registry.io, but it hung for a while without any signal about what's going on..
NAME                         LASTUNPACKED   SERVING   AGE
redhat-community-operators   1s             True      14m

Notice the last unpacked (1s) vs the age (14m).
When I inspected the logs, I saw that the unpacker ran into intermittent issues, before it was able to pull the image successfully. For the first 14m though, there was no signal on the CR for what's going on (you can see the unpacker struggling if you inspect the controller logs though).
We should restructure our controller to be a state machine, so that the unpacking operation can be transitioned through the different states, instead of the entire operation being an all-in blocking operation.
The different states for the transition machine could be something like:
Initiated->Pulling->Unpacking->Unpacked

I believe this is a result of synchronous image pulling. The fix for this should be to design and implement async image pulling. At that point, the reconciler could kick off a background goroutine to pull an image, and then immediately update status to say "Pulling" somehow.
Then later when the image pull is complete (or fails), we could signal the reconciler to tell it to reconcile again at which point it could find the result and continue unpacking/processing it.

@joelanford you'd mentioned this call but I don't think I picked up your points that well in the call:
What was the reason you mentioned again for not doing a state machine again? What you're saying here

At that point, the reconciler could kick off a background goroutine to pull an image, and then immediately update status to say "Pulling" somehow.
Then later when the image pull is complete (or fails), we could signal the reconciler to tell it to reconcile again at which point it could find the result and continue unpacking/processing it.

seems much easier to do if we just requeueAfter in each state, until we reach a "final" state (of success or error), with exponential backoff in case of errors.
This is a level-based design too.

we could signal the reconciler to tell it to reconcile again

sounds like you're suggesting an edge-based design.

Maybe I'm misinterpreting what you mean by a state machine design, but I've seen cases where controllers are implemented as state machines where they make assumptions that the status of an object is correct and then execute a state's logic based on that assumption. But if the status is incorrect, then the logic that is executed is also incorrect and/or incomplete, and that might mean we never successfully run other parts of the state machine that we think will always run.
The correct design of a controller is to read the actual and current state of the world and then do as much as possible to drive toward that desired state.

we could signal the reconciler to tell it to reconcile again

sounds like you're suggesting an edge-based design.

Not quite. Think of the async image puller a lot like the apiserver and our client that fetches from it, and the watches we register with it.

We ask it: "do you have the data we expect?" If so, great we use it and continue on with that data.
If it doesn't have the data we expect, we ask it to fetch that data in the background (kind of like how we used to create a unpack pod), and then update our status to say "don't have it yet, but it's being fetched"
Just like when the apiserver sends an event via an informer about a change in an object in etcd, we could have something like an informer against the async image puller, and we'd register that as a source.Channel (or similar) on our controller such that we would just get a generic event for "this catalog needs to be reconciled"

So then anytime we reconcile, no matter why we're reconciling, we're looking at current state (what is in our APIserver cache, what is in our image cache) and then taking action based on what we see there.

I think I see what you mean, let me try to find some time to prototype this. I'll update here once the prototype is ready.

repo: operator-framework/catalogd


text:
Update CI scripts to also support podman

There are current bugs with running docker on recent versions of fedora and rhel[1]. Can we make podman work as well? I tried making it work with just an alias, but it doesn't seem to like it:
$ alias docker=podman
$ docker
Error: missing command 'podman COMMAND'
Try 'podman --help' for more information.
$ make operator.test OP_PATH=upstream-community-operators/grafana-operator
Pulling docker image                              [  Processing  ]
scripts/ci/run-script: line 12: docker: command not found
Pulling docker image                              [  FAILED  ]
make: *** [Makefile:26: force_pull_image] Error 1

[1] https://github.com/docker/for-linux/issues/841

We are using new test scripts. More info https://operator-framework.github.io/community-operators/operator-test-suite/

authors: etsauer mvalarh

repo: operator-framework/community-operators


text:
Kill some blank line.

Description of the change:
Motivation for the change:
Reviewer Checklist

[ ] Implementation matches the proposed design, or proposal is updated to match implementation
[ ] Sufficient unit test coverage
[ ] Sufficient end-to-end test coverage
[X] Docs updated or added to /docs
[ ] Commit messages sensible and descriptive

@gyliu513 The quickstart manifests are generated from helm charts and the release make rule, which means your changes will be lost the next time a release is cut. In order to have them persist through the next release, you should update the helm charts.

Thanks @njhale , let me check.

source: gharchive/pull-request

license_source: bigquery


text:
Bundle validation fails if CSV contains multiple entries for one owned CRD at different api versions

Bug Report

What did you do?

Create a New Project

$ operator-sdk new someservice-operator --repo github.com/example/someservice-operator
INFO[0000] Creating new Go operator 'someservice-operator'.
INFO[0000] Created go.mod
INFO[0000] Created tools.go
INFO[0000] Created cmd/manager/main.go
INFO[0000] Created build/Dockerfile
INFO[0000] Created build/bin/entrypoint
INFO[0000] Created build/bin/user_setup
INFO[0000] Created deploy/service_account.yaml
INFO[0000] Created deploy/role.yaml
INFO[0000] Created deploy/role_binding.yaml
INFO[0000] Created deploy/operator.yaml
INFO[0000] Created pkg/apis/apis.go
INFO[0000] Created pkg/controller/controller.go
INFO[0000] Created version/version.go
INFO[0000] Created .gitignore
INFO[0000] Validating project
INFO[0004] Project validation successful.
INFO[0004] Project creation complete.

$ cd someservice-operator/

Create the initial example.com/v1beta1 API

$ operator-sdk add api --kind SomeService --api-version example.com/v1beta1
INFO[0000] Generating api version example.com/v1beta1 for kind SomeService.
INFO[0000] Created pkg/apis/example/group.go
INFO[0002] Created pkg/apis/example/v1beta1/someservice_types.go
INFO[0002] Created pkg/apis/addtoscheme_example_v1beta1.go
INFO[0002] Created pkg/apis/example/v1beta1/register.go
INFO[0002] Created pkg/apis/example/v1beta1/doc.go
INFO[0002] Created deploy/crds/example.com_v1beta1_someservice_cr.yaml
INFO[0002] Running deepcopy code-generation for Custom Resource group versions: [example:[v1beta1], ]
INFO[0011] Code-generation complete.
INFO[0011] Running CRD generator.
INFO[0011] CRD generation complete.
INFO[0011] API generation complete.

Mark this API as the stored version in pkg/apis/example/v1beta1/someservice_types.go as described here https://book.kubebuilder.io/multiversion-tutorial/api-changes.html#storage-versions

added +kubebuilder:storageversion
// SomeService is the Schema for the someservices API
// +kubebuilder:subresource:status
// +kubebuilder:resource:path=someservices,scope=Namespaced
// +kubebuilder:storageversion
type SomeService struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   SomeServiceSpec   `json:"spec,omitempty"`
    Status SomeServiceStatus `json:"status,omitempty"`
}

Add another API version for this resource

$operator-sdk add api --kind SomeService --api-version example.com/v1beta2
INFO[0000] Generating api version example.com/v1beta2 for kind SomeService.
INFO[0002] Created pkg/apis/example/v1beta2/someservice_types.go
INFO[0002] Created pkg/apis/addtoscheme_example_v1beta2.go
INFO[0002] Created pkg/apis/example/v1beta2/register.go
INFO[0002] Created pkg/apis/example/v1beta2/doc.go
INFO[0002] Created deploy/crds/example.com_v1beta2_someservice_cr.yaml
INFO[0002] Running deepcopy code-generation for Custom Resource group versions: [example:[v1beta1 v1beta2], ]
INFO[0019] Code-generation complete.
INFO[0019] Running CRD generator.
INFO[0020] CRD generation complete.
INFO[0020] API generation complete.

Generate a CSV

I end up with empty version strings here so I just edited the version from someservice-operator.v to somesevice-operator.v0.1.0 so that the bundle validation doesn't complain about it since it's irrelevant to this bug.
$ operator-sdk generate csv
INFO[0000] Generating CSV manifest version
INFO[0017] Fill in the following required fields in file someservice-operator.clusterserviceversion.yaml:
	spec.description
	spec.keywords
	spec.provider
INFO[0017] CSV manifest generated successfully

Build the bundle

$ operator-sdk bundle create -d deploy/olm-catalog/someservice-operator/manifests/ --generate-only
INFO[0000] Building annotations.yaml
INFO[0000] Building Dockerfile

Validate the bundle and observe the error

$ operator-sdk bundle validate deploy/olm-catalog/someservice-operator/
FATA[0000] Bundle content validation failed: Bundle validation errors: Error: Value someservices.example.com: owned CRD "someservices.example.com" not found in bundle "someservice-operator.v0.1.0"  bundle-dir=/Users/myusername/scratch/someservice-operator/deploy/olm-catalog/someservice-operator

The owned CRDs are listed as follows in the CSV
spec:
  apiservicedefinitions: {}
  customresourcedefinitions:
    owned:
    - description: SomeService is the Schema for the someservices API
      kind: SomeService
      name: someservices.example.com
      version: v1beta2
    - description: SomeService is the Schema for the someservices API
      kind: SomeService
      name: someservices.example.com
      version: v1beta1

What did you expect to see?
I expected the bundle to validate even if with a single a CRD exposing a kind at multiple versions.
Not sure if that means fixing how things are represented in the CSV (should we have multiple entries for one Group/Kind at multiple api-versions?) or if this is just an expectation that's incorrect in the bundle validation logic. Couldn't find documentation on it, so it could be documented and I just missed it.
What did you see instead? Under which circumstances?
The bundle validation complains (somewhat unclearly) that it cannot find a matching CRD yaml file in the bundle associated with an owned CRD.
Environment

operator-sdk version:
operator-sdk version: "v0.17.1", commit: "bb0c8f0108d1fd1be896d66c2d9e9044f10565a0", kubernetes version: "unknown", go version: "go1.13.10 darwin/amd64"

go version:

go version go1.13 darwin/amd64

Kubernetes version information:
N/A

Kubernetes cluster kind:
N/A
Are you writing your operator in ansible, helm, or go?
Go
Possible Solution

N/A
Additional context

@komish I think this is fixed in v0.18.0 in which we bumped our validation library version.
/triage support

@estroz Thanks! Just saw this come up in bug triage. I'll test and report back if anything else, or close if not.

[someservice-operator]$ operator-sdk version
operator-sdk version: "v0.18.0", commit: "921c33982687d6e9219fe0a1a326515fb37ae040", kubernetes version: "v1.18.2", go version: "go1.13.10 darwin/amd64"

@estroz Seems like bundle validate isn't seeing either CRD now?
[someservice-operator]$ operator-sdk bundle validate deploy/olm-catalog/someservice-operator/ --verbose
DEBU[0000] Debug logging is set
DEBU[0000] Found manifests directory                     bundle-dir=deploy/olm-catalog/someservice-operator container-tool=docker
DEBU[0000] Found metadata directory                      bundle-dir=deploy/olm-catalog/someservice-operator container-tool=docker
DEBU[0000] Getting mediaType info from manifests directory  bundle-dir=deploy/olm-catalog/someservice-operator container-tool=docker
INFO[0000] Found annotations file                        bundle-dir=deploy/olm-catalog/someservice-operator container-tool=docker
INFO[0000] Could not find optional dependencies file     bundle-dir=deploy/olm-catalog/someservice-operator container-tool=docker
DEBU[0000] Validating bundle contents                    bundle-dir=deploy/olm-catalog/someservice-operator container-tool=docker
ERRO[0000] Error: Value someservices.example.com: owned CRD "someservices.example.com" not found in bundle "someservice-operator.v0.0.1"
ERRO[0000] Error: Value someservices.example.com: owned CRD "someservices.example.com" not found in bundle "someservice-operator.v0.0.1"

The files
[someservice-operator]$ tree deploy/olm-catalog/someservice-operator/
deploy/olm-catalog/someservice-operator/
├── manifests
│   ├── example.com_someservices_crd.yaml
│   └── someservice-operator.clusterserviceversion.yaml
└── metadata
    └── annotations.yaml

2 directories, 3 files

I also tried this with only a single API version and it's not seeing the CRD. Something going on with the generation logic, or user error on my part that I'm missing?
[someservice-operator]$ operator-sdk bundle validate deploy/olm-catalog/someservice-operator/
INFO[0000] Found annotations file                        bundle-dir=deploy/olm-catalog/someservice-operator container-tool=docker
INFO[0000] Could not find optional dependencies file     bundle-dir=deploy/olm-catalog/someservice-operator container-tool=docker
ERRO[0000] Error: Value someservices.example.com: owned CRD "someservices.example.com" not found in bundle "someservice-operator.v0.0.1"

Can confirm this is a bug
/kind bug
/remove-triage support

There are at least 2 bugs in the validation library we use:

Only v1beta1 CRDs are validated and the SDK generates v1 CRDs by default, which is the source of this bug.
https://github.com/operator-framework/api/pull/32 was reverted, such that multiple CRD versions per CSV cause "duplicate CRD" errors.

I'll fix this upstream and release v0.18.1.

authors: estroz komish

repo: operator-framework/operator-sdk


text:
[ansible-operator] Status false when type/reason = Successful

Type of question

General operator-related help
Question
What did you do?
I develop an Ansible operator and let it manage its status (and I prefer that way - it will probably manage it better by itself).
However our testers recently complained about the following result:
    {
      "lastTransitionTime": "2023-07-25T07:14:46Z",
      "message": "Last reconciliation succeeded",
      "reason": "Successful",
      "status": "False",
      "type": "Successful"
    }

They would like to see "True" instead of "False" in the status field which would be more coherent with the actual results. (They use some script to check all statuses, my operator being just one amongst many, and wait for True I guess.)
What did you expect to see?
I'd rather expect something like:
    {
      "lastTransitionTime": "2023-07-25T07:14:46Z",
      "message": "Last reconciliation succeeded",
      "reason": "Successful",
      "status": "True",
      "type": "Successful"
    }

but maybe I missed something there.
What did you see instead? Under which circumstances?
We actually see status: False and don't understand why.
Environment
Operator type:

/language ansible

Kubernetes cluster type:
OpenShift
$ operator-sdk version
operator-sdk version: "v1.31.0"
$ kubectl version
Client Version: v1.28.1
Additional context

Hi @PYLochou Would you please provide the the full status for your custom resource you observed the above behavior on? From the operator-sdk code, it seems while reconcile is ongoing, the Running condition would be set to true while the Successful condition would be set to false.
I may need to see all the conditions in the status block to be able to aptly determine if we are looking at a bug.

Thank you for the explaination, Edmund. You're right, at this moment there was a reconciliation running:
"conditions": [
    {
      "lastTransitionTime": "2023-07-24T14:02:37Z",
      "message": "",
      "reason": "",
      "status": "False",
      "type": "Failure"
    },
    {
      "lastTransitionTime": "2023-07-25T07:14:46Z",
      "message": "Last reconciliation succeeded",
      "reason": "Successful",
      "status": "False",
      "type": "Successful"
    },
    {
      "lastTransitionTime": "2023-07-24T14:06:40Z",
      "message": "Running reconciliation",
      "reason": "Running",
      "status": "True",
      "type": "Running"
    }
  ]

I'll transmit your explaination to our testers, thank you!

Another closely-related issue, if I may...?
On another run (with buggy input in the CR, hence the errors), they got the following conditions:
"conditions": [
    {
      "lastTransitionTime": "2023-07-24T08:59:32Z",
      "message": "",
      "reason": "",
      "status": "False",
      "type": "Successful"
    },
    {
      "ansibleResult": {
        "changed": 0,
        "completion": "2023-07-24T13:23:28.4525",
        "failures": 1,
        "ok": 96,
        "skipped": 68
      },
      "lastTransitionTime": "2023-07-24T13:23:29Z",
      "message": "Could not find the following secrets ['ibm-cp4ba-db-ssl-secret-for-svl']. Create it or them before installing the cloudpak.",
      "reason": "Failed",
      "status": "False",
      "type": "Failure"
    },
    {
      "lastTransitionTime": "2023-07-24T13:23:34Z",
      "message": "Running reconciliation",
      "reason": "Running",
      "status": "True",
      "type": "Running"
    }
  ]

and this time, for the type Failure, they are waiting for status True (which may seem logical), is it again because the reconciliation is running?

Interesting find. So, it seems in the previous reconcile loop, the Failure condition was set to true and failure reason captured. However, on subsequent reconcile, the Failure condition is set to false and the Running condition was set to true.
I think, it is logical argument to leave the Failure  condition as previously set since it is accurate as of time of recording. However, when marking the reconcile to be complete / successful deployment, then the previous error could be reset.

Yes, I agree ("it is logical argument to leave the Failure condition as previously set since it is accurate as of time of recording") and remove it when the deployment gets successful. No False status should be set anytime on this one IMHO.

Roger that. I will table this in the next meeting and see if there is consensus to make the change. I may explore creating a PR for this in the interim.

The PR to solve this issue has been moved to the ansible-operator-plugins repository https://github.com/operator-framework/ansible-operator-plugins/pull/14

authors: OchiengEd PYLochou


text:
updating logrus dependency to latest to resolve CVE

Description of the change:
Updating logrus to latest version.
Motivation for the change:

Closes: #6502

Checklist
If the pull request includes user-facing changes, extra documentation is required:

[ ] Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
[ ] Add or update relevant sections of the docs website in website/content/en/docs

@everettraven You already addressed the go-restful issues in another PR, I linked that PR to the issue.

You already addressed the go-restful issues in another PR, I linked that PR to the issue.
@acornett21 Ah thanks - I totally forgot I did that :sweat_smile:


text:
several issues with captive portal

OPNSense 1.8.5, captive portal setup as in Wiki, but with IPv4+IPv6.

Browsing to HTTP site does not redirect me to captive portal if I am not signed in, instead opens site even if I am not signed in (example site).
Browsing to HTTPS site does not redirect me to captive portal if I am not signed in, instead times out (example site).

Browsing to http://guestaddress:8000 opens the captive portal sign-in page (and rewrites URL to http://guestaddress:8000/index.html?redirecturl=guestaddress).
After signing in, both the HTTP and HTTPS web sites can be reached.
After signing out, initial situation is back.
There are several recent reports in the forum about a non-working redirection to the captive portal sign-in page.

I set all firewall rules on guest net to IPv4 only. The behavior is the same.

I disabled IPv6 now. I also removed a NAT rule for the transparent proxy on the guest net - this rule seemingly caused most of the problem (actually I need to check the HTTP transparent proxy option in the captive portal zone settings).
The second issue remains, though: browsing to HTTPS site does not redirect me to captive portal if I am not signed in, instead times out.

Can this be concluded?

Will test in the next days.

Tested with OPNsense 18.7.9-amd64: Browsing to HTTPS site does not redirect me to captive portal if I am not signed in, instead times out.
Is this expected because captive portal does not capture HTTPS?

yes, it can't do a man in the middle, so your browser won't trust the certificate it got served from the CP

Conclusions:

Captive Portal does not support IPv6
Captive Portal redirects HTTP requests to login page but not HTTPS requests

authors: AdSchellevis borisneubert fichtner

repo: opnsense/core


text:
Suricata et-open-extra missing rules

Important notices
Our forum is located at https://forum.opnsense.org , please consider joining discussions there in stead of using GitHub for these matters.
Before you ask a new question, we ask you kindly to acknowledge the following:

[x] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
[x] I have searched the existing issues, open and closed, and I'm convinced that mine is new.

After upgrade to 24.1, I was expecting to see my PR #3644 working.
Despite the minor changes involved, 24.1 doesn't show all ET Open rules when installed, but only 3coresec and threatview_CS_c2 while the PR includes many more.
No clue why they are not populated on ui/ids#download_settings.

@0nnyx I assumed you tested the xml earlier to be honest, can you re-add the et_open- prefixes in the keys?
e.g
<file description="botcc.portgrouped" url="inline::rules/botcc.portgrouped.rules">botcc.portgrouped.rules</file>

should look like:
<file description="botcc.portgrouped" url="inline::rules/botcc.portgrouped.rules">et_open-botcc.portgrouped.rules</file>

@AdSchellevis Right, the prefix must be the cause. But which one is it exactly ? et_open- or et_open. ?
<file description="botcc.portgrouped" url="inline::rules/botcc.portgrouped.rules">et_open-botcc.portgrouped.rules</file>
<file description="botcc" url="inline::rules/botcc.rules">et_open.botcc.rules</file>

@0nnyx I would go for the et_open. variant as most of the previous ones used these.

Just did another PR #3795 after testing and now it's all fine.
It was my first PR here and I thought you would do some testing before approval.

repo: opnsense/plugins


text:
net/vnstat: make db persistEnt in MFS installation

Closes #1061

Merged, thanks!

authors: fichtner mimugmail


text:
Siproxd plugin

Siproxd plugin as requested in #246
In #246 are some screenshots how it looks like.
Current problem are the "Current registrations", the newlines are not correctly formatted. Perhaps you have some ideas? I could als try to write a script but a oneliner would be smarter I think.
setup.sh needs a chmod 755 as always when I contribute :)

you can use <textarea> or <pre> for monospace output

I removed the WIP, if you don't find anything else we could merge and let @NicholasRush test the whole thing.

Merged, have some time, will fix my comments in the tree


text:
Automatic "clear()" is annoying

I just ran the script to see the options and it just cleaned up my entire terminal. That's not cool :(

Ok, I'll remove the automatic clear.
Best regards,
oppsec.

authors: Nickguitar oppsec

repo: oppsec/juumla

license: MIT


text:
add support for HVM instances

When using the ebs_raid provider to create and attached volumes to a HVM instance you will receive the following error:
RightAws::AwsError
------------------
InvalidParameterValue: Value (/dev/sdi1) for parameter device is invalid. /dev/sdi1 is not a valid EBS device name.

This is because the ebs_raid provider hard-codes a digit after the device name (i.e. sdi1, sdi2, etc.)
http://docs.amazonaws.cn/en_us/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html states:

Hardware virtual machine (HVM) AMIs (such as the base Windows and Cluster Compute images) do not support the use of > trailing numbers on device names (xvd[a-p][1-15]).

Existing ticket:
https://tickets.opscode.com/browse/COOK-4600
My pull request addresses the issue by adding support for HVM instances without effecting the existing functionality of the provider because hvm is set to false by default.

+1 We need this or something like this.

As you can see above- we pulled this code into our own (public) fork, since we could no longer wait for this PR to get merged.  It's working great!
Please press forward with this merge, so that we can get back to the trunk and stop depending on our own fork.

Thanks @DrMerlin!

This is certainly something we need since most users are going to be on HVM instances at this point.  I don't think the user should have to specify HVM though.  We can / should detect that and just do the right thing.

I'm closing this out since we want to auto detect paravirt vs. HVM and do the right thing without the user having to know what kind of instance they're on. This would allow the provider to run on a mixed fleet. Feel free to open this back up if you can get support for both instance types without user knowledge of the instance type.

authors: DrMerlin jhauf-nativex tas50

repo: opscode-cookbooks/aws


text:
Re_run on SSH doesn't update the machine file in .chef/provisioning/ssh

Pretty sure the issue is in the ssh driver but we should get it fixed.

This is fixed in the latest update of the ssh driver which will be released tomorrow or thursday

@double-z Amazing news!! Lets test it as soon as it is released! :+1:

@double-z any update on this Zack? :smile:

authors: afiune double-z jonsmorrow

repo: opscode-cookbooks/delivery-cluster


text:
Refactor runit supervisor to use a resource

Behaves exactly the same as it did before, but using an LWRP instead of a series of recipes.
/cc @chef/visual-interaction @chef/lob

I'll review tomorrow

Looks like it needs a rebase.

@jtimberman rebased, added your suggestions, and removed all of the stuff related to the delete action. Please take a look.

LGTM

authors: jtimberman smith


text:
Documentation Revamp

Description
We should probably rework our documentation.
README.md
The README diagram is outdated as it presents our original vision of what the live indexing should be, and is missing the Access API. Maybe it could be interesting to look into ways to generate an SVG diagram from the code and embed it in the README instead, so that we can avoid the maintenance costs of that diagram.
The architecture part is outdated as well, and the milestones have been reordered when it comes to priority, multiple times so far, so I'm also not convinced they should be featured here anymore.
The dependencies and description are fine though. We could add some extra details on the different binaries in the repo and how they work, as well as links to the rest of the documentation (beyond the documentation button at the top which I'm pretty sure no one would ever notice/use :p).
Benchmark.md
Maybe this one could be generated rather than hand written, when we run the benchmarks? I think it could also make sense to add some context on this, describing how the benchmark is made. What sets of data are used, what operations we run, etc. As it stands, it is useful for us but would not be taken seriously from an external POV IMO. It should also provide the steps to follow to run the benchmark for anyone curious to verify it (even if it's as simple as running go benchmarks.)
Introduction.md
Overall I think this document is fine, as it's more about Flow than it is about the DPS. Maybe we should extract the Developer Guide part though, because it is currently buried deep within this document and is easy to miss. It could be its own document.
Architecture.md
This one probably needs some more love as well, but I'm worried about the maintenance costs it carries. We probably want to keep it as high-level as possible and just give an overview of what packages interact together and for what purpose. It currently only describes the core packages used for indexing, maybe it should also talk a bit about how the APIs depend on the DPS API, about packages related to Rosetta/Access API such as the Invoker for example, the storage package, the metrics, etc.
Snapshots.md
There is a dead link (#what-are-snapshots should be #what-are-index-snapshots) and some small issues we missed, but very little change is needed IMO.
API docs
Those are fine on a technical level, but it would probably make sense to give more info about how they work under the hood, in which case they should be used, etc. Since the DPS API docs are currently (partly) generated automatically, they do not contain anything besides endpoint docs. It would be nice to describe the API and its purpose within our architecture as well.
Missing docs
IMO, if we want our project to be accessible for external users, we probably want:

A contributing guide
A minimal code style guide (we can avoid this one but it will cost us more in PR review effort in the future. We can choose to only write this one once we need it, as long as we're confident we'll never end up in a situation where we must choose between business priorities and community support due to time constraints.)
Maybe some more? Feel free to comment on this issue if you have further suggestions (or even edit this description to add your suggestion to the list.)

Additional Notes
I'm making this an Epic for now, since I think it will have to be ongoing work over time and shouldn't be done in one massive PR. If we change the scope of the issue, it might no longer need to be an epic and that decision can be rolled back.

What else we need:

one document that shows how to orchestrate the different binaries;
the document to explain the Construction API CLI testing (#408).

repo: optakt/flow-dps


text:
secret nullresource trigger will run always

This PR is to change python to bash script to create secret and the trigger will make the null resource for secret to run every time.

Please redo from 4.x

authors: hyder karthicgit


text:
SSL: CERTIFICATE_VERIFY_FAILED - Downloading MY_ATP.pdb

I am getting "SSL: CERTIFICATE_VERIFY_FAILED" during db startup, could you please help.
Find below my full log,
[user@BRL72J3 ~]$ podman logs -t f2f6d724008ad02027d72c8c393a8221cafe9b5c3a1bbd08e0d6880f5a3f70fd
WARN[0000] Using cgroups-v1 which is deprecated in favor of cgroups-v2 with Podman v5 and will be removed in a future version. Set environment variable PODMAN_IGNORE_CGROUPSV1_WARNING to hide this warning.
2024-05-22T15:31:39.345767000-04:00 Archive:  /u01/POD1.zip
2024-05-22T15:31:39.347480000-04:00    creating: /u01/app/oracle/oradata/
2024-05-22T15:31:39.348648000-04:00    creating: /u01/app/oracle/oradata/POD1/
2024-05-22T15:31:39.349110000-04:00    creating: /u01/app/oracle/oradata/POD1/17204CBA6183008CE063C4D75E6492EC/
2024-05-22T15:31:39.349152000-04:00    creating: /u01/app/oracle/oradata/POD1/17204CBA6183008CE063C4D75E6492EC/datafile/
2024-05-22T15:31:39.349178000-04:00    creating: /u01/app/oracle/oradata/POD1/175E8D561B6D4CDEE0636402000A1C40/
2024-05-22T15:31:39.349644000-04:00    creating: /u01/app/oracle/oradata/POD1/175E8D561B6D4CDEE0636402000A1C40/datafile/
2024-05-22T15:31:39.442155000-04:00   inflating: /u01/app/oracle/oradata/POD1/redo01.log
2024-05-22T15:31:39.532134000-04:00   inflating: /u01/app/oracle/oradata/POD1/redo02.log
2024-05-22T15:31:39.532407000-04:00    creating: /u01/app/oracle/oradata/POD1/datafile/
2024-05-22T15:31:39.585161000-04:00   inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_temp_m33n3tv9_.tmp
2024-05-22T15:32:06.735885000-04:00   inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_system_m33lvr7m_.dbf
2024-05-22T15:32:06.814266000-04:00   inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_undotbs1_m33n3t3g_.dbf
2024-05-22T15:32:15.172292000-04:00   inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_sysaux_m33lz7xh_.dbf
2024-05-22T15:32:15.601036000-04:00   inflating: /u01/app/oracle/oradata/POD1/datafile/o1_mf_data_m33m2s61_.dbf
2024-05-22T15:32:15.618548000-04:00    creating: /u01/app/oracle/oradata/POD1/175E8F7D3A3D4D65E0636402000AA841/
2024-05-22T15:32:15.618700000-04:00    creating: /u01/app/oracle/oradata/POD1/175E8F7D3A3D4D65E0636402000AA841/datafile/
2024-05-22T15:32:15.660253000-04:00    creating: /u01/app/oracle/oradata/POD1/onlinelog/
2024-05-22T15:32:15.673523000-04:00    creating: /u01/app/oracle/oradata/POD1/171F1841E82CF4B3E063C4D75E643770/
2024-05-22T15:32:15.673625000-04:00    creating: /u01/app/oracle/oradata/POD1/171F1841E82CF4B3E063C4D75E643770/datafile/
2024-05-22T15:32:15.696223000-04:00    creating: /u01/app/oracle/oradata/POD1/175E81D3E11F4989E0636402000AD075/
2024-05-22T15:32:15.696298000-04:00    creating: /u01/app/oracle/oradata/POD1/175E81D3E11F4989E0636402000AD075/datafile/
2024-05-22T15:32:15.696324000-04:00    creating: /u01/app/oracle/oradata/POD1/controlfile/
2024-05-22T15:32:15.696351000-04:00   inflating: /u01/app/oracle/oradata/POD1/controlfile/o1_mf_m33m65kk_.ctl
2024-05-22T15:32:16.320877000-04:00 TIME ELAPSED Unzipping /u01/POD1.zip: 0 minutes and 37 seconds elapsed
2024-05-22T15:32:18.065494000-04:00 User input JSON not found
2024-05-22T15:32:18.128315000-04:00 MY ADB WORKLOAD_TYPE is ATP
2024-05-22T15:32:18.128971000-04:00 MY ADB CUSTOM NAME is MYATP
2024-05-22T15:32:18.151187000-04:00 BUILDER: Configuring TCPS
2024-05-22T15:32:18.152329000-04:00 BUILDER: Cleanup /u01/app/oracle/wallets/tls_wallet
2024-05-22T15:32:18.164291000-04:00 BUILDER: Creating auto login wallet for server
2024-05-22T15:32:18.456193000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:18.456334000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:18.456874000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:18.456945000-04:00
2024-05-22T15:32:18.489135000-04:00 Enter password:
2024-05-22T15:32:18.489506000-04:00 Enter password again:
2024-05-22T15:32:18.733760000-04:00 Operation is successfully completed.
2024-05-22T15:32:18.749360000-04:00 BUILDER: Creating a self-signed certificate using orapki utility; VALIDITY: 10 years
2024-05-22T15:32:18.909708000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:18.911516000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:18.911602000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:18.911624000-04:00
2024-05-22T15:32:19.240145000-04:00 Cannot modify auto-login (sso) wallet
2024-05-22T15:32:19.241408000-04:00 Enter wallet password:
2024-05-22T15:32:21.144281000-04:00 Operation is successfully completed.
2024-05-22T15:32:21.151481000-04:00 BUILDER: exporting server's cert
2024-05-22T15:32:21.347089000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:21.347278000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:21.347709000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:21.347776000-04:00
2024-05-22T15:32:22.043356000-04:00 Operation is successfully completed.
2024-05-22T15:32:22.059921000-04:00 BUILDER: exporting server's cert
2024-05-22T15:32:22.318691000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:22.319136000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:22.319746000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:22.319829000-04:00
2024-05-22T15:32:23.128622000-04:00 Operation is successfully completed.
2024-05-22T15:32:23.145920000-04:00 BUILDER: exporting encrypted private key
2024-05-22T15:32:23.437932000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:23.438636000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:23.438723000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:23.438760000-04:00
2024-05-22T15:32:23.453315000-04:00 Private key password:
2024-05-22T15:32:23.454948000-04:00 Enter password:
2024-05-22T15:32:23.469735000-04:00 Enter password again:
2024-05-22T15:32:41.651260000-04:00 Enter wallet password:
2024-05-22T15:32:41.979993000-04:00 Operation is successfully completed.
2024-05-22T15:32:41.994723000-04:00 BUILDER: exporting private and certificates together in PEM
2024-05-22T15:32:41.999597000-04:00 BUILDER: generating keystore.jks and truststore.jks
2024-05-22T15:32:42.152267000-04:00 Oracle PKI Tool Release 23.0.0.0.0 - Production
2024-05-22T15:32:42.152415000-04:00 Version 23.0.0.0.0
2024-05-22T15:32:42.152441000-04:00 Copyright (c) 2004, 2024, Oracle and/or its affiliates. All rights reserved.
2024-05-22T15:32:42.152480000-04:00
2024-05-22T15:33:00.351039000-04:00 Enter wallet password:
2024-05-22T15:33:00.687456000-04:00 Enter Key store password:
2024-05-22T15:33:00.688343000-04:00 Enter Trust store password:
2024-05-22T15:33:00.785159000-04:00 Operation is successfully completed.
2024-05-22T15:33:00.800159000-04:00 BUILDER: generating sqlnet.ora for client
2024-05-22T15:33:00.800328000-04:00 BUILDER: Generating  tnsnames.ora based on the new CN
2024-05-22T15:33:00.803804000-04:00 BUILDER: Overriding service names based on user input
2024-05-22T15:33:00.805435000-04:00 BUILDER: generating ojdbc.properties
2024-05-22T15:33:00.805852000-04:00 BUILDER: zipping wallet for ORDS
2024-05-22T15:33:00.814854000-04:00 updating: README (stored 0%)
2024-05-22T15:33:00.814996000-04:00 updating: adb_container.cert (deflated 24%)
2024-05-22T15:33:00.815033000-04:00 updating: cwallet.sso (stored 0%)
2024-05-22T15:33:00.815062000-04:00 updating: cwallet.sso.lck (stored 0%)
2024-05-22T15:33:00.815108000-04:00 updating: ewallet.p12 (stored 0%)
2024-05-22T15:33:00.815139000-04:00 updating: ewallet.p12.lck (stored 0%)
2024-05-22T15:33:00.815194000-04:00 updating: ewallet.pem (deflated 27%)
2024-05-22T15:33:00.815272000-04:00 updating: keystore.jks (stored 0%)
2024-05-22T15:33:00.815342000-04:00 updating: ojdbc.properties (deflated 49%)
2024-05-22T15:33:00.815451000-04:00 updating: sqlnet.ora (deflated 16%)
2024-05-22T15:33:00.815494000-04:00 updating: tnsnames.ora (deflated 87%)
2024-05-22T15:33:00.815539000-04:00 updating: truststore.jks (deflated 5%)
2024-05-22T15:33:00.817348000-04:00 TIME ELAPSED Wallet Generation: 0 minutes and 42 seconds elapsed
2024-05-22T15:33:01.258150000-04:00 User has requested to download '.pdb' archive file from Object Storage bucket
2024-05-22T15:33:01.258531000-04:00 Downloading MY_ATP.pdb..
2024-05-22T15:33:01.267569000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.267744000-04:00   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
2024-05-22T15:33:01.267786000-04:00     chunked=chunked)
2024-05-22T15:33:01.267823000-04:00   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request
2024-05-22T15:33:01.267860000-04:00     self._validate_conn(conn)
2024-05-22T15:33:01.267895000-04:00   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in validate_conn
2024-05-22T15:33:01.267931000-04:00     conn.connect()
2024-05-22T15:33:01.267972000-04:00   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 358, in connect
2024-05-22T15:33:01.268007000-04:00     ssl_context=context)
2024-05-22T15:33:01.268043000-04:00   File "/usr/lib/python3.6/site-packages/urllib3/util/ssl.py", line 354, in ssl_wrap_socket
2024-05-22T15:33:01.268078000-04:00     return context.wrap_socket(sock, server_hostname=server_hostname)
2024-05-22T15:33:01.268153000-04:00   File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket
2024-05-22T15:33:01.268195000-04:00     _context=self, _session=session)
2024-05-22T15:33:01.268226000-04:00   File "/usr/lib64/python3.6/ssl.py", line 810, in init
2024-05-22T15:33:01.268259000-04:00     self.do_handshake()
2024-05-22T15:33:01.268360000-04:00   File "/usr/lib64/python3.6/ssl.py", line 1070, in do_handshake
2024-05-22T15:33:01.268424000-04:00     self._sslobj.do_handshake()
2024-05-22T15:33:01.268462000-04:00   File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake
2024-05-22T15:33:01.268491000-04:00     self._sslobj.do_handshake()
2024-05-22T15:33:01.268526000-04:00 ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
2024-05-22T15:33:01.268564000-04:00
2024-05-22T15:33:01.268594000-04:00 During handling of the above exception, another exception occurred:
2024-05-22T15:33:01.268623000-04:00
2024-05-22T15:33:01.268650000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.268677000-04:00   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
2024-05-22T15:33:01.268709000-04:00     timeout=timeout
2024-05-22T15:33:01.268737000-04:00   File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
2024-05-22T15:33:01.268762000-04:00     _stacktrace=sys.exc_info()[2])
2024-05-22T15:33:01.268798000-04:00   File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
2024-05-22T15:33:01.268826000-04:00     raise MaxRetryError(_pool, url, error or ResponseError(cause))
2024-05-22T15:33:01.268855000-04:00 urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
2024-05-22T15:33:01.269425000-04:00
2024-05-22T15:33:01.269500000-04:00 During handling of the above exception, another exception occurred:
2024-05-22T15:33:01.269540000-04:00
2024-05-22T15:33:01.269567000-04:00 Traceback (most recent call last):
2024-05-22T15:33:01.269588000-04:00   File "/u01/scripts/download_my_container_pdb.py", line 102, in
2024-05-22T15:33:01.269607000-04:00     downloader.download()
2024-05-22T15:33:01.269634000-04:00   File "/u01/scripts/download_my_container_pdb.py", line 61, in download
2024-05-22T15:33:01.269655000-04:00     r = requests.get(download_url, stream=True)
2024-05-22T15:33:01.269676000-04:00   File "/usr/lib/python3.6/site-packages/requests/api.py", line 75, in get
2024-05-22T15:33:01.269696000-04:00     return request('get', url, params=params, **kwargs)
2024-05-22T15:33:01.269718000-04:00   File "/usr/lib/python3.6/site-packages/requests/api.py", line 60, in request
2024-05-22T15:33:01.269738000-04:00     return session.request(method=method, url=url, **kwargs)
2024-05-22T15:33:01.269760000-04:00   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 535, in request
2024-05-22T15:33:01.269781000-04:00     resp = self.send(prep, **send_kwargs)
2024-05-22T15:33:01.269799000-04:00   File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 648, in send
2024-05-22T15:33:01.269820000-04:00     r = adapter.send(request, **kwargs)
2024-05-22T15:33:01.269840000-04:00   File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
2024-05-22T15:33:01.269861000-04:00     raise SSLError(e, request=request)
2024-05-22T15:33:01.269884000-04:00 requests.exceptions.SSLError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

@sxm525
Does this always happen during container start ? I don't expect to see this failure.

Yes, I tried multiple times and it failed with the same error.

Which Operating system ?

What is the podman version ?

podman version

Are you running podman in root or rootless mode ?

What is the output of following curl command on the host machine ?

curl https://objectstorage.us-phoenix-1.oraclecloud.com:443

OS - Windows 10
podman version - 5.0.3
Podman running in rootless mode.
Find below the curl output,
curl https://objectstorage.us-phoenix-1.oraclecloud.com:443
{"code":"NotFound","message":"Not Found"}

Could you SSH to the podman virtual machine and then try the same curl request ?
podman machine ssh
curl https://objectstorage.us-phoenix-1.oraclecloud.com:443

What is the container start command used ?

Find below my curl output from podman machine,
[user@BR~]$ curl https://objectstorage.us-phoenix-1.oraclecloud.com:443
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
I used below syntax,
podman run -d
-p 1521:1522
-p 1522:1522
-p 8443:8443
-p 27017:27017
-e WORKLOAD_TYPE=ATP
-e WALLET_PASSWORD=***
-e ADMIN_PASSWORD=***
--cap-add SYS_ADMIN
--device /dev/fuse
--name adb-free
ghcr.io/oracle/adb-free:latest-23ai

The issues seems to be that on the podman virtual machine which is running on the Windows host, there is no TLS CA bundle to verify the certificate's validity.  Could you connect to any host using HTTPS on the podman VM ?
I don't have a Windows machine myself but have been trying to find someone to reproduce this. This does not happen on Linux or MacOS.
Another option could be download the root CA for objectstorage.us-phoenix-1.oraclecloud.com and add it to the podman VM's truststore
sudo cp <oci-root-ca.pem> /etc/pki/ca-trust/source/anchors
sudo update-ca-trust

I copied the PEM file and tried but same issue. Build failed during download "MY_ATP.pdb" file.
But I am able to download this file from windows machine & podman machine without any issue. Find below my output.
This is failing only build runtime,
ERROR,
#############################################
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='objectstorage.us-phoenix-1.oraclecloud.com', port=443): Max retries exceeded with url: /n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))
######################################
Find below my curl out for file MY_ATP.pdb
Windows machine,
##############
C:\Users>curl --output MY_ATP.pdb https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  363M  100  363M    0     0  3405k      0  0:01:49  0:01:49 --:--:-- 2838k
Podman machine,
############
[user@BRL72J3 ~]$ curl --output MY_ATP.pdb https://objectstorage.us-phoenix-1.oraclecloud.com/n/dwcsdev/b/adb-free-23c/o/ADBS-24.4.4.2-23ai/MY_ATP.pdb
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
24  363M   24 87.6M    0     0  2674k      0  0:02:19  0:00:33  0:01:46 2449k

Is there a fix for this. I'm getting the same results even though i added all certificates to the VM.
I can fetch the image from the VM but not the container during startup.
I'm running podman 5.0.3 and RPM 4.19.1.1
I'm not able to update the OS packages since rpm-ostree is not present, just regular rpm.

@henrikedsparr @sxm525
After some reading and consulting with users using Windows, It is recommended to use Window Subsystem for Linux (WSL2)
Please refer the link : https://medium.com/@sociable_flamingo_goose_694/setup-wsl-for-local-docker-development-on-windows-f0767e0a72d4 to setup WSL2 and install docker engine.
This will start an Linux VM on your Windows host using WSL in which the adb-container can run using docker.

@aosingh Thanks for the information. Yesterday i managed to solve it by building my own container image and adding the certificates to /etc/pki/ca-trust/source/anchors and running update-ca-trust.
So if you are using Zscaler you need to add certificates to the container this way for it to work.

can this be closed?

Yes, please close it.
Thanks
[Graphical user interface  Description automatically generated]
Sudhan Madhavan
The Sherwin-Williams Company
Cell:216-906-0489
@.@.>
From: Alexander Escamilla @.>
Sent: Friday, September 20, 2024 1:09 PM
To: oracle/adb-free @.>
Cc: Sudhan Madhavan @.>; Mention @.>
Subject: [EXTERNAL] Re: [oracle/adb-free] SSL: CERTIFICATE_VERIFY_FAILED - Downloading MY_ATP.pdb (Issue #22)
[Caution] External email. Be sure you trust or verify the sender before entering usernames or passwords when prompted by a link.
can this be closed?
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/oracle/adb-free/issues/22*issuecomment-2364159189__;Iw!!Cg_6rE7FVGHU6vd7!4qgoi8HBvX10sUKC0pd7vTU9qKuVKrhKHNfZXj9kcW9c7B3CLlZNd3NWLGl6OqmMSDiySGMjb2owPFSWH3VUWY7UtaHgHA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ASYZJAURHAGEU2Q565IITX3ZXRJBHAVCNFSM6AAAAABIEJEC4GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRUGE2TSMJYHE__;!!Cg_6rE7FVGHU6vd7!4qgoi8HBvX10sUKC0pd7vTU9qKuVKrhKHNfZXj9kcW9c7B3CLlZNd3NWLGl6OqmMSDiySGMjb2owPFSWH3VUWY47VZW-Rw$.
You are receiving this because you were mentioned.Message ID: @.@.>>

authors: alexesca aosingh henrikedsparr sxm525

repo: oracle/adb-free


text:
[Bug] table materialization fails when another MV with the same name exists in database

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
the table materialization for a table fails if there exists a Materialized view with the same name in a different schema.
the original table is not renamed to backup and the renaming of the new table fails on "an object with the same name already exists in the schema".
the problem is with the "oracle__list_relations_without_caching" macro
in https://github.com/oracle/dbt-oracle/blob/main/dbt/include/oracle/macros/adapters.sql.
it filters out tables which share a name with a materialized view in the whole database instead of just the relation schema.
Expected Behavior
it should be fixed to
{% macro oracle__list_relations_without_caching(schema_relation) %}
  {% call statement('list_relations_without_caching', fetch_result=True) -%}
    with tables as
      (select SYS_CONTEXT('userenv', 'DB_NAME') table_catalog,
         owner table_schema,
         table_name,
         case
           when iot_type = 'Y'
           then 'IOT'
           when temporary = 'Y'
           then 'TEMP'
           else 'BASE TABLE'
         end table_type
       from sys.all_tables
       where upper(table_name) not in (
          select upper(mview_name)
          from sys.all_mviews
          where upper(owner) = upper('{{ schema_relation.schema }}')
)
       union all
       select SYS_CONTEXT('userenv', 'DB_NAME'),
         owner,
         view_name,
         'VIEW'
       from sys.all_views
       union all
       select SYS_CONTEXT('userenv', 'DB_NAME'),
        owner,
        mview_name,
        'MATERIALIZED VIEW'
        from sys.all_mviews
  )
  select table_catalog as "database_name"
    ,table_name as "name"
    ,table_schema as "schema_name"
    ,case table_type
      when 'BASE TABLE' then 'table'
      when 'VIEW' then 'view'
      when 'MATERIALIZED VIEW' then 'materialized_view'
    end as "kind"
  from tables
  where table_type in ('BASE TABLE', 'VIEW', 'MATERIALIZED VIEW')
    and upper(table_schema) = upper('{{ schema_relation.schema }}')
  {% endcall %}
  {{ return(load_result('list_relations_without_caching').table) }}
{% endmacro %}

Steps To Reproduce
No response
Relevant log output using --debug flag enabled
No response
Environment
- OS: Debian Gnu/Linux 11 (bullseye)
- Python: 3.9.16
- dbt: 1.6, 1.7

What Oracle database version are you using dbt with?
19c
Additional Context
No response

Ah, i remember adding this filter for a reason. Let me try to reproduce it.

i think its because materialized views are always created as an MV and table, probably when you added the support for mv you didnt want the mv to show up as a table(maybe because of the special drop statement needed to drop them?), so the filter was added, i guess it was an overlook to not filter only MVs in the schema the relation should be materialized.

Hi @push2prod
Thank you for reporting the issue and also suggesting the fix. I have included the fix in the next release candidate version
pip install dbt-oracle==1.7.2rc1

Could you please test with this and let me know if it works ?

I checked the fixed version on the macro independently and it works great, but for now can't test version 1.7 because of cluster issues. But seems good to me!
Would welcome a 1.6 release as well

dbt-oracle==1.7.2 is released with the fix.
We are also working on a backport for 1.6. I will let you know once it is released

repo: oracle/dbt-oracle