text:
Workaround for wrong errno on QNX for mq_timedsend/mq_timereceive results in error message in latest QNX

Required information
Operating system:
QNX SDP7.1
Compiler version:
QCC 8
Details
On QNX SDP7.0 the errno for a timeout in mq_timedsend/mq_timedreceive was EINTR instead of ETIMEDOUT. To fix this, the posix::MessageQueue has a TIMEOUT_ERRNO constexpr which is set to EINTR on QNX and ETIMEDOUT on all other platforms.
Since we switched to unix domain sockets and the mqueue is not used, the class is not used. We need to decide whether we want to just use ETIMEDOUT and ignore QNX SDP 7.0 or add a compiler switch like #ifdef __QNX__ && __GCC__MAJOR_VERSION__ == 5. Since this class is not used anyway, removing the #ifdef might be the best option.

@elBoberido @dkroenke  I would suggest to add mq_timedsend and mq_timedreceive to the iox platform files (include/mqueue.hpp) as iox_mq_timesend and iox_mq_timedreceive (like we already do for iox_mq_open) and correct the behavior directly there and set the errno like it is stated in the POSIX manual.
Then this behaves identical on every platform and we can get rid of macros, compiler checks and ifdefs in the production code.
This should also be our default approach whenever we discover platform specific code/macros inside the production code (except tests) to move that into the platform files.

@elfenpiff that's not quite the question here. The question is if we want to remove the workaround for SDP7.0 which means to handle this with the errno like it is stated in the POSIX manual or if we want to keep the workaround but tie it specifically to SDP7.0. Since this class is currently not used in iceoryx after the switch to unix domain sockets, we could take the easy route and remove the workaround and mark this as known limitation for this class on SDP7.0

@elBoberido I would invest a little bit more time here since in the future someone may be interested in the hoofs as a standalone library and would like to use the message queue. Then it would be nice that it is working.
But this is only when we are talking about maybe some more hours and not days.
As addendum to my previous comment: I would remove TIMEOUT_ERRNO from the message queue and handle everything consistently in the iox_mq_timed{send,receive} functions.
Or maybe I got the problem wrong :/

Developer Meeting: iceoryx posh doesn't use the message queue anymore. So this is only a problem if someone uses the abstraction for message queue standalone. Therefore low prio for now

authors: budrus elBoberido elfenpiff

repo: eclipse-iceoryx/iceoryx


text:
if there is already an existing ca.pem and netbird certificate, new certificates from setup string cannot be used in EDGAR managed setup

if there is already an existing ca.pem and netbird certificate, new certificates from setup string cannot be used in EDGAR managed setup. Workaround: manually delete certificates on peer

Fixed that behaviour in #171

source: gharchive/issue

repo: eclipse-opendut/opendut

license_type: permissive

license_source: github-api


text:
Pass correct request body in materialize provider

Fix https://github.com/eclipse-symphony/symphony/issues/155

materialize.go passs InstanceState, TargetState and SolutionState to API request body
catalogprovider.go pass CatalogState to UpsertCatalog function
remove duplicate UpsertTarget function in symphony-api.go (use CreateTarget instead)
Rename SiteManager functions to GetState and ListState
Update multisite scenario using catalog-catalog.yaml

Tests:

Unit tests
Multi-site manual test with catalog synchronization and remote jobs

https://github.com/eclipse-symphony/symphony/blob/f8d82c7f4480ebe54cd2a6554df5cf3ca7841567/api/pkg/apis/v1alpha1/providers/stage/materialize/materialize.go#L164-L165
I think we need to also fix this lines, use catalog, not catalog.Spec in json.Marshal.

@Haishi2016 gentle ping for review.

source: gharchive/pull-request

authors: RemindD msftcoderdjw

repo: eclipse-symphony/symphony

license: MIT


text:
OP - BPDM Orchestrator Component (Concept, Architecture, Implementation) (ext: CXAR-1020)

+Benefit Hypothesis & Problem Statement+

As an operator, I need a way to process Business Partner data records based on different states and to be independent of the service provider used, so that I could replace it in a modular way or build the process steps in a more modular way.
The proposed way is by implementing an Orchestrator Layer that acts as a passive component and offers for each processing steps individual endpoints and sets according to this the current state for the data record for further processing. The processing of data will be in chunks.
(Chunks are necessary when thinking about checking for duplicates.)
As an Architect, I have the requirement, that the orchestrator component is flexible designed, so that new "states" for the data processing can easily added.
Following diagram illustrates the processing flow and interaction between the components.
sequenceDiagram
    Gate->>Orchestrator: Upload new or changed BP
    Orchestrator->>Orchestrator: Set State: new
    Orchestrator->>Orchestrator: Set External ID / Request ID: (Linkage)
    Orchestrator-->>Gate: Send Req ID
    loop Every Hour
        Simulator->>Orchestrator: Fetch for "new"
        Orchestrator-->>Simulator: BP with "new" State
    end
    Simulator->>Simulator: Duplicate Check
    Simulator->>Simulator: Natural Person Screening
    Simulator->>Simulator: Curation
    Simulator->>Simulator: Enrichment
    opt BPN does not exist
        Simulator->>Pool: Create new BP
        Pool->>Pool: Create BPN
        Pool-->>Simulator: BPN
    end
    Simulator-->>Orchestrator: Cleaned BP with BPN
    Orchestrator->>Orchestrator: Set State: CLEANED
    loop Every Hour
        Pool->>Orchestrator: Fetch "CLEANED" BP
        Orchestrator-->>Pool: BP with "CLEANED" State
    end
    Pool->>Pool: Update BP
    Pool->>Orchestrator: Commit
    Orchestrator->>Orchestrator: Set State: "FINISH"
    loop Every Hour
        Gate->>Orchestrator: Fetch for "FINISH" && "ReqID"
        Orchestrator-->>Gate: BP that match "FINISH" && "ReqID"
        Gate-->>Gate: Update BP
    end
    loop Every Hour
        Gate->>Pool: Fetch Changelog
        Pool-->>Gate: Send Changelog with BPNs
        Gate->>Gate: Filter for relevant BPNs where timestamp is newer than existing
        Gate->>Pool: Fetch changed BP
        Pool-->>Gate: Send changed BP
        Gate->>Gate: Update changelog for BPNs
    end

Components: BPDM_GR
Sprints: N/A
Fix Versions: N/A
StoryPoints: N/A
Attachments: CXAR-1020-mermaid-diagram-2023-07-19-144421.svg

Updated Proposal to introduce Orchestration of Sharing Service Process:
sequenceDiagram
Gate->>GateDiscovery: Register
GateDiscovery-->>Gate: Commit
SharingServiceProvider->>GateDiscovery: Ask for Gates/URLs
GateDiscovery-->>SharingServiceProvider: Gates URLS
SharingMember->>Gate: Upload Business Partner Input
Gate->>Gate: Set State new
Gate-->>SharingMember: Commit
    loop Every Hour
        SharingServiceProvider->>Gate: Fetch for "new"
        Gate-->>SharingServiceProvider: BP with "new" State
    end
    SharingServiceProvider->>SharingServiceProvider: Duplicate Check
    SharingServiceProvider->>SharingServiceProvider: Natural Person Screening
    SharingServiceProvider->>SharingServiceProvider: Curation
    SharingServiceProvider->>SharingServiceProvider: Enrichment
    opt BPN does not exist
        SharingServiceProvider->>Pool: Get new BPN
        Pool->>Pool: Create BPN
        Pool-->>SharingServiceProvider: BPN
    end
    SharingServiceProvider-->>Gate: Write Output Cleaned BP with BPN
    Gate->>Gate: Set State: CLEANED
    Gate->>Pool: Upsert Business Partner
    Pool-->>Gate: Commit
    Gate->>Gate: Set State: "FINISH"
    loop Every Hour
        Gate->>Pool: Fetch Changelog
        Pool-->>Gate: Send Changelog with BPNs
        Gate->>Gate: Filter for relevant BPNs where timestamp is newer than existing
        Gate->>Gate: Set Business Partner state 'TO_UPDATE'
    end
    loop Every Hour
        SharingServiceProvider->>Gate: Fetch for "TO_UPDATE"
        Gate-->>SharingServiceProvider: BP with "TO_UPDATE" State
SharingServiceProvider->>SharingServiceProvider: Perform Cleaning
    end
SharingServiceProvider->>Gate: Write Output Cleaned BP with BPN (no Pool upsert needed)
Gate->>Gate: Set BP state "FINISH"

A variant communication model with an intermediate event queue in place:
sequenceDiagram
SharingMember->>Gate: Upload Business Partner Input
Gate->>Gate: Set State new
Gate-->>SharingMember: Created Business Partner Input
Gate->>EventQueue: Topic "New BP": Send Business Partner Input
EventQueue->>ServiceProvider:  Send Business Partner Input
ServiceProvider->>ServiceProvider: Clean and Process Business Partner
ServiceProvider->>EventQueue: Topic "Processed New BP": Send Cleaned Business Partner
EventQueue->>Gate: Send Cleaned Business Partner
Gate->>Gate: Set State: CLEANED_STEP_1
opt BPN does not exist
    Gate->>Pool: Ask for BPN by BPN-Request-ID
    Pool-->Gate: Return BPN for BPN-Request-ID
end
Gate->>Gate: Set State: CLEANED_STEP_2
Gate->>Pool: Upsert Business Partner
Pool-->>Gate: Upserted Business Partner
Gate->>Gate: Set State: "FINISH"
Pool->>EventQueue:  Topic "Pool BP Update": Send Pool Business Partner
EventQueue->>Gate: Get Pool Business Partner
Gate->>Gate: Filter for relevant BPNs where timestamp is newer than existing
Gate->>Gate: Set Business Partner state 'TO_UPDATE'
Gate->>EventQueue: Topic "TO_UPDATE_BP": Send Input Business Partner

Variant for orchestration showcasing a central orchestrator component responsible for updating the Pool and being the access point for the Gates. Where possible introduces the principle of pushing changes to the next service, with the exception of the Gates which still need to query for the final result.
sequenceDiagram
SharingMember->>Gate: Upload Business Partner Input
Gate->>Gate: Update business partner input
Gate-->>SharingMember: Created Business Partner Input
Gate->>Orchestrator: POST new cleaning request with mode HAS_UPDATE
Orchestrator->>Orchestrator: Create Cleaning Request with ID
Orchestrator->>Orchestrator: Set Cleaning Request State: TO_CLEAN
Orchestrator-->>Gate: Created cleaning request
Gate->>Gate: Save cleaning request ID for business partner
Orchestrator->>DummyBridge: POST process cleaning request
opt BPN does not exist
    DummyBridge->>DummyBridge: Create new BPN request ID
end
DummyBridge-->>Orchestrator: POST cleaning request result
Orchestrator->>Orchestrator: Set Cleaning Request State: TO_PUBLISH
Orchestrator->>Pool: POST new business partners without BPN
Pool-->>Orchestrator: Created business partners
Orchestrator->>Pool: PUT update business partners with BPN
Orchestrator->>Orchestrator: Set Cleaning Request State: FINISHED
loop Every Hour
        Gate->>Orchestrator: Fetch finished cleaning requests with request ID
        Orchestrator-->>Gate: Finished cleaning request
        Gate-->>Gate: Update Business Partner output data
end
loop Every Hour
        Gate->>Pool: Fetch Changelog
        Pool-->>Gate: Send Changelog with BPNs
        Gate->>Gate: Filter for relevant BPNs where timestamp is newer than existing
        opt newer timestamp for BPN found
            Gate->>Orchestrator: POST new cleaning request with mode ONLY_CLEAN
            Orchestrator->>Orchestrator: Cleaning process (see steps above)
        end
 end

The orchestrator plays the central role here as it manages identifiable cleaning requests. Further the orchestrator offers the Gates information about the requests, pushes those requests to the Dummy Bridge (later the cleaning process provider) and pushing cleaning results to the Pool.

How implement the orchestration in BPDM coming forward should be an open discussion. There are several ways to connect Gates and Pool together with an exchangable cleaning service. I will provide some proposals concentrating on advantages and disadvantages.
I'm open to feedback and counter-proposals so we can determine how to go forward with this feature.

Variant 1: Orchestrator Service with Pollable APIs
Heart of this variant is a new orchestration service that offers REST API endpoints for the other BPDM services to consume. The BPDM Gates as well as the Pool can connect to this orchestration service. In addition, the cleaning service (this will be a dummy implementation for now) will also connect with this specialized orchestration service. The so called orchestrator manages the state of the cleaning requests and handles offering the cleaning requests in its endpoint queues. The orchestrator is designed as being passive. The respective information for each service is accessed by authorized endpoint through polling mechanisms. The gates would also poll the Pool about udpates via the changelog.
sequenceDiagram
    SharingMember->>Gate: Upsert business partner A input
    Gate-->>Gate: Persist business partner A input
    Gate-->>SharingMember: Upserted business partner A input
    Gate->>Orchestrator: Request cleaning of business partner A input
    Orchestrator-->>Orchestrator: Create timestamped Cleaning Request B for business partner A input
    Orchestrator-->>Orchestrator: Enqueue in 'Generic Cleaning' Queue
    Orchestrator-->>Gate: Created cleaning request B
    Gate-->>Gate: Associate cleaning request ID with business partner A
    Gate-->>Gate: Upsert sharing state of business partner A
    loop Every X Minutes
        Gate->>Orchestrator: Ask cleaning request B state
        Orchestrator-->>Gate: State of cleaning request B
        Gate-->>Gate: Update sharing state
    end
    loop Every X Minutes
        CleaningServiceDummy->>Orchestrator: Fetch and reserve next cleaning requests in 'Generic Cleaning' queue
        Orchestrator-->>CleaningServiceDummy: Cleaning request B
        CleaningServiceDummy-->>CleaningServiceDummy: Perform dummy cleaning
        opt BPN does not exist
            CleaningServiceDummy->>CleaningServiceDummy: Add BPN request IDs
        end
        CleaningServiceDummy-->>Orchestrator: Send cleaning result
        Orchestrator-->>Orchestrator: Enqueue in 'BPN Processing' queue
        Orchestrator-->>CleaningServiceDummy: Commit
    end
    loop Every X Minutes
        Pool->>Orchestrator: Dequeue next cleaning requests in 'BPN Processing' queue
        Orchestrator-->>Pool: Cleaning request B cleaning result
        Pool-->>Pool: Upsert business partners with result
        Pool-->>Pool: Add BPNs from BPN request IDs
        Pool->>Orchestrator: Send cleaning result
        Orchestrator-->>Orchestrator: Commit
        Orchestrator-->Orchestrator: Enqueue in 'FINISHED' queue
    end
    loop Every X Minutes
        Gate->>Orchestrator: Dequeue cleaning requests by request ID in 'FINISHED' queue
        Orchestrator-->Gate: Cleaning request B cleaning result
        Gate-->>Gate: Resolve cleaning request B
        Gate-->>Gate: Update sharing state of business partner A
    end
    loop Every X Minutes
        Gate->>Pool: Fetch BPN changelog entries from last timestamp
        Pool-->>Gate: Entries from given timestamp
        opt Business Partner in Gate associated with BPN and newer timestamp
            Gate->>Gate: Start new cleaning process (see above)
        end
    end

Advantages:

Communication between well known BPDM services whose APIs can be standardised
Gates can't access requests and data from other Gates
Cleaning services only communicate with the orchestrator directly

Disadvantages:

Polling mechanisms introduce overhead
Polling mechanisms are not real-time
Polling can be replaced by an additional event queue signalling when new updates are available. But this introduces overhead to manage accessing orchestrator APIs and integrating an event queue at the same time

Variant 2 - Orchestration via Event Queue
Here the RabbitMq replaces a separate orchestration service. Cleaning request processing are realised by chaining events that are generated and consumed by the Gates, Pool and Cleaning service(s). Gates here are responsible for finding a unique request ID. A UUIDv4 together with a timestamp should mitigate any possible conflicts between Gates.  Requests are pushed into the respective topics of the queue and so passed along from Gate, Cleaning Service, Pool and back to Gate. The Gates can stay informed about the sharing state by subscribing to the cleaning process step topics and updating their sharing state accordingly.
sequenceDiagram
SharingMember->>Gate: Upsert Business partner A
Gate-->>Gate: Persist business partner input A
Gate-->>Gate: Create timestamped Cleaning Request B for business partner A input
Gate-->>SharingMember: Created business partner A input
Gate->>RabbitMq: Send cleaning request B to topic 'New Cleaning Request'
RabbitMq-->>Gate: Commit
RabbitMq->>CleaningServiceDummy: Send cleaning request B
CleaningServiceDummy-->>RabbitMq: Commit
opt Input has no BPNs
    CleaningServiceDummy-->>DummyBridge: Create unique BPN request IDs
end
CleaningServiceDummy-->>CleaningServiceDummy: Create Cleaning Result for cleaning request B
CleaningServiceDummy->>RabbitMq: Send cleaning result for B to topic 'New Process Step Generic Cleaning Result'
RabbitMq-->>CleaningServiceDummy: Commit

RabbitMq->>Pool: Send cleaning result B
Pool-->>RabbitMq: Commit
Pool-->>Pool: Upsert LSA business partner based on BPNs/BPN-Request-IDs
Pool->>RabbitMq: Send  update result for B to topic 'New Pool Update Result'
RabbitMq-->>Pool: Commit

RabbitMq->>Gate: Send update result B
Gate-->>RabbitMq: Commit
Gate-->>Gate: Resolve cleaning request B
Gate-->>Gate: Persist business partner A output

Advantages:

No additional BPDM service needed which reduces implementation overhead
Real-time pushing of results leads to faster information propagation

Disadvantages:

Services in the chain need to 'play ball', they need to integrate into each other very well so well-defined payloads is important (Event Queue will just take any payload at first naturally)
Finding errors and their reasons is much more complicated in an event-based setup
Problematic: Cleaning requests in the queue are visible to every Gate. Even if business partners are anonymous  in principle this could be a security issue. Separate queues can also be problematic as it makes visible in RabbitMq which Gate shared what business partner

Variant 3: Event Queue with Lightweight RequestProvider
Just like Variant 2 heart of the orchestration is RabbitMq but in order to solve security issues, resulting from visible business partner data, we introduce request provider service. This request provider service is responsible for generating request IDs and caching the results. This service provides Gates with a way to receive cleaning results via a private request ID that is only known to the Gate which originally created the cleaning request. In this scenario the CleaningRequestProvider is responsible for creating a private and public request ID that it shares with the requesting Gate. The gate then published a cleaning request similar to Variant 2. At the end of the process though only the CleaningRequestProvider receives the actual cleaning result and broadcasts the notification of a finished cleaning request to the Gates. Only the Gate with the private request ID is then allowed to access the actual request result.
sequenceDiagram
SharingMember->>Gate: Upsert Business partner A
Gate-->>Gate: Persist business partner input A
Gate-->>SharingMember: Created business partner A input
Gate->>CleaningRequestProvider: Request new cleaning request ID
CleaningRequestProvider->>CleaningRequestProvider: Create timestamped Cleaning Request with public ID B and private ID C
CleaningRequestProvider-->>Gate: Cleaning request public ID B and private ID C
Gate-->>Gate: Associate business partner input A with cleaning request IDs
Gate->>RabbitMq: Send cleaning request B with input A to topic 'New Cleaning Request'
RabbitMq-->>Gate: Commit
RabbitMq->>DummyBridge: Send cleaning request B
DummyBridge-->>RabbitMq: Commit
opt Input has no BPNs
    DummyBridge-->>DummyBridge: Create unique BPN request IDs
end
DummyBridge-->>DummyBridge: Create Cleaning Result for cleaning request B
DummyBridge->>RabbitMq: Send cleaning result for B to topic 'New Process Step Generic Cleaning Result'
RabbitMq-->>DummyBridge: Commit

RabbitMq->>Pool: Send cleaning result B
Pool-->>RabbitMq: Commit
Pool-->>Pool: Upsert LSA business partner based on BPNs/BPN-Request-IDs
Pool->>RabbitMq: Send  update result for B to topic 'New Pool Update Result'
Pool->>RabbitMq: Send updated BPNs to topic 'New BPN Update'
RabbitMq-->>Pool: Commit

RabbitMq->>CleaningRequestProvider: Send update result B
CleaningRequestProvider-->>RabbitMq: Commit
CleaningRequestProvider-->>CleaningRequestProvider: Persist update result
CleaningRequestProvider->>RabbitMq: Send notification for cleaning request B 'Cleaning Request FINISHED'
RabbitMq-->>CleaningRequestProvider: Commit

RabbitMq->>Gate: Send notification cleaning request B finished
Gate-->>RabbitMq: Commit
Gate-->>CleaningRequestProvider: Request result for B via private key C
CleaningRequestProvider->>Gate: Result for cleaning request B
Gate-->>Gate: Resolve cleaning request B
Gate-->>Gate: Persist business partner A output

RabbitMq->>Gate: Send updated BPNs
Gate-->>RabbitMq: Commit
opt Business Partner in Gate associated with BPN and newer timestamp
     Gate->>Gate: Start new cleaning process (see above)
end

Advantages:

Solves security considerations of Variant 2
Other advantages of variant 2

Disadvantages:

More overhead due to the need of an additional service
More complexity due to the Gates having to integate to an event queue as well as an additonal service
Event-based disadvantages raised in Variant 2

In my opinion, as I can see it from the description, Variant 3 combines the disadvantages of the other two variants. Because also here you would need again a polling mechanism for some parts. Or am I wrong?
In my first impression, I would recommend variant 1. Here you have the greatest security and sovereignty over the data and data exchange formats. You also have a good abstraction for the different cleaning states and can easily add new ones by just adding a new API Endpoint in the orchestrator service.
If polling every minute is not enough, you can think about adapting the APIs and the control flow from pull to push. But as far as I know, at the moment, there are no concrete requirements in that regard.

In V2, does it need both a CleaningServiceDummy and a DummyBridge, or is this redundant?
And in V3, is the DummyBridge the same as the CleaningServiceDummy?
One thing about the duplicate check in the cleaning service: If it finds out the BP is a duplicate and already in the Pool it could and should directly set its BPN, not just BPN-RequestID, right?
Thinking about V1, couldn't the Orchestrator work without persisting any data and just keep the queue data in RAM?
As the Gates should poll for the state of their cleaning requests they could detect if some BP goes missing and just add it again for cleaning.
What is the worst that could happen? Maybe if BPs from the FINISHED queue go missing, so they already got a BPN assigned and were added to the Pool.
But even then, after adding the BP again the CleaningService should detect it as a duplicate and assign the same BPN / BPN-RequestID and Gate should eventually get the updated BP.
If not persisting anything in the Orchestrator is indeed feasible, this would mitigate security issues and also simplify implementation.

For V1 the mentioned disadvantages regarding polling don't seem critical.
My main issue for me: Is there a legal problem to have this central Orchestrator component that stores (temporarily) all the different Gates' data?
About V3: On first sight this looks quite convoluted
As Tim mentioned, doesn't the Gate need to poll the CleaningRequestProvider to get the cleaning result? So there is no real-time pushing of results.
And if the CRP needs to persist the cleaning results this component gets quite complicated instead of the made-redundant Orchestrator.
Maybe I'm overlooking something but I don't see any advantage of V3 vs V1.
About the event-based approaches (V2 and V3):
I think this is not optimal to standardize as it's a different and additional protocol needed and there is no formal way that I know of to standardize what topics are allowed and which content schema is allowed for each topic.
General idea about persistence:
If we really have to persist anything (see my previous comment) in some kind of Orchestrator component, maybe it makes sense to not use a relational DB, as this component doesn't care for the content but just acts as a generic state machine. Maybe just use a file or document-style approach.

We can create multiple queue from rabbitmq for each Gate component. Now Gate will subscribe the queue.
Cleaning_Service will handle all messages and it contains the BPN with the queue identity so when we need to publish the message to sharing member's then we will publish it onto the rabbitmq.
@startuml
participant Sharing_Member as Sharing_Member
control GATE as GATE
database Gate_Database as Gate_Database
queue RabbitMQ as RabbitMQ
control Cleaning_Service as Cleaning_Service
database Cleaning_Database as Cleaning_Database
control Service_Provider as Service_Provider
control Pool as Pool
database Pool_Database as Pool_Database

RabbitMQ <- Cleaning_Service: Subscribe to the topic
RabbitMQ --> Cleaning_Service: Subscription has been done (Now when the message will publish on the topic \nIt will consume by the Cleaning_Service)

note over Sharing_Member, Pool_Database
Create a topic for each Gate component.
end note

GATE -> RabbitMQ: Subscribe to the topic that was created for Gate (Gate topic)
GATE <-- RabbitMQ: Subscription has been done (Now when the message will publish on the topic \nIt will consume by the Gate component)

Sharing_Member -> GATE:  Share Partners data

GATE -> GATE: Valid the requested data
GATE --> Sharing_Member: Return with a validation error message
GATE -> Gate_Database: save the partner's details with the input stage
GATE -> RabbitMQ: Publish data that need to clean on the topic (GR topic)

RabbitMQ -> Cleaning_Service: Push the data to the GR service
Cleaning_Service -> Cleaning_Service: Validation of the received data from the service bus
Cleaning_Service --> RabbitMQ: Publish the request with the validation failed
Cleaning_Service-> Cleaning_Database: If data is valid then save it in the database
Cleaning_Service -> Cleaning_Service: Prepare the request for Service_Provider
Cleaning_Service -> Cleaning_Database: Save the data to the table that was listened to by Service_Provider

note over Cleaning_Database
Need to create a User who has access to Service_Provider tables.
Share the user details with Service_Provider.
end note

Cleaning_Database <-- Service_Provider: Corn will be scheduled from Service_Provider for lookup of the input table
Service_Provider -> Service_Provider: Process the validation
Service_Provider -> Service_Provider: The cleanup process has been initiated
Service_Provider -> Cleaning_Database: Save the data in the output table

Cleaning_Service-> Cleaning_Database: Corn will be scheduled to lookup the output table \nwhere the cleaned data stored
Cleaning_Service -> Pool: Initiate the BPN generation process
Pool-> Pool: Validate the BPN issuing request
Pool --> Cleaning_Service: Return with the error if validation will be failed
Cleaning_Service --> RabbitMQ: Return the requested data with the validation failed message
Pool -> Pool: Issue the BPN for partner
Pool -> Pool_Database: Save the partner's details to the database
Pool -> Cleaning_Service: Return with the data which contains the BPNL/S/A
Cleaning_Service -> Cleaning_Database: Save the data with BPNL/S/A

Cleaning_Service -> Cleaning_Database: Fetch the topic from the database used for publishing \nThe topic that will indicate the Gate component
Cleaning_Service -> RabbitMQ: Publish the cleaned data on the multiple topics \nMultiple topic comes when multiple Gate has the same BP
GATE <- RabbitMQ: Push the data to the relevant gate
GATE -> Gate_Database: Save the cleaned data on the output stage
GATE <- Sharing_Member: Now sharing member can access the cleaned data with BPNL/S/A
@enduml

@martinfkaeser No, it's no knock-out criteria. Also with Variant 1 you can "abstract" the storing mechanism via interfaces. So basically, you can use in-memory, Redis Cache or PostgreSQL in a concrete implementation. And after processing the records, in all solutions you would be able to delete the records if this is a requirement. Also with message queueing you would have the issue that records get stored in the queue for some time.

@pd0856 can you explain please in a little bit more detail what the advantages of you solution vs. the outlined Variant 1 would be?
Also this seems for me a little bit unclear:

Now Gate will subscribe the queue. Cleaning_Service will handle all messages and it contains the BPN with the queue identity so when we need to publish the message to sharing member's then we will publish it onto the rabbitmq.

To publish messages back to a Gate you would need an extra topic which the Gate subscribes on. So basically you would need two topics per Gate, one for publishing, one for subscribing. And in the cleaningService you have to handle/store for each record where you have to publish the cleaned results? And how do you distribute the resulting updates to other Gates while preventing that they can see the whole uploaded data from the "Uploader-Gate"?
According to you PlantUML, you also still have polling inside the End-To-End Flow? So still no "real-time" communication?
And can you please also explain the goal/purpose and functions of the CleaningService in more detail?

Variant 1: Why does Orchestrator ist polled by the CleaningserviceDummy?
In Variant 2 the RabbitMQ sends to and polls from the CleaningserviceDummy.
I would prefer this for Variant 1.
             How long is the cleaning process expected to last (Seconds, Minutes, Hours)?
             Are asynchronous REST services  an option instead of polling?

@pd0856 can you explain please in a little bit more detail what the advantages of you solution vs. the outlined Variant 1 would be?
Also this seems for me a little bit unclear:

Now Gate will subscribe the queue. Cleaning_Service will handle all messages and it contains the BPN with the queue identity so when we need to publish the message to sharing member's then we will publish it onto the rabbitmq.

To publish messages back to a Gate you would need an extra topic which the Gate subscribes on. So basically you would need two topics per Gate, one for publishing, one for subscribing. And in the cleaningService you have to handle/store for each record where you have to publish the cleaned results? And how do you distribute the resulting updates to other Gates while preventing that they can see the whole uploaded data from the "Uploader-Gate"?
According to you PlantUML, you also still have polling inside the End-To-End Flow? So still no "real-time" communication?
And can you please also explain the goal/purpose and functions of the CleaningService in more detail?

Yeah sure, Let me discuss the overall the sequence diagram. And also regarding the Cleaning_Service, this service will have the central database where we will manage the Gate with it's relevant topic. Based on the request that was received by rabbitmq Cleaning_Service will identify where to publish the message.
Here also i assume that the service provider will read the requests from the database. But that we can change. Or also i assume that service_provider is not support the webhook mechanism so we need to lookup the database which contains the processed data.
Here also if BPN will be relevant to multiple Gate then Cleaning_service will identify that stuff as well and publish the same business partner on the multiple queues.
Additionally, No other Gate will see if the partner is not a part of there component.

Variant 1:
Why does Orchestrator ist polled by the CleaningserviceDummy?
In Variant 2 the RabbitMQ sends to and polls from the CleaningserviceDummy.
I would prefer this for Variant 1.

The CleaningServiceDummy is an exchangable service which in production can be replaced by a more elaborate cleaning service. Keeping the orchestrator passive allows us to keep the orchestrator the same while exchanging the cleaning service and letting it integrate into the orchestrator. The other way around is possible as well of course but would require a different strategy: The orchestrator would need exchangable components that integrate against the respective cleaning service. In my opinion the first option is simpler and allows exchanging the cleaning service by configuration of deployed services.

How long is the cleaning process expected to last (Seconds, Minutes, Hours)? Are asynchronous REST services an option instead of polling?

I think nobody can answer that at the moment as this is dependent on the actual cleaning service provider who performs the cleaning. In what way would asynchronous REST be different from polling? Polling is part of asynchronous calls, except if you want to use callbacks.

When I review all 3 variants I'm missing the following scenarios:

At the beginning all incoming upserted data records from each CX Member are screened by NPS to secure no
NPS record is getting passed into the cleaning service
We have to enrich the process flow beyond the initial upload process. More important is the running mode
where the external service provider pushes updates in a volume of thousands a day. The Orchestrator has to know
for which CX Member persistence those updates are relevant and has to push those updates into the appropriate
outbound persistence of the CX member. The CX Member system polls via the Gate API the logfile to identify
changes.
The CX outbound persistence logfile has to contain all changes by attribute the Gate API provides to enable
the CX Member system to identify each change.
The Orchesteration layer has to contain a memory function to store historical data matched to the architecture
of the service provider. Thosedata are also important for the DQ Dashboard solution
The Orchastration layer has to be sensitive on start and end date content by defined attributes and support the
predecessor and successor function.

Thanks Nico for documenting the alternatives! Here are a few thoughts / comments on these:

While v2 attracts with its simplicity there are potential security concerns listed. What would be attack vectors here and would a similar attack in the other architecture proposals need higher access levels to succeed?
Does the current architecture already cover the concept of "Hardening Grade" for records? This requires a backend service provider to know about record ownership, which partially breaks through the requirement for complete anonymity of records.
Sharing of a changed address requires projections of Golden Records to customer-specific versions, since customers should only receive the fields of an address that they are actually also inserting into the process. How can we cover this in the current designs?
How do we want to process records returned from a provider? Does the bridge need to iterate through all gate instances to check to which gate to send a message to via the bus? Or do we put changed records more generically as a broadcast onto the bus and each gate instance decides for itself what to do with it, i.e. filtering if BPN is not relevant for the gate instance, projection to relevant fields in the gate etc. The latter seems to be much simpler and provides better decoupling, but it brings us back to the security concern from above?!
To simplify debugging/reporting a logging mechanism would be helpful. Is this already part of the current designs? If we would extensively use a bus, this infrastructure could also be used as a basis for logging with a logger component simply listening on the bus?

While v2 attracts with its simplicity there are potential security concerns listed. What would be attack vectors here and would a similar attack in the other architecture proposals need higher access levels to succeed?
v2 seems just from the first view "simple". When it comes to implementation you are introducing a completely new system (message bus) and need new expertise in terminology, operating and managing, which takes a lot of time. v1 instead would build up on known technology components by "just" implementing a new Service Component. In addition, in v1 the shared business partners stay anonym via a "Ticket-ID" so no one except the Sharing Member itself knows about his business partner. In a message bus approach, you would have at least a 1:1 Relation between sharing member and topic where sharing member specific business partner could be tracked.

Does the current architecture already cover the concept of "Hardening Grade" for records? This requires a backend service provider to know about record ownership, which partially breaks through the requirement for complete anonymity of records.
Having an Identifier for Ownership in the data model is planned for the next release. Also for this, with v1 you can decide via your API Endpoints in the Orchestrator, who should have access to this field and how not. Via Topic approach, you would just push the complete business partner to the topic and a cleaning provider or someone else would see the complete message.

Sharing of a changed address requires projections of Golden Records to customer-specific versions, since customers should only receive the fields of an address that they are actually also inserting into the process. How can we cover this in the current designs?
All Gates periodically pull the Pool Changelog for changed BPNs. If there are changed BPNs that match with BPNs that are stored within the Gate, the Gate triggers a new cleaning request which starts the golden record process for this record. In the end it can fetch the result via the generated Ticket-ID.

How do we want to process records returned from a provider? Does the bridge need to iterate through all gate instances to check to which gate to send a message to via the bus? Or do we put changed records more generically as a broadcast onto the bus and each gate instance decides for itself what to do with it, i.e. filtering if BPN is not relevant for the gate instance, projection to relevant fields in the gate etc. The latter seems to be much simpler and provides better decoupling, but it brings us back to the security concern from above?!
See my previous comment. If we would do it like this via "broadcast", all other gates would also receive sharing member specific data which might cause a security issue. For example, all members might use the nameParts Field for different internal purposes and no other member should be able to recognize this.

To simplify debugging/reporting a logging mechanism would be helpful. Is this already part of the current designs? If we would extensively use a bus, this infrastructure could also be used as a basis for logging with a logger component simply listening on the bus?
For sure logging is an important topic. But logging can be realized with and without a message bus. So, logging and messaging are separate topics for me.

I have created a pull request proposing an ADR on how to implement the orchestrator component. It should focus on KISS principle and simplicity.
https://github.com/eclipse-tractusx/bpdm/pull/418

When it comes to alternative one, i.e. no message bus, how do we share a changed record? I understand the gates are polling for updates in this approach and I assume that maybe a timestamp could be a parameter here, i.e. "give me back updates since timestamp". But since it was argued that only relevant records should be passed to a gate instance, how do we know in the orchestrator which updated records to return to a polling gate?

Isn't it true that we have to differentiate between different change situations:

GR engine fetches a record out of a CX Member Inbound Persistence and the GR process failed.
CX Member gate polls and identifies this record which failed to receive a BPNID. Meaningful is the time stamp
when the record was fetched and time stamp when the GR engine updates the logfile of the CX Member
Outbound Persistence. This happens only to the Outbound Persistence of the owner of this record.
A change occurs to a BP data record in the pool which already contains a valid BPNID. A significant change
occurs to that the record will loose and Golden Record status at the date of this change - means the data record can only
keep his Golden Record status and BPNID until the date the change occured. Also in case we can a time stamp to
document this data, but we need a GR trigger field which has to set to false at that date.
Furthermore the GR process has to rerun to generate a new golden record with a new BPNID
I estimated that in a full swing model around 7.000 changes will happen daily of which a fair amount of data records
will loose their golden record status.

When it comes to alternative one, i.e. no message bus, how do we share a changed record? I understand the gates are polling for updates in this approach and I assume that maybe a timestamp could be a parameter here, i.e. "give me back updates since timestamp". But since it was argued that only relevant records should be passed to a gate instance, how do we know in the orchestrator which updated records to return to a polling gate?

@cofinity-x-stephan based on the changelog in the pool on which the gates are polling, they check if they have an BPN which was updated in the pool. If so, the according gate will trigger a new cleaning request for his business partner with that BPN and the cleaning process starts for the gate. So not the whole Business Parter Object gets passed through all gates.

thanks tim! but this means that for an address that is used in 20 gates we also have 20 runs of the cleaning process even though only a single run actually is needed? or would the first run in your concept be different from the consecutive runs? we need to be cautious here since the backend service may also include human data curators that have to be paid, thus we need to minimize calls to that service to provide a cost effective solution.

Since we do not know about the concrete cleaning service (provider) we are open for discussions and how you would propose the end-to-end flow. Generally, speaking we have to take a decision for overall architecture and target goal which in my opinion should be the API based service orchestrator approach because of its simplicity and flexibility. If we come up at some point that a messaging system is really mandatory, of course we can introduce one. But for now, focusing on what is really needed and how we can deliver fast, the API based approach matches best!
In regard to your concrete question. Yes, it was intended like you describe, because it would be a solution that works out of the box for a first minimal viable product. For sure we can also think about introducing an additional state in the orchestrator for business partner records that should not be "cleaned", because they already are, but should just be updated with latest changes from Golden Record Pool. The Problem here comes with the details. Because Golden Record Pool data model differentiates from Gate Model and somehow a mapping between both have to be done when we say that the business partner record should not again run through the whole cleaning process. Especially thinking about that every sharing member has different semantics of gate specific data model fields, i.e. the name parts field is used differently through all sharing members.

I have prepared a cleaned and slimmed down variant 1 overview which is intended to act as a base for more issues detailing work packages needed to achieve what we need. This diagram still expects that each Gate will start its own cleaning process after it identifies a BPN update in the Pool. But optimizations here are easy to make in this regard. With a few tweaks the Gates needing only an update could trigger a slimmed down version of the cleaning process for example.
sequenceDiagram
    SharingMember->>Gate: (1) Upsert business partner A input
    Gate-->>Gate: (2) Persist business partner A input
    Gate->>Orchestrator: (3) Request cleaning of business partner A input
    Orchestrator-->>Orchestrator: (4) Create timestamped Cleaning Request B for business partner A input
    Orchestrator-->>Orchestrator: (4) Update Cleaning Request B State: QUEUED_GENERIC_CLEANING
    Orchestrator-->>Gate: Created cleaning request B
    Gate-->>Gate: (5) Associate cleaning request ID with business partner A
    Gate-->>Gate: (5) Upsert sharing state of business partner A
    Gate-->>SharingMember: Upserted business partner A input
    loop Poll Generic Cleaning
        CleaningServiceDummy->>Orchestrator: (6) Fetch and reserve next cleaning requests in 'Generic Cleaning' queue
        Orchestrator-->>Orchestrator: (4) Update state of cleaning request B: RESERVED_GENERIC_CLEANING
        Orchestrator-->>CleaningServiceDummy: Cleaning request B with business partner input
            CleaningServiceDummy->>CleaningServiceDummy: (7) Create dummy cleaning results
        opt BPN does not exist
            CleaningServiceDummy->>CleaningServiceDummy: (7) Add BPN request IDs into results
        end
        CleaningServiceDummy->>Orchestrator: (8) Send cleaning result for request B
        Orchestrator-->>Orchestrator: (4) Update Cleaning Result of request B
        Orchestrator-->>Orchestrator: (4) Update state of cleaning request B: QUEUED_BPN_PROCESSING
        Orchestrator-->>CleaningServiceDummy: Accept
    end
    loop Poll BPN Processing
        Pool->>Orchestrator: (9) Fetch and reserve next cleaning requests in 'BPN Processing' queue
        Orchestrator-->>Orchestrator: (4) Update state of cleaning request B: RESERVED_BPN_PROCESSING
        Orchestrator-->>Pool: Cleaning request B cleaning result
        opt L/S/A result has changed
            Pool-->>Pool: (10) Upsert L/S/A business partners with result
        end
        Pool-->>Pool: (11) Add BPNs from BPN request IDs to cleaning result
        Pool->>Orchestrator: (12) Send BPN cleaning result for request B
        Orchestrator-->>Pool: Accept
        Orchestrator-->>Orchestrator: Update state of cleaning request B: FINISHED
    end
    loop Poll for finished Cleaning Requests
        Gate->>Orchestrator: (13) Fetch cleaning request cleaning result by request ID
        Orchestrator-->Gate: Cleaning request B cleaning result
        Gate-->>Gate: (5) Resolve cleaning request B
        Gate-->>Gate: (5) Update sharing state of business partner A
    end
    loop Poll Pool BPN Changes
        Gate->>Pool: Fetch BPN changelog entries from last timestamp
        Pool-->>Gate: Entries from given timestamp
        opt Business Partner in Gate associated with BPN and newer timestamp (14)
            Gate->>Gate: Start new cleaning process (see above)
        end
    end

I also created a simple dependency chart showing which work packages are dependent on each other in what way:
flowchart TD
    model[Generic Business Partner DTO Interface]

    subgraph GateLSA[Gate LSA]
        lsaMapLegalEntity[Logic: Map legal Entities to generic model]
        lsaConnectLegalEntity[Logic: Connect legal entity endpoints to generic service logic]
        lsaMigrateLegalEntity[Db-Script: Migrate existing legal entities to generic business partners]
        lsaMapSite[Logic: Map sites to generic model]
        lsaConnectSite[Logic: Connect site endpoints to generic service logic]
        lsaMigrateSite[Db-Script: Migrate existing sites to generic business partners]
        lsaMapAddress[Logic: Map addresses to generic model]
        lsaConnectAddress[Logic: Connect address endpoints to generic service logic]
        lsaMigrateAddress[Db-Script: Migrate existing addresses to generic business partners]
    end

    lsaMapLegalEntity-->lsaConnectLegalEntity
    lsaConnectLegalEntity-->lsaMigrateLegalEntity
    lsaMapSite-->lsaConnectSite
    lsaConnectSite-->lsaMigrateSite
    lsaMapAddress-->lsaConnectAddress
    lsaConnectAddress-->lsaMigrateAddress

    subgraph GateOrch[Gate Orchestration]
         gateGenericInputEndpoints["(1) Input Endpoints for Generic Business Partner"]
         gateGenericPersistence["(2) Persist Generic Business Partners"]
         gateRequestCleaning["(5) Request Cleaning for Own Update"]
         gateResolveRequest["(13) Resolve finished Cleaning Results"]
         gateCheckBpnUpdate["(14) Request Cleaning for BPN Update"]
    end

    gateGenericInputEndpoints-->gateGenericPersistence
    gateGenericPersistence-->gateRequestCleaning
    gateRequestCleaning-->gateResolveRequest
    gateRequestCleaning-->gateCheckBpnUpdate

    subgraph Orchestrator[Orchestrator]
        orchBase[Setup module]
        orchRequestCleaningEndpoint["(3) Request Business Partner Cleaning Endpoint"]
        orchCleaningQueueEndpoint["(6) Reserve Cleaning Requests for Cleaning Endpoint"]
        orchCleaningResolveEndpoint["(8) Resolve Cleaning Request for Cleaning Endpoint"]
        orchBpnQueueEndpoints["(9) Reserve Cleaning Requests for BPN Processing Endpoint"]
        orchBpnResolveEndpoint["(12)  Resolve Cleaning Request for BPN Processing Endpoint"]
        orchFinishedEndpoint["(13) Fetch Finished Cleaning Request Endpoint"]
        orchStateMachine["(4) Manage Cleaning Request State and Data"]
    end

    orchBase-->orchRequestCleaningEndpoint
    orchBase-->orchCleaningQueueEndpoint
    orchBase-->orchCleaningResolveEndpoint
    orchBase-->orchBpnQueueEndpoints
    orchBase-->orchBpnResolveEndpoint
    orchBase-->orchFinishedEndpoint
    orchRequestCleaningEndpoint-->orchStateMachine
    orchCleaningQueueEndpoint-->orchStateMachine
    orchCleaningResolveEndpoint-->orchStateMachine
    orchBpnQueueEndpoints-->orchStateMachine
    orchFinishedEndpoint-->orchStateMachine

    subgraph CleaningDummy[Cleaning Service Dummy]
        dummyBase[Setup module]
        dummyServiceLogic["(7) Create Dummy Cleaning Result"]
    end

    dummyBase-->dummyServiceLogic

    subgraph Pool[Pool]
        poolUpsert["(10) Upsert Business Partners from Cleaning Result"]
        poolResult["(11) Send BPN Processing Result"]
    end

    poolUpsert-->poolResult

    orchRequestCleaningEndpoint-->gateRequestCleaning
    orchCleaningQueueEndpoint-->dummyServiceLogic
    orchBpnQueueEndpoints-->Pool
    orchBpnResolveEndpoint-->Pool
    orchFinishedEndpoint-->gateResolveRequest

    gateGenericPersistence-->lsaConnectLegalEntity
    gateGenericPersistence-->lsaConnectSite
    gateGenericPersistence-->lsaConnectAddress

    model-->GateLSA
    model-->GateOrch
    model-->Orchestrator
    model-->CleaningDummy
    model-->Pool

I created also issues for work packages which give more details:
(1) #384
(2) #428
(3) #425 #429
(4) #427
(5) #429
(6) #388
(7) #433
(8) #392
(9) #434
(10, 11) #432
(12) #435
(13) #426 #430
(14) #431
Please not that the linked issues need refinment still and will undergo change in the next days.

This is a squence diagram of the orchestration flow depicting the use case of updating a Gate business partner entry based on a BPN update from the pool. In this flow the Gate business partner does not have any new information and just needs the newest state from the Pool incorporated. The service provider is responsible for incorporating the Pool state to the Gate business partner. No extensive cleaning is needed in this case.
sequenceDiagram
    loop Poll Pool BPN Changes
        Gate->>Pool: Fetch BPN changelog entries from last timestamp
        Pool-->>Gate: Entries from given timestamp
        opt Business Partner in Gate associated with BPN C and newer timestamp (14)
            Gate->>Orchestrator: Request update for business partner A from Pool
            Orchestrator-->>Orchestrator: Create timestamped Cleaning Request D for business partner A input
            Orchestrator-->>Orchestrator: Update Cleaning Request B State: QUEUED_UPDATE_LSA
            Orchestrator-->>Gate: Created cleaning request D
            Gate-->>Gate: Associate cleaning request D with business partner A
            Gate-->>Gate: Upsert sharing state of business partner A
        end
    end
    loop Pool L/S/A Update queue
        CleaningServiceDummy->>Orchestrator: Fetch and reserve next cleaning requests in 'Update L/S/A' queue
        Orchestrator-->>Orchestrator: Update state of cleaning request B: RESERVED_UPDATE_LSA
        Orchestrator-->>CleaningServiceDummy: Cleaning request D with business partner input A
        CleaningServiceDummy->>Pool: Request L/S/A business partner with BPN C
        Pool-->>Pool: Fetch L/S/A business partner C form database
        Pool-->>CleaningServiceDummy: Return L/S/A business partner C
        CleaningServiceDummy-->>CleaningServiceDummy: Create dummy update result
        CleaningServiceDummy->>Orchestrator: Send update result for request D
        Orchestrator-->>Orchestrator: Update Update Result of request D
        Orchestrator-->>Orchestrator: Update state of cleaning request D: FINISHED
        Orchestrator-->>CleaningServiceDummy: Accept
    end
    loop Poll for finished Cleaning Requests
        Gate->>Orchestrator: (13) Fetch cleaning request cleaning result by request ID
        Orchestrator-->Gate: Cleaning request D cleaning result
        Gate-->>Gate: (5) Resolve cleaning request D
        Gate-->>Gate: (5) Update sharing state of business partner A
    end

Since the concept of the orchestration layer has evolved alot in our discussion I have opened a new issue #455 in which the newest state of our concept is condensed. This helps us keep the concept concise and readable. Please post further comments on this to the new issue #455.

authors: Phil91 Rossisep cofinity-x-stephan gerlitmcoding martinfkaeser nicoprow pd0856 rainer-exxcellent

repo: eclipse-tractusx/bpdm


text:
W3C Caused by: com.apicatalog.jsonld.JsonLdError: Unexpected response code [429]

My name is Tuan, I am getting the following error:
Caused by: com.apicatalog.jsonld.JsonLdError: Unexpected response code [429].
JsonLdError[code=There was a problem encountered loading a remote context [code=LOADING_REMOTE_CONTEXT_FAILED]., message=There was a problem encountered loading a remote context [code=LOADING_REMOTE_CONTEXT_FAILED].]

According to my understanding of this error, I seem to have sent many requests to two urls:
"https://www.w3.org/2018/credentials/v1",
"https://catenax-ng.github.io/product-core-schemas/SummaryVC.json",

Can I ask if the w3.org organization seems to have limited my request?
Can you help me solve this problem? I really need help to complete my work
Sincerely thank

Hi Tuan,
please in the future supply more information about the MIW version you're trying to use.
I'm assuming you're on something < 0.2.0. There was an issue with rate-limiting which is now solved. Please use the current 0.2.0 or later.

authors: borisrizov-zf tuanphampmt


text:
App Marketplace - Update TechnicalUserProfile by allowing "none"

Summary
The current implementation of the app release process expects at least 1 technical user role selected for connecting the app(s) to central applications.
Till now a technical user was mandatorily needed - BUT changed recently and we need to allow with 24.05. that an user can release an app even without technical user connection.

Details
Adjust the release process of apps.
The generic workflow stays as-is; but following changes are needed:

[ ] in addition to the shown/displayed technical user permission; a field called "none" should get added (similar like we have it for privacy information)

de: "Nicht notwendig"
en: "none"

[ ] add following description to the "none" permission

de: "für die Integration ist kein technischer User notwendig"
en: "no technical user needed to run the app/service"

[ ] if the user selects "none" any other selected permissions get unselected
[ ] if the user has "none" selected and clicks on top on a checkmark of a role, the "none" option gets unselected
[ ] business logic adjustment:

if the GET endpoint /api/apps/AppReleaseProcess/{appId}/technical-user-profiles was empty ([]) and the user selects "none" - do not trigger and PUT /api/apps/appreleaseprocess/{appid}/technical-user-profiles endpoint when the user clicks on "save" or "save & proceed"
if the GET endpoint /api/apps/AppReleaseProcess/{appId}/technical-user-profiles was not empty and the user selects now "none" - trigger the PUT /api/apps/appreleaseprocess/{appid}/technical-user-profiles endpoint when the user clicks on "save" or "save & proceed". In this case following body should get send to the backend

[
  {
    "technicalUserProfileId": "e28e7e6c-8bcf-4056-b044-8355f7b6b17f",
    "userRoleIds": []
  }
]

NOTE - currently the backend expects the technicalUserProfile since it is mandatory - this is getting updated with ticket https://github.com/eclipse-tractusx/portal-backend/issues/497 & https://github.com/eclipse-tractusx/portal-backend/issues/499

NOTE: this is a breaking change and should only get merged when the backend change is available as well.
Please highlight this in the PR and link the FE and Backend PRs

repo: eclipse-tractusx/portal-frontend


text:
Definition of Goal-Oriented Skills

Benefit Hypothesis
Having skill templates for common dataspace situations will help the use case and application developers to implement their business logic
Skills typically enwork data (assets) from the preconditions to the final output.
Assets need to be restricted/found on the meta-level in the reverse direction (from the goal type to the precondition type)
Having a template which can bring both directions together in a ping-pong style will allow very flexible data processing patterns.

Beneficiaries
any use case developer
any application developer

Acceptance Criteria
Necessary extensions to the CX-0084 and CX-0067 standards related to the description of input and output types of assets
Skill Template
Example Skill realising ping-pong behaviour
Invocation of remote skills
Aggregating individual results
Calling the Catalogue Graph as an Asset

Estimated Story Points:
16 SP

please also integrate the acceptance criteria of https://github.com/eclipse-tractusx/sig-release/discussions/203#discussioncomment-7324180

Story "Extend Catena-X Standards with SHACL-Descriptions of Graph Assets"
Repository https://github.com/eclipse-tractusx/eclipse-tractusx.github.io
Statement
As a Catena-X stakeholder, I want to have a standard for describing graph assets which allows me to describe the actual shape/content of a graph in a fashion that could be used to drive a meta-computation.
Acceptance Criteria

standard CX-0084 extended
optionally CX-0067 extended
descriptions of examples in both the standard and the kit

Estimation
5 SP

Story "Build a Blueprint and an Example Set of Assets and a Skill Demonstrating Goal-Oriented Computations"
Repository https://github.com/eclipse-tractusx/eclipse-tractusx.github.io
Statement
As a Graph or Skill Asset developer/deployer, I want to have a blueprint and realistic example which shows me how to encode a goal-oriented skill such that I can adopt this for my own use case.
Acceptance Criteria

blueprint with SPARQL snippets
set of asset descriptions
ready-made skill

Estimation
5 SP

Story "Document Some Important Features in the Agent Kit"
Repository https://github.com/eclipse-tractusx/eclipse-tractusx.github.io
Statement
As a KA developer, I want to have documented examples for important standard features, such as remote skill invocation, aggregation and dealing with the catalogue such that I can make use of them in my creation and invocation of skills.
Acceptance Criteria

Example for the invocation of remote skills in both distribution/run modes
Example for aggregating (in subselects)
Example for binding against the catalogue

Estimation
3 SP

Hi @drcgjung and @TomBuchertTSystems could we close this issue? seems that is already done?

authors: FaGru3n TomBuchertTSystems drcgjung

repo: eclipse-tractusx/sig-release


text:
[Model Update]: Address and Contact

Update Reason

Correct typos
improve descriptions

--> no new version needed. Descriptions do not semantically change
MS1 Criteria

[ ] The model that should be updated exists
[ ] The referenced use case exists
[ ] The potential updates are discussed with all stakeholders
[ ] A decision on versioning was made in consensus with the affected parties according to the guidelines.
[ ] The potential update will be either

[ ] backward compatible or
[ ] a (chain of) migration strategy(-ies) from all non-deprecated previous versions to the new model will be developed

[ ] A modeling steward has been assigned

@MasonTsystems

repo: eclipse-tractusx/sldt-semantic-models


text:
issues/#298 fix + test.

This PR addresses GitHub issue: #298
Briefly describe the changes proposed in this PR:

...
...
...

Make sure you've followed the Contributor Guidelines. In particular (please tick to indicate you've taken care of it):

[ ] RDF4J code formatting has been applied
[ ] tests are included
[ ] all tests succeed

Signed-off-by: Mark Hale mark.hale@physics.org

@pulquero I put up this PR to make sure - looks good to you?

yep

Ok thanks. Running hudson, if nothing breaks this can be merged.

authors: jeenbroekstra pulquero

license_source: bigquery


text:
Error Handling and Jupyter Notebooks

Problem is that currently xacc::error exit()'s the process, killing the jupyter kernel.

My thought here is that we could provide an is-py-api command line option, setup pyxacc.py to always initialize the framework with this option, and then in xacc::error() check if this is-py-api and throw a runtime_error if it is.

repo: eclipse/xacc


text:
Declarative Design of Layout/widgets

I have attempted a new way of declaring the widgets instead of appending to parent in conventional way and using .property for each. The below approach uses JSON to declaratively design Navigation, Page, and underneath widgets and clearly can show which is appended into where. This handles, events, animation, child components easily.
// Example:
var designer = new contentDesigner();

var content = {
     _self: tabris.ui.contentView,
     _children:[
       {
       _self: designer.create('NavigationView'},
      _children:[
      {
        _self: designer.create('Page'),
       title:'My Page'
      }]
    ]
};

designer.design(content);

// The declarative Designer module;
var contentDesigner = function(){
  return this;
}

contentDesigner.prototype.create = function(widget){

    let typeofParam = typeof widget;

    if (typeofParam == 'string')
    {
      	if (widget == 'NavView')
          {
            return new tabris.NavigationView({left:0, right:0, top:0, bottom:0});
          }
      	else if (widget == 'ExNavView')
          {
            return tabris.ui.contentView.find('NavigationView').first();
          }
      	else if (widget == 'Page')
          {
            return new tabris.Page();
          }
      	else if (widget === 'Composite')
          {
            return new tabris.Composite();
          }
      	else if (widget == 'TextView')
          {
            return new tabris.TextView();
          }
      	else if (widget == 'ImageView')
          {
            return new tabris.ImageView();
          }
      	else if (widget == 'ScrollView')
          {
            return new tabris.ScrollView();
          }
      	else if (widget == 'CollectionView')
          {
            return new tabris.CollectionView();
          }
      	else if (widget == 'Button')
          {
            return new tabris.Button();
          }
      	else if (widget == 'TextInput')
          {
            return new tabris.TextInput();
          }
      	else if (widget == 'Picker')
          {
            return new tabris.Picker();
          }
      	else if (widget == 'RadioButton')
          {
            return new tabris.RadioButton();
          }
      	else if (widget == 'WebView')
          {
            return new tabris.WebView();
          }
      	else if (widget == 'ActivityIndicator')
          {
            return new tabris.ActivityIndicator();
          }
      	else if (widget == 'Video')
          {
            return new tabris.Video();
          }
      	else
          return null;

    }
    else
      {
        return  widget;
      }
  }

contentDesigner.prototype.design = function(widgetTree){

  var excludeProperties = ['_self', '_children','_eventListeners', '_animate'];

  function designWidget(parentWidgetDefn){

    var parentCont = (parentWidgetDefn._self);
    var ifNotBottom = parentWidgetDefn._children != undefined;

    if (ifNotBottom){
      parentWidgetDefn._children.forEach(function(child){
        var cont = (child._self);
        var propKeys = Object.keys(child);
        propKeys.forEach(function(prop){
          if (excludeProperties.indexOf(prop) == -1)
          {
            cont[prop] = child[prop];
          }
          if (child._eventListeners != undefined){
            child._eventListeners.forEach(function(eachEvntReg){
              cont.on(eachEvntReg.type, eachEvntReg.listener);
            });
          }
          if (child._animate != undefined){
              cont.animate(child._animate[0], child._animate[1]);
          }
        });
        cont.appendTo(parentCont);
        designWidget(child);
      });
    };
  }

  designWidget(widgetTree);
}

module.exports = contentDesigner;

Yes, the nice thing about the Tabris API is that is gives you the flexibility to easily create your own renderer. Some thoughts on your approach:

The content object contains widget references, so it's not declarative. As an effect, you won't be able to serialize it to JSON. Why don't you replace the _self field with a type that contains the widget type and let the renderer create it?
The huge switch case (`if (widget == 'TextView') return new tabris.TextView()``etc.) could be replaced by something as simple as:

if (name in tabris) {
  return new tabris[name](properties);
}

authors: natanraj ralfstx

repo: eclipsesource/tabris-js


text:
图表会跟随滑动进行滑动

提问前应该做的事
请确保提问前做了以下事，将完成的项目的 [] 改为 [x]：

[X] 我已通读过 README
[X] 我已阅读过 FAQ

需提供的信息
将符合项的 [] 改为 [x]，并补充需要的信息：
简单描述问题：
把小程序的page设置成高度100%后，图表会跟随向下滑动进行滚动，去掉100%后就不会出现这种情况了，有没有什么方法可以禁止图表滚动的
预期效果：
？？？
（如有需要请提供预期的图）
实际效果：
？？？
（如有需要请提供截图）
复现环境：

[] 在微信开发工具中存在该问题
[X] 在真机上存在该问题

position 是什么？

在图表组件的外层包了一层容器，没有设置position

同问

加个 position:relative 试试？

请问楼主是否已经解决？？

authors: Ovilia chudongyang suyi0509 taotingting

repo: ecomfe/echarts-for-weixin


text:
如何清除ehcarts每次渲染图后的定时器？

提问前应该做的事
请确保提问前做了以下事，将完成的项目的 [] 改为 [x]：

[] 我已通读过 README
[] 我已阅读过 FAQ

需提供的信息
将符合项的 [] 改为 [x]，并补充需要的信息：
简单描述问题：
每次进入需要图表的页面都预留定时器事件，多次更换ehcarts数据会叠加定时器，导致安卓机多次切换图表页面时卡顿，如何在退出页面是清除echarts指定的定时器？
预期效果：
正常清除ehcarts自带的定时器。
（如有需要请提供预期的图）
实际效果：
（如有需要请提供截图）

复现环境：

[] 在微信开发工具中存在该问题
[] 在真机上存在该问题

退出页面的的时候能销毁图表么？chart.dispose()

退出页面的的时候能销毁图表么？chart.dispose()
可以销毁，但是echarts里面的setTimeout没有清除

@Ovilia dispose会消除echarts里自带的setTimeout吗？我dispose后还是存在未清除定时器

同样的问题

你这调用跟echart本身没有关系啊

你这调用跟echart本身没有关系啊

这个问题没人解决吗？

怎么清楚啊，我这现在也遇到这个问题，可能会导致其他页面卡顿

怎么清楚啊，我这现在也遇到这个问题，可能会导致其他页面卡顿

setTimeOut会返回一个对象，要取消可以用这个返回的对象取消

setTimeOut会返回一个对象，要取消可以用这个返回的对象取消


text:
echarts BUG 返回NaN

调用map时 给map设置series出现错误
1.当series为空时 不会有以上错误
2.当series设置为{name:'四川',value:100}  出现以上错误
3.引用的是echarts/2.1.10/echarts-all.js
出错源代码
var a = 1;
var legenddata;
pieChart = function (showData, showName, showId) {
var dataString;
if(a==1) {
legenddata = eval("({ " + showName[1] + " : false," + showName[2] + " : false," + showName[3] + " : false})");
}
$(showId).show();
$("#container").hide();
var myCharts = echarts.init(document.getElementById(showId));
var optionBin = {
tooltip : {
trigger: 'item',
formatter: '{b} {a}占全国百分比：{c}%'
},
legend: {
orient: 'vertical',
x:'left',
selected: legenddata,
data:[showName[0],showName[1],showName[2],showName[3]]
},
dataRange: {
min: 0,
max: 100,
x: 'left',
y: 'bottom',
text:['高','低'],           // 文本，默认为数值文本
calculable : true
},
toolbox: {
show: true,
orient : 'vertical',
x: 'right',
y: 'center',
feature : {
restore : {show: true},
saveAsImage : {show: true}
}
},
series : [
{
name: '中国',
type: 'map',
mapType: 'china',
roam: false,
itemStyle:{
normal:{label:{show:true}},
emphasis:{label:{show:true}}
},
data:showData[0]
},{
name: showName[1],
type: 'map',
mapType: 'china',
itemStyle:{
normal:{label:{show:true}},
emphasis:{label:{show:true}}
},
data:showData[1]
},{
name: showName[2],
type: 'map',
mapType: 'china',
itemStyle:{
normal:{label:{show:true}},
emphasis:{label:{show:true}}
},
data:showData[2]
},{
name: showName[3],
type: 'map',
mapType: 'china',
itemStyle:{
normal:{label:{show:true}},
emphasis:{label:{show:true}}
},
data:showData[3]
}
]
};
a++;
myCharts.setOption(optionBin);
// 动态添加默认不显示的数据
    myCharts.on(echarts.config.EVENT.LEGEND_SELECTED, function (param){
        var clickName = param.target;
        if (clickName == showName[0]) {
            legenddata = eval("({ "+showName[1]+" : false,"+showName[2]+" : false,"+showName[3]+" : false})");
            pieChart(showData,showName,showId);
        }
        if (clickName == showName[1]) {
            legenddata = eval("({ "+showName[0]+" : false,"+showName[2]+" : false,"+showName[3]+" : false})");
            pieChart(showData,showName,showId);
        }
        if (clickName == showName[2]) {
            legenddata = eval("({ "+showName[0]+" : false,"+showName[1]+" : false,"+showName[3]+" : false})");
            pieChart(showData,showName,showId);
        }
        if (clickName == showName[3]) {
            legenddata = eval("({ "+showName[0]+" : false,"+showName[1]+" : false,"+showName[2]+" : false})");
            pieChart(showData,showName,showId);
        }
    });
}

1月23？我们没发布过版本啊，你用源码自己构建的？
如果问题还存在，请给一个可以在example里指教执行的option，谢谢

我使用的是echarts-2.1.10版本出现的这个问题

我已经用别的方式绕过了这个问题实现了功能， 还是感谢了

authors: dolphineor kener

repo: ecomfe/echarts


text:
是否有微信小程序版本？

如题，如果没有，有计划开发吗？

没有，没计划


text:
关于dataRange.splitList读写问题。

如：dataRange: {
x: 'left',
y: 'bottom',
splitList: [
{start: 1500},
{start: 900, end: 1500},
{start: 310, end: 1000},
{start: 200, end: 300},
{start: 10, end: 200, label: '10 到 200（自定义label）',color: 'blue'},
{start: 5, end: 5, label: '5（自定义特殊颜色）', color: 'black'},
{end: 10}
],
color: ['#E0022B', '#E09107', '#A3E00B']
},
当执行自定义赋值dataRange.splitList[0].start = 5000;没响应。怎么读写splitList数组内容？谢谢。

setOption 更新

你好，读dataRange.splitList[0].start值也不行，但读dataRange.color[0]正常。??  谢谢。

读写dataRange的splitList的值出错，读写dataRange的其它元素如min,max都正常。什么原因呢?

authors: ansonnw pissang


text:
echarts 3.7.0 座标轴文字颜色异常

version： 3.7.0
"valueAxis": {
    "axisLine": {
        "show": false,
        "lineStyle": {
            "color": "#666"
        }
    },
    "axisTick": {
        "show": false,
        "lineStyle": {
            "color": "#333"
        }
    },
    "axisLabel": {
        "show": true,
        "textStyle": {
            "color": "#999999"
        }
    },
    "splitLine": {
        "show": true,
        "lineStyle": {
            "color": [
                "#eee"
            ]
        }
    },
    "splitArea": {
        "show": false,
        "areaStyle": {
            "color": [
                "rgba(250,250,250,0.3)",
                "rgba(200,200,200,0.3)"
            ]
        }
    }
},

axisLabel color设置无效，所有的color都使用的是axisLine的color

不知你这段代码，是否是 theme？
如果是的话，那么可以把 axisLabel 改成：
"axisLabel": {
    "show": true,
    "color": "#999999"
}

因为 3.7.0 版本去掉了 label 中的 textStyle 层级。虽然，也兼容了老了 textStyle 写法，但是没有兼容 theme。这个兼容会马上修复。

对，是theme文件，去掉textStyle 层级就好啦，3q


text:
生成HTML时去注释

有没有可能给个选项，生成HTML时把注释都去了？

感觉这不是模板引擎的职责

理解楼上的说法，但是考虑到模板引擎在parse阶段其实有这些元数据……

有多迫切？

不是那么迫切，慢慢来
errorrik notifications@github.com于2016年6月21日 周二上午11:05写道：

有多迫切？
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ecomfe/etpl/issues/72#issuecomment-227328703, or mute
the thread
https://github.com/notifications/unsubscribe/AAnCPVs9RR_naJobMo8nGPYu_xMYxTqpks5qN1UIgaJpZM4Ivh10
.

其实我是在问有多想要这个功能，实现的话应该没几行

好吧其实现在来看我不需要了，本来是想通过注释去搞那个依赖管理的，你都直接做完发版本了- -
errorrik notifications@github.com于2016年6月22日周三 下午1:35写道：

其实我是在问有多想要这个功能，实现的话应该没几行
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ecomfe/etpl/issues/72#issuecomment-227648062, or mute
the thread
https://github.com/notifications/unsubscribe/AAnCPQCasgTuLmUvBIqdhXLV8UywH__Oks5qOMmLgaJpZM4Ivh10
.

authors: errorrik otakustay tower1229

repo: ecomfe/etpl


text:
create archive Jurors widget

after switching to new edition - current jurors are gone.

a722c75f6603f2f8f152eaabfb28cbb5ff746651


text:
show votes

@az the number is not visible in preview so I assume that it's API case again

@KarolinaTEH yep...im unblocking this in the API (its dual-way because there were situations they inspected the data-source and have had some "problems" why the number of votes decreased :)

authors: KarolinaTEH az


text:
Need instructions for how to get Black running locally

@alanlujan91 will review the docs that Mridul writes

It's already available at https://github.com/econ-ark/HARK/blob/master/Documentation/contributing/CONTRIBUTING.md#stylistic-guidelines

authors: MridulS llorracc

repo: econ-ark/HARK


text:
When Total=0 and Miss=0 I would expect the result to be SUCCESS rather than FAILED

Environment
python 3.7
interrogate 1.3.1
Description of the bug
When nothing is missed but the coverage is 0 because there was nothing to document, I would expect the result to be SUCCESS.

Summary -
| Name  | Total | Miss | Cover | Cover% |
|-------|-------|------|-------|--------|
|-------|-------|------|-------|--------|
| TOTAL |     0 |    0 |     0 |   0.0% |
RESULT: FAILED (minimum: 60.0%, actual: 0.0%) -

What you expected to happen
I would expect the result to be successful
How to reproduce (as minimally and precisely as possible)
Create a module that do not contain any methods or properties. (In my case it was a module only defining type aliases.
Use this configuration:
[tool.interrogate]
ignore-init-method = true
ignore-init-module = true
ignore-magic = true
ignore-semiprivate = true
ignore-private = true
ignore-property-decorators = true
ignore-module = true
ignore-nested-functions = true
fail-under = 60
exclude = ["setup.py", "docs", "build", "test"]
verbose = 1
quiet = false
whitelist-regex = []
color = true

FYI the fix has been released in 1.3.2 - thanks!

authors: JulienLajugie econchick

repo: econchick/interrogate


text:
🛑 Sandspur is down

In a6d480d, Sandspur (https://sandspur.co.uk/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Sandspur is back up in c1c02f7 after 6 minutes.

repo: eddiebt/BTCWebMonitor


text:
Error during generation of java files from multiple schema files

If several schema files are used to describe a set of related entities, the generation fails with this message:
[ant:xjc] [ERROR] "file:/dev/project/src/main/xsd/type.xsd" is not part of this compilation. Is it a mistake for "file:/dev/project/src/main/xsd/product.xsd" ?

The problem occurs because the plugin process each file separately. All the schema files should be processed in one pass to resolve the issue.

Thanks for all the help with this! I've released version 1.3 to the plugin portal.

authors: edeandrea lbar

repo: edeandrea/xjc-generation-gradle-plugin


text:
Нові змінні в helm.charts повинні додаватись для всіх середовищ

Додати авто-скрипт

duplicates #1393

authors: AlexKovalevych lymychp


text:
Error $ data - invalid_age PROD (j228)

$ data - invalid_age : лікарі із сертификатом сімейного лікаря не можуть заключити декларацію із 17-річними.
MIS:
MedCard Plus
Середовище:
Prod
Steps to reproduce:
Попередня умова:
Лікарі лікарні працюють на посаді терапевта, але мають 2 сертифіката: терапевта та сімейного лікаря (у системі)
Кроки:

Створюють декларацію на пацієнта 08.02.2001 року народження
При відправлені декларації на реєстрацію вртається помилка $ data - invalid_age
https://drive.google.com/drive/u/0/folders/1Ym3qPa_sPNrG_OJnm4ybnqo2hwmkouOY

Expected results:
До серпня місяця ці лікарі успішно підписували декларації із 17-ти річними, бо мають сертифікат сімейного лікаря.

Добрый день.
Попрошу вас написать больше данных по запросу: ид врача, ид пациента, логи запроса.
Также могли бы пожалуйста пояснить что именно вы подразумеваете под "сертификатом сімейного лікаря"?
Это массовая ошибка или наблюдается у одного лишь врача?
Спасибо большое.

Ошибка массовая.
ранее было так - врач со СПЕЦИАЛЬНОСТЬ семейная медицина может заключать декларации со всеми возрастами. У врачей этой поликлиники 2 специальности- 2 сертификата: терапия и семейная медицина. Должность у них - терапевт, почти у всех. Они терапевты в том числе и по договору. Теперь вопрос: если логика центрального компонента в этом аспекте поменялась, где об этом можно почитать и что теперь делать этим врачам?
Лог добавлен в txt файле:
https://drive.google.com/drive/u/0/folders/1Ym3qPa_sPNrG_OJnm4ybnqo2hwmkouOY

Добрый день. Логика не менялась. Если у врача две специальности, значит у врача есть два employee. У каждого employee есть speciality_officio, согласно которой врач именно с этой специальностью может или нет заключать декларацию именно с этим поциентом.
В данном случае у врача только одна специальность - THERAPIST, а поциенту еще нет 18-ти лет, следовательно декларация между ними заключена быть не может.

authors: ehealthua-tech svetavedmed


text:
Mithril testing

[ ] Підтвердження

[x] редагувати підтвердження:

[x] scopes should be editable
[x] ID shouldn't be editable

[x] створити нове підтвредження: change button name to зберегти підтвердження

done

authors: ivchenkodima lymychp


text:
KeePassDX icon: fix typos in IDs

😳 Apologies for the snafu here. keypassdx -> keepassdx

Easily done :-)

authors: admvx edent

repo: edent/SuperTinyIcons


text:
Very poor performance on migrating to huge schemas

Hey,
it seems that the migration hangs when trying to initialize this ESDL schema using edgedb --tls-security=insecure -P 5656 migration create --non-interactive.
What I've tried

I have tried multiple times and waited 1+ hour (and counting) on it to finish.
I have also tried using the WSL edgedb instance instead of running edgedb from docker, but I end up with the same result.

My Config Setup

Docker-compose file
EdgeDB Version: 1.4+ffe1237
OS Version: Win11 (docker/wsl)

Server log
(base) C:\work\code\ifcdb\dbs\ifcedge>make up
docker-compose up
[+] Running 1/0
 - Container ifcedge-edgedb-1  Created                                                                                0.0s
Attaching to ifcedge-edgedb-1
ifcedge-edgedb-1  | Applying schema migrations...
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:18:59.925 edb.server: starting EdgeDB server 1.4+ffe1237, built on 2022-05-16T18:45Z from revision e4c6e5f1c (2022-05-16T01:08Z) for x86_64-unknown-linux-gnu
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:19:01.168 edb.server: Using 159 max backend connections based on total memory.
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:01.219 postgres: starting PostgreSQL 13.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:01.222 postgres: listening on Unix socket "/tmp/tmp.NTtyXnaDNi/.s.PGSQL.5432"
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:01.232 postgres: database system was shut down at 2022-06-07 15:18:01 UTC
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:01.238 postgres: database system is ready to accept connections
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:19:01.915 edb.server: Connection established to backend database: edgedb
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:19:03.876 edb.server: started 1 compiler worker process
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:19:03.879 edb.server: Serving on 127.0.0.1:35411
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:19:03.879 edb.server: Serving admin on /tmp/tmp.NTtyXnaDNi/.s.EDGEDB.admin.35411
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:19:04.519 edb.server: Connection established to backend database: edgedb
ifcedge-edgedb-1  | Everything is up to date. Revision initial
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:19:04.646 edb.server: Received signal: Signals.SIGTERM.
ifcedge-edgedb-1  | INFO 37 2022-06-07T15:19:04.646 edb.server: Shutting down.
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:04.832 postgres: received fast shutdown request
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:04.837 postgres: aborting any active transactions
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:04.838 postgres: terminating connection due to administrator command
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:04.838 postgres: terminating connection due to administrator command
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:04.838 postgres: background worker "logical replication launcher" (PID 60) exited with exit code 1
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:04.845 postgres: shutting down
ifcedge-edgedb-1  | INFO 53 2022-06-07T15:19:04.883 postgres: database system is shut down
ifcedge-edgedb-1  | INFO 1 2022-06-07T15:19:06.995 edb.server: starting EdgeDB server 1.4+ffe1237, built on 2022-05-16T18:45Z from revision e4c6e5f1c (2022-05-16T01:08Z) for x86_64-unknown-linux-gnu
ifcedge-edgedb-1  | INFO 1 2022-06-07T15:19:08.061 edb.server: Using 159 max backend connections based on total memory.
ifcedge-edgedb-1  | INFO 156 2022-06-07T15:19:08.101 postgres: starting PostgreSQL 13.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
ifcedge-edgedb-1  | INFO 156 2022-06-07T15:19:08.104 postgres: listening on Unix socket "/run/edgedb/.s.PGSQL.5432"
ifcedge-edgedb-1  | INFO 156 2022-06-07T15:19:08.113 postgres: database system was shut down at 2022-06-07 15:19:04 UTC
ifcedge-edgedb-1  | INFO 156 2022-06-07T15:19:08.117 postgres: database system is ready to accept connections
ifcedge-edgedb-1  | INFO 1 2022-06-07T15:19:08.772 edb.server: Connection established to backend database: edgedb
ifcedge-edgedb-1  | INFO 1 2022-06-07T15:19:10.975 edb.server: started 6 compiler worker processes
ifcedge-edgedb-1  | INFO 1 2022-06-07T15:19:10.977 edb.server: Serving on {0.0.0.0, ::}:5656
ifcedge-edgedb-1  | INFO 1 2022-06-07T15:19:10.978 edb.server: Serving admin on /run/edgedb/.s.EDGEDB.admin.5656
ifcedge-edgedb-1  | INFO 1 2022-06-07T15:19:15.993 edb.server: Connection established to backend database: edgedb
ifcedge-edgedb-1  | INFO 1 2022-06-07T15:23:08.973 edb.server: Connection discarded to backend database: edgedb

Can it be due to the size of the schema? If so should I break up my schema into modules, or could there be some definitions inside my schema that are causing stuff to break? Any help on this is much appreciated!
Let me know if there is anything else I can provide to help advance this issue,
Best Regards
Kristoffer

The size of the schema is definitely a factor here and is most likely due to https://github.com/edgedb/edgedb-cli/issues/652.  I'll see if we can fix this shortly.

A small update.
The last migration I started yesterday was finally done today. However, when I run edgedb --tls-security=insecure -P 5656 migrate I end up with a out of memory error:
error message
edgedb --tls-security=insecure -P 5656 migrate
edgedb error: InternalServerError: out of memory
  Hint: This is most likely a bug in EdgeDB. Please consider opening an issue ticket at https://github.com/edgedb/edgedb/issues/new?template=bug_report.md
  Server traceback:
      edb.errors.InternalServerError: out of memory

server log
edgedb_1  | INFO 1 2022-06-07T13:09:04.928 edb.server: Serving on {0.0.0.0, ::}:5656
edgedb_1  | INFO 1 2022-06-07T13:09:04.930 edb.server: Serving admin on /run/edgedb/.s.EDGEDB.admin.5656
edgedb_1  | INFO 1 2022-06-07T13:09:55.584 edb.server: Connection established to backend database: edgedb
edgedb_1  | INFO 1 2022-06-07T13:13:01.950 edb.server: Connection discarded to backend database: edgedb
edgedb_1  | INFO 1 2022-06-07T15:48:53.127 edb.server: Connection established to backend database: edgedb
edgedb_1  | INFO 1 2022-06-07T15:50:53.122 edb.server: Connection discarded to backend database: edgedb
edgedb_1  | INFO 1 2022-06-07T15:52:53.141 edb.server: Connection discarded to backend database: edgedb
edgedb_1  | INFO 1 2022-06-08T06:43:52.330 edb.server: Connection established to backend database: edgedb
edgedb_1  | INFO 1 2022-06-08T06:43:53.360 edb.server: Backend connections to database edgedb: 1 were established in at least the last 1.0 seconds.
edgedb_1  | INFO 1 2022-06-08T06:47:52.379 edb.server: Connection discarded to backend database: edgedb
edgedb_1  | ERROR 207 2022-06-08T07:25:21.644 postgres: out of memory
edgedb_1  | INFO 207 2022-06-08T07:25:21.644 postgres: Cannot enlarge string buffer containing 0 bytes by 1161355502 more bytes.
edgedb_1  | INFO 1 2022-06-08T07:25:21.663 edb.server: Connection discarded to backend database: edgedb
edgedb_1  | INFO 1 2022-06-08T07:25:22.690 edb.server: Backend connections to database edgedb: 1 were established in at least the last 1.0 seconds.

this was using docker on Windows 10.

Okay, I see. Thanks for the feedback!
Would it be possible for me to improve performance by splitting up the schema into smaller independent chunks and do multiple migrations to complete the entire schema?

Hello again, I noticed https://github.com/edgedb/edgedb-cli/issues/652 was closed and https://github.com/edgedb/edgedb-cli/pull/744 was merged.
I am happy to help with testing the latest changes! Is there a development build available that I can test or should I build the latest from the master branch?
Anyways, let me know if there's anything I can do to help advance this issue?
Best Regards
Kristoffer

Hello again!
I tried this again recently using EdgeDB v2.1+c600476 and the migration create step finished in about 3,5 hours (which is a whole lot better than what I experienced before raising this issue). For reference the complete schema can be found here -> https://github.com/Krande/ifcdb/blob/master/src/server/schema/complete_schema/dbschema/default.esdl
Unfortunately, applying the migration fails with a edgedb error: ClientConnectionEosError in my termina.
And I've noted the following message as a potential error source from the container logs:
INFO 207 2022-08-11T19:04:37.396 postgres: invalid message length
Full trace from docker container from the beginning of migration apply
DEBUG 1 2022-08-17T08:51:57.628 edb.server: received connection request by edgedb to database edgedb
2022-08-17T08:51:57.629559100Z DEBUG 1 2022-08-17T08:51:57.629 edb.server: successfully authenticated edgedb in database edgedb
2022-08-17T08:51:57.745304300Z DEBUG 1 2022-08-17T08:51:57.744 edb.server: Establishing new connection to backend database: edgedb
2022-08-17T08:51:57.768206800Z INFO 1 2022-08-17T08:51:57.767 edb.server: Connection established to backend database: edgedb
2022-08-17T08:51:57.775936900Z DEBUG 1 2022-08-17T08:51:57.775 edb.server: Establishing new connection to backend database: edgedb
2022-08-17T08:51:58.797130100Z INFO 1 2022-08-17T08:51:58.796 edb.server: Backend connections to database edgedb: 1 were established in at least the last 1.0 seconds.
2022-08-17T08:55:57.780565800Z DEBUG 1 2022-08-17T08:55:57.779 edb.server: Discarding a connection to backend database: edgedb
2022-08-17T08:55:57.780656900Z INFO 1 2022-08-17T08:55:57.780 edb.server: Connection discarded to backend database: edgedb
2022-08-17T09:33:46.672554400Z INFO 180 2022-08-17T09:33:46.671 postgres: invalid message length
2022-08-17T09:33:46.689495700Z DEBUG 1 2022-08-17T09:33:46.689 edb.server: Discarding a connection to backend database: edgedb
2022-08-17T09:33:46.689552500Z INFO 1 2022-08-17T09:33:46.689 edb.server: Connection discarded to backend database: edgedb
2022-08-17T09:33:46.689568600Z DEBUG 1 2022-08-17T09:33:46.689 edb.server: Establishing new connection to backend database: edgedb
2022-08-17T09:33:47.721956200Z INFO 1 2022-08-17T09:33:47.721 edb.server: Backend connections to database edgedb: 1 were established in at least the last 1.0 seconds.

Server: EdgeDB server 2.1+c600476
CLI: EdgeDB CLI 2.1.0-dev.836+e76c780
OS: Windows 10
At the moment I have been managed to test my schema conversion by limiting the conversion to smaller portions of the schema. One example of a partial schema I've successfully ported to EdgeDB so far is around 229 unique entities. I've attached a figure showing the partial schema in all of its glory using the EdgeDB UI :)
I have pondered the possibility of maybe splitting the schema into modules. However, it is not necessarily easily done, and I am not certain that it would actually solve my issue.
So if anyone has any advice on this I would really appreciate it!?
Best Regards
Kristoffer

You don't have to split into modules, just expand your schema incrementally (add types/create migration/migrate etc).  The issue here is the sheer size of the generated migration DDL, both at EdgeQL and SQL level, the latter hitting PostgreSQL internal limits.

@elprans Thank you for the advice! I really appreciate you taking the time.
So I am currently testing various migrations incrementally with the aim of finding a reasonable increment size that does not hit the internal limits.
FYI:  The way I am approaching this is to
First write the .ESDL schema with the number of entities equal to the increment size,

run migration create --non-interactive
run migration apply

The add a number of entities equal to my increment size and re-do the steps above.
My first attempt with 300 entities per increment did successful reach 900 entities (out of 1467), but the increment was probably too large, so it resulted in the following error:

Great Success!
After having tried increments of 600 and 300 (both resulting in invalid memory alloc errors)I finally found a working increment size of 100 :) It took approximately 20 hours to complete, but at least I have a complete schema in my database!
Any chance there is a way to export and transfer my instantiated schema to another EdgeDB database instance without having to spend another 20 hours to create/apply my migrations?

Any chance there is a way to export and transfer my instantiated schema to another EdgeDB database

The easiest way would be to do a filesystem copy of the data directory.  Shut down the instance first (edgedb instance stop).  You can find the data directory by running edgedb instance status --extended -I <instance>

Hello again! Thank you for the response! I really appreciate it :)
I have some more issues related to migrating larger schema and performance issues,
When running migration create on a schema consisting of 254 types it takes approximately 5-10 minutes locally on a docker-compose setup. While doing the exact same on my azure deployment it runs for hours without finalizing.
To circumvent this I am now resorting to the same "stepwise migration" that I did for my entire schema of 1000+ types.
If that fails I will resort to a filesystem copy of the data directory as proposed for my entire schema.
Regarding my azure deployment. Are there any configuration options that I can use to improve the migration speeds on my azure deployment (either on the posgresql or edgedb side)?

Just wanted to chime in and say that I've also been experiencing lengthy (15+ minute) migration times even after breaking my initial migration into multiple smaller pieces.  Unfortunately that has not alleviated the issue of each successive migration taking longer and longer, ultimately culminating in edgedb error: ClientConnectionEosError.  To be clear I'm getting that error during the migration creation process, not the migration itself, even with a workaround I found in the discussions (configure instance set session_idle_transaction_timeout := <duration>'0';).  Frustratingly, sometimes schema errors will only be relayed at the end of a long wait as well.
Model declarations in edgedb have been a nice change of pace from ORMs so far, but I am honestly concerned about the migration issues going forward. The time it takes for the migration process of a non-trivial project doesn't feel sustainable, and unlike ORMs the entire process is opaque with no lower level escape hatch available.

Hello! :wave: Jumping in as well to say I'm new to edgedb and also being hit by this.
I have found Krande's stepwise approach in python and plan to do something similar in Go. The need for this in two languages makes me think incremental migration might be something useful for the CLI to support natively.
Alternatively/additionally, it would also be helpful to have client support for migrations, so that at least users wouldn't need to shell out to the CLI for each increment.

authors: Krande elprans stone-z zerosym

repo: edgedb/edgedb


text:
init ttls only if config was set

Signed-off-by: Leonard Cohnen leonard.cohnen@gmx.de

redundant https://github.com/edgelesssys/edgelessrt/blob/master/src/tools/erthost/ttls.cpp#L114

repo: edgelesssys/edgelessrt