Continuing on from the previous couple of articles, I’m going to introduce a number of integration approaches and provide a basic overview of the benefits they provide. This won’t be an exhaustive list, merely a brief sample.
Using an Integration Approach
These conditions could be used to determine when to consider implementing advanced system integration (as a general guideline):
- A write action occurs (when considering an integration bus),
- An orchestration – a request which requires access to more than one LOB application/service – is required,
- An existing service or application doesn’t support asynchronous operations, but there exists an integration requirement to consume the service or application asynchronously,
- A request is required from a data store which does not publish an API (e.g. a database),
- Data format/transformation is required,
- Queuing (message resilience, ordered delivery) is required
- An auditable (non-repudiated) action needs to be supported end-to-end,
- These points are further defined, with examples, in the following section.
System integration may be required based on some of the following needs:
- Tailoring integration to operational need (e.g. end-to-end auditing of critical business operations, ordered delivery etc.),
- To provide support for two-phase commits (commit to source of truth, then apply eventual consistency – BASE – logic to dependency apps),
- Message resiliency built-in (system-to-system calls require this to be explicitly implemented),
- Implementing whole of enterprise data model and publish contracts/interfaces which transact with this model (data definition uniformity),
- Abstraction of underlying systems,
- When a “fire-and-forget” operation is suitable,
- High volume/low latency transactions (although sometimes a direct read or write might be a better fit, e.g. BCP into SQL)
Integration Approaches and Benefits
This is a small sample of some of the approaches, I’ll add more later.
System/Source Abstraction
There is value in abstracting line of business systems from enterprise integration implementation. The goal is to ensure that if a system or application needs to be retired, it can be replaced by simply implementing the same abstraction system interface previously defined, potentially leading to reduced effort and impact to integrated systems. This can also apply to writing an interface to “wrap” a COTS API, where the bespoke interface maps the COTS interface to your own domain model.
Data transformation
Working with many different systems and proprietary products typically introduces a number of different data formats and approaches. Integration solutions can cater for different data formats by providing rich data mapping and transformation functionality, enabling system integration between systems which feature normally incompatible data structures.
This is particularly advantageous when working with proprietary interfaces or standards based data definitions.
With the establishment of a mature middleware solution, for example IBM Integration Bus or Microsoft’s BizTalk Server, it is possible to fulfil multi-system integration and data transformation which consuming an array of different data integration options, e.g. through APIs and direct data access.
Support for concurrency
There are two types of concurrency models: pessimistic and optimistic concurrency.
Pessimistic Concurrency
Pessimistic concurrency reserves resources by placing a lock on them which prevents other processes from making changes to (or even reading from) resources which are locked. In a distributed orchestration scenario, this would require the use of distributed transactions to maintain locks and resource state for orchestrated operations.
Pessimistic concurrency is stateful, in that locks must be maintained for this model to be effective in preventing concurrency violations. Transactions typically implement the principles of ACID (see below), and this can be extended to distributed orchestrations, in that an orchestration (particularly within Microsoft’s BizTalk Server) can maintain rollback functionality which can be triggered by failed/rolled back database transactions.
Optimistic Concurrency
By contrast, optimistic concurrency does not employ resource locks, and attempts to manage resources under the presumption that the resources have not changed since they were read from. Implementing this model requires strong consideration of how concurrency violations should be handled.
The most common scenario I’ve seen successfully implemented is “last write wins”, where the most recent changes overwrite older ones, even if the new changes do not incorporate some of the earlier changes. This model is stateless as locks do not need to be maintained. Most implementations rely on the use of a timestamp or another kind of signature which represents a “version” of a record, which indicates a point-in-time state for the record (to check if a record has changed).
In scenarios where concurrency models have been implemented in integrated services or applications, integration middleware must be designed to be aware of the concurrency used and designed to handle concurrency violations appropriately. I’ve successfully developed large scale integration architectures which use optimistic concurrency and the Eventual Consistency (BASE) model.
Related: Transactional (ACID)
In computer science[1], ACID (Atomicity, Consistency, Isolation, Durability) is a set of properties of database transactions.
Within the scope of integration, we can expand ACID to include distributed transactions, particularly the principal of atomicity, which roughly translates to “all actions succeed or all changes are rolled back” to ensure correct operational state of data.
This is typically found developed as part of integration orchestration implementations which contain error handling logic which is used to revert changes made by the orchestration in the event of an error or processing issue, in place of manual remediation.
Related: Eventual Consistency (BASE)
Eventually consistent services[2] are often classified as providing BASE (Basically Available, Soft state, Eventual consistency) semantics, in contrast to traditional ACID (Atomicity, Consistency, Isolation, Durability) guarantees.
Integration designs can incorporate the principle of BASE consistency through a variety of means. One approach is to commit data or process changes firstly to a “source of truth” application or service and then replay these changes to effected systems, either directly or indirectly. This leads to eventual consistency of the data related to the integration operation.
BASE consistency can also be achieved in scenarios where manual remediation of integration issues may apply; for example manual re-queuing of failed messages.
Auditing/Non-repudiation
Solutions may be able to incorporate auditing of end-to-end message orchestration (or processing) by incorporating integration middleware into designs. This involves creating an audit record of each step of a message lifecycle, independently of when and how a message’s data is consumed by line of business and other interested systems.
A key principal benefit of message/orchestration auditing is the concept of non-repudiation, which asserts that the processing of a message occurs exactly once mainly to avoid message tampering. Any attempts to reprocess a message or orchestration does so against a brand new copy of the data, thus creating a clear audit record of actions taken against each instance of an integration orchestration or integration action.
More to follow…
[1] Source: https://en.wikipedia.org/wiki/ACID
[2] Source: https://en.wikipedia.org/wiki/Eventual_consistency
