Blog
Integration Landscape & Org. structure
What's the system integration all about?
System to system integration landscape has evolved over the years. It’s about how two systems communicate, manage information exchange, etc. and go about doing their respective tasks. Like everything else, the ecosystem is changing at a pace that is faster than what organizations can adapt to. Organizations have inherited, acquired, created, maintained different versions of the various software products as per their needs from time to time. Many of these software assets need to be continuously fed upon with data in various forms and shapes to make them serve the organization honorably.
A dig little deeper, each system has a role play in the organization. Systems are responsible for certain piece of work, they may be managing/governing a set of data, a part or whole of a process for the organization. The systems produce and/or consume data in their own defined formats. The modern systems have been enhanced to support consuming and producing additional formats as well.
System integration landscape put forth some interesting patterns. It's what I call as "Pattern of Communication" .
Before we go any further on the systems integration, lets understand how this matches up with the Organization structure.
Sometime back many of us would have seen the organization structure of some IT companies doing rounds in the social media.
(Manu Cornet organizational chart, http://bonkersworld.net/organizational-charts)
Each of these organizational charts, apart from mocking upon the confusion & complexity of the structure also emphasized upon various ways of interactions between people in the organization. It also in a way manifests the different architectural constructs in the Systems Integrations world. As ultimately all of it boils down to "Pattern of Communication"
Following components constitute the "Pattern of Communication"
• Channel: Where to communicate. Whom to communicate. Can be a point to point or broadcast or listening to a broadcast.
- Channels are all about the end points.
- What's the address of the end points.
- Details on how to establish connection to the end points.
• Format: How to communicate. A pre-set data formats.
- Format of the payload.
- Format as per the producer of the message.
- Format as per the consumer of the message.
- Depending on the type of data exchange agreement between the Channels
(producer - consumer) there can be multiple types of formats
- Producer owned format
- Consumer understandable format
- Common format (agnostic of producer - consumer)
• Payload: What's the communication all about
- The actual meat of the message.
- More to do with the size of the payload.
- Information being exchanged between the producer - consumer
As we understand the above basics of Integration. There are various other advanced nuances
• Channel types:
- In the current world there's a plethora of endpoints available
- Cloud based end points like (some examples)
- Data steams (MS Azure Event Hubs, Google Cloud Pub/Sub, Amazon Kinesis, etc.)
- Blob Stores (MS Azure Blob Stores, Google Cloud Storage, Amazon S3, etc.)
- API Gateways (MS Azure API Management, Google Cloud Endpoints, AWS API Gateway, etc.)
- Other Enterprise and/or hosted options
- FTP and sFTP
- Enterprise Service Bus
- Message Queues (Kafka, RabbitMQ, IBM MQSeries, Apache Active MQ, etc.)
- Data Connectors (ODBC, JDBC, databases, etc.)
• Communication type
- Synchronous Communication
- Request-Response messaging
- Producer sends the message and waits on the consumer to respond. In this world its more so if the systems being called as Client & Server rather than being producer - consumer end points
- Request-Response messaging
- Whether the producer just fires the message and moves on
- Pub/Sub model or more so there can be one or more subscribers to a "topic" or "type" of message.
- The Producer sends the message and all subscribers receive the message
- Producer sends the message along with a callback URL.
- Consumer responds back using the callback URL.
• Transforms
- ETL - Whether the producer transforms the data for the consumer to understand
- ELT - Whether the consumer collects the data first and then transforms it as per its convenience
- No transform - The producer - consumer handshake on the same format there by its purely transfer of payload (data)
These are only way way high-level constructs that as an integration product owner, engineer or executive should be aware of. Each of the topics here in is worth a book in itself.
We'll go deeper in the future posts.