Middleware concepts: An introduction

Middleware concepts: An introduction

Organizations have external relationships with partners and service providers to enable the delivery of products and services to their clients. Internally, business units and disparate corporate systems need to exchange data with each other as well as with external partners to facilitate the automation of business processes.
With corporate technology policies evolving more and more towards buying off the shelf over in-house systems development, integration mechanisms (i.e. middleware) are becoming increasingly more important to connect these off-the-shelf systems to core business systems such as the corporate ERP. Besides a number of other integration concerns like Master Data and Meta Data management, automating the integration of off-the-shelf systems is becoming the central focus of many software corporate development teams. The move towards the use off-the-shelf systems has gained significant momentum as the number of such offerings on the market increases rapidly. It is becoming harder to find a reason to do custom development over buying off the shelf every day, if not buying off the shelf one should at least be looking into buying into solution platforms on which fit for purpose solutions can be assembled rather than built from scratch.
I have found that the term ‘middleware’ was increasingly being used by both IT and non-IT staff without a clear understanding of how the technology is actually used to deliver value. This article intends to give some insight into how middleware is used in the corporate environment. The objective is to provide a high level understanding of the technology so as to align understanding and facilitate communication between technologists and the technologically challenged. We discuss why middleware is used, assess some key design considerations and then go through a use case to demonstrate how the technology is used in practice.

Why middleware?

As explained in the opening paragraph, disparate systems need to integrate internally and externally. Most computer systems cannot automatically just talk to each other, either technically due to their technology platforms or in terms of how the data that they exchange is structured or defined.
Today, corporates invest in proven off-the-shelf software to increase corporate agility through reduced time to market, reduced risk and cost to implement amongst many other benefits. These new systems also need to talk to existing internal systems like the corporate ERP.
To facilitate communication between these systems, we need an intermediate system called middleware.

Key considerations

So how does one get System A to talk to System B? Simply put, we need an intermediate system that speaks the language of both systems to translate between the two. This translation is is not necessarily as simple as reformatting a data value, sometimes the meaning of a data element is different between systems, an example could be that "Total" in system A could be a value that includes tax whilst in system B it excludes tax, so the transformation of data would involve removing the tax value from the "Total" in system A when transferring it to system B.
If one of these systems are externally owned and operated by a business partner, it is unlikely that we could modify it. However, if you were able to modify either of the interfaces from system A or system B so that they speak the same language, what would happen when the next system update or upgrade comes along for either system A or system B? Updating a modified system may result in the modification being lost as files are overwritten or the way the base system works is changed, it is likely that you will have to plan a project to apply the same modifications to the new or updated system version with all the same development, testing and deployment overhead. Ideally the middleware should be implemented in such a way that the source and destination systems do not have to be modified at all. The middleware should simply translate system A speak to system B speak and vice-versa without changing either source or destination systems. This is not always possible, ideally middleware should avoid maintaining transactional system state or the transactional data of either system A or system B, in other words it should avoid becoming tightly coupled to or form part of system A and / or System B. Middleware should preferably be seamless without affecting source and destination systems.

Middleware Use Case

Middleware is used wherever two or more distributed systems need to exchange data, these systems may be internal or external and they may be private or public. Below is a use case that demonstrates a concrete example of how middleware is used in the corporate, maybe you can even see where it would be used in the company in which you work.
The scenario demonstrates the use of middleware between internal and external partner entities:

MyCompany has a corporate credit card expense management system that has been purchased off the shelf. This system processes corporate credit card transaction data from their card issuing bank, MyIssuing Bank. Corporate card owners approve transactions that they have performed monthly and allocates them to appropriate cost centres together with supporting documentation. Their line managers are notified and they in turn approve these transactions as appropriate. Approved transactions are submitted to the ERP for internal cost centre allocation processing. The ERP may require exchange rates information to convert transaction amounts to a US dollar base currency, a publicly available daily exchange rates service provided by Reuters is used to update the ERP daily.

Step by Step Discussion

Path A

  • Step 1
    The middleware system downloads daily card transaction data over a secure encrypted channel from MyIssuing Bank, the downloaded data files are additionally encrypted with a unique key – this is decrypted locally in the middleware. Now that the data is accessible, the middleware transforms some of the data to align it with the card expense management system’s data formats, this task could include simple transformations like matching the input date format of the card system.
  • Step 2
    The card expense management system notifies card holders and executes its workflow as described in the use case above. Processed data is scheduled at a fixed time to be generated as output files in an output folder.
  • Step 3
    The middleware system monitors the output folder in which file are generated by the card expense management system, when a file arrives it loads the processed file and transforms some of the data to align it with ERP data formats. The middleware inserts the transformed data directly into the ERP database staging area and executes an ERP program to process this data into the ERP system.

Path B

  • Step 1
    The middleware downloads daily exchange rates data over a secure encrypted channel from Reuters, it transforms some of the data to align it with ERP data formats.
  • Step 2
    The middleware inserts the transformed exchange rates data directly into the ERP database staging area and executes an ERP program to process the newly loaded data.

In summary ...

Middleware is an immensely powerful tool to integrate systems across boundaries and add scalability between systems, e.g. a busy web system could use middleware to interface to a backend transactional system for order processing and so on. There are also a significant number of ways that middleware can be implemented to meet varying integration and non-functional requirements.

In the next middleware article, I will step up the tech speak and talk about three middleware related technology concepts that I found were being used quite often in recent projects that we have been involved in, i.e. EAI vs ESB vs MFT (Enterprise Application Integration, Enterprise Service Bus and Managed File Transfer).