IBM MQ is more than just queues and topics - it's a messaging network

When application developers think of IBM MQ, they think of queues and topics. That’s understandable, as that’s what their application interacts with. And, it’s great that they don’t really need to know any more, as that’s kind of the point of IBM MQ!

IBM MQ administrators on the other hand, typically recognize the product as something much more powerful. They see a connected set of queue managers spread across a broad IT landscape, seamlessly making queues and topics accessible locally to applications wherever they are connected. That "messaging network" can enable message transmission that spans multiple platforms, locations, and cloud providers without the applications using it needing to know how that happens. This is the real hidden beauty of IBM MQ, and one of the reasons it remains to this day the most deployed messaging network in the world.

What is a messaging network?

I want to dive into what we mean by a messaging network, why it’s so important at this particular time in the evolution of IT, and why it is that IBM MQ is so very good at it.

The idea of messaging is that once a message has been placed on a queue or topic, the application no longer needs to worry about it, and it can move on to the next task.

However, what if the recipient of that message is in a completely different IT domain? How does the message get there, and who is responsible for it while it is in transit?

A messaging network is a set of loosely connected messaging servers that enable reliable, cross-domain transmission and delivery of messages over a broad, potentially hybrid multi-cloud IT landscape.

When I defined messaging network above, I deliberately used the term “cross-domain” message transmission because I wanted to recognize an increasingly common need. Modern solutions are built across ever more heterogeneous landscapes. New innovations typically tie together functionality from very separate parts of a business (different domains), each of which has significant autonomy over their technological choices. For example, the runtimes that they use, the platforms that they run on (containers, virtual machines, mainframes), when they are available, their performance characteristics, and where they run (on premises or in one of the major hyperscalers)?

Frequently, many of the parts of a modern solution will be distant from one another both physically and technologically, especially if your solution uses capabilities from business partners external to your organization. This scenario is where the messaging network comes in, providing an asynchronous backplane that enables applications to find each other over a suitably decoupled transport.

For more theoretical discussion on the characteristics and benefits of a messaging network, read my “Medium article”. However, in this article, we're describing the key capabilities in IBM MQ that make it an enterprise-grade messaging network. First though, lets very briefly remind ourselves why we're choosing messaging in the first place.

What about APIs?

The go-to protocol for bridging these domains is often HTTP-based APIs.

They certainly have their place and enable a simple programming model for many simple tasks. However, for more complex interactions, it can push a lot of complexity back into your application. Interactions are dependent on the entire chain from caller to provider being available, and performant.

To make an application robust to the very likely possibility of issues in that chain, you have to build in retry logic, additional persistence, idempotence. You also have to consider the effect on your application’s performance and resources if the intervening network latency increases, or the downstream application is struggling with workload. Learn more about when to use APIs, when to use messaging, or when to use both, in this article, "An introduction to APIs and messaging."

This is the problem that messaging was designed for. Designing your solution to interact with queues in between your components decouples them from one another’s performance and availability characteristics.

Queue managers

In IBM MQ, queues are housed within a queue manager which makes queues available to suitably authorised clients over a variety of protocols.

A queue manager takes care of persisting the messages, performing any transactionality required, and assuring exactly once delivery. While the queue manager is a lightweight component, it is also highly optimized for performance and provides platform-specific high availability and disaster recovery options. Queue managers are the all critical nodes within a messaging network that provide the core asynchronous decoupling.

It's important to note at this point that although IBM MQ clearly has queues at its heart, the destination doesn’t have to be a single application. IBM MQ has also long had the notion of publish/subscribe via “topics”. These too are administered via queue managers and as such can be made available across a messaging network.

MQ channels

If you are communicating across domain boundaries, wherever the messaging server sits, at least one side will still have to perform a request over the domain gap, which means it is still dependant on the vagaries of the intervening network connectivity.

This is where the second construct of the messaging network comes to the rescue: “MQ channels”. Each application can speak to a messaging server in its own domain and then let the messaging servers (queue managers) worry about the gap. In IBM MQ, they do this via an efficient, highly resilient and scalable connection known as a channel.

Channels assure exactly once delivery between the queue managers. The applications need only interact with their nearest queue, and the queue managers and channels do the rest.

To long time users of IBM MQ, this benefit from delegating transmission to queue managers and their channels will be very familiar since server-to-server communication is one of the most mature areas of IBM MQ.

Most applications talk over the local network to their queue managers because cross-domain network communication can still be complex and susceptible to lag. In increasingly hybrid multi-cloud landscapes, there is significant benefit in persisting messages within the local domain and delegating across domains to queue managers and channels.

The real benefit of a channel is that once it is in place, it can be used to make any number of queues or topics visible across the gap, marshalling messages between an ever growing number of applications on each side.

Learn more about channels in this tutorial, "Sharing MQ JMS conversations over channel instances."

MQ clusters

What if we want to connect lots of queue managers together? It’s not unusual for a large IBM MQ landscape to have hundreds or even thousands of queue managers. In other words, what if we want to build a messaging network? Surely this would mean configuring all the different permutations of channels between all the queue managers?

Fortunately, IBM MQ has long had an answer to this challenge in the form of an MQ “cluster”. When queue managers are grouped into a cluster, they are all aware of one another’s queues.

An application can place a message onto any queue in the cluster via their local queue manager, and then IBM MQ will reliably route that message to wherever in the MQ network that queue exists. If the queue doesn’t exist on that initial queue manager, messages destined for it will be persisted in a transient queue ready for transmission across a channel. The application at this point can continue with other work as it is confident that the message will get to its destination without having to know anything about the route the message will take through the cluster. It doesn’t need to know what networks it will traverse, what bandwidth might be available, or the performance and availability of the queue managers along that route. The channels between the queue managers will handle all that.

The really clever thing about MQ clusters is that you don’t have to configure the channels at all, or even know where the queues are on the network. When a queue manager is added to a cluster, the channels are configured automatically and dynamically adjusted if the cluster changes. Queue definitions are also automatically and efficiently shared throughout the network such that messages can always find a path to their intended queue. This means the cluster has the knowledge to dynamically route around failures, keeping your messages moving regardless of network or component outages.

MQ clusters are easy to set up, and they provide huge simplifications to the overall configuration to the extent that some sites use clusters as their default way of adding queue managers into their landscape.

The messaging network takes care of the complexities of reliable transmission of messages regardless of how separated applications are, on different networks in different sites, and from differing platform providers. Once the channel (or cluster) is set up, messages can flow. Of course, there are plenty of access controls available, so we can still govern who can send what to whom.

Conclusion: Join your apps together with an MQ-based messaging network

We have discussed that IBM MQ is extremely well suited to the creation of a messaging network. It was designed specifically for this purpose decades ago with these capabilities built in to queue managers, channels, and clusters.

What you might not know is how much IBM MQ has continued to innovate to align with modern development principles and platforms. Did you know that you can spin up IBM MQ in a container within seconds, in a fraction of a CPU? Did you know that IBM MQ is available as a managed service? Perhaps it’s worth taking a fresh look at how you can robustly and securely join your applications together with queues and topics across a hybrid cloud landscape using the world’s most deployed messaging network?

IBM MQ, a high-performance, security-rich and assured delivery messaging network solution. To really learn all that IBM MQ has to offer, complete the IBM MQ Developer Essentials learning path.

Acknowledgements

Many thanks to the following for their invaluable insights and review: Anthony Beardsmore, Andy Garratt, Callum Jackson, Matt Roberts, David Ware.