Thursday, 19 September 2013

MQ Queue with multiple consumers but only one active

MQ Queue with multiple consumers but only one active

We have one MQ Queue which receives messages from an external system out
of our control. Our system processing the incoming messages is a critical
one and needs to be up and running 27x7 no matter what.
The order in which the incoming messages are processed is also not
negotiable which means we need to process them in exactly the order they
arrived.
To make sure our system is 100% available we deployed our system to a
bunch of physical machines able to process those messages.
Once the messages reached our system we put in place a mechanism to make
sure the messages processing does not go out of order while also getting
some performance gain as a result of the parallel processing. For us the
performance gain is a good to have, however it is rather a side effect as
our main goal is the high availability while assuring the right processing
order.
My thoughts were to have on every single machine an MDB able to process
the incoming messages but only have one active consumer at a time.
We are using Webshere MQ as a JMS provider and Webshere Application Server
8.5 to deploy our application.
The problem with multiple consumers listening to the same queue does not
seem to be a workable solution as when messages arrive in bulk they would
be round-robin passed to all consumers and there is no way to control how
this is going to happen and the messages easily go out of sequence.
When I manually stopped all the listeners but one obviously the messages
got processed in order. But manually shutting down and starting up such
listeners is definitely not a HA solution.
We could put in place monitoring processes to check for health of the
system and shut things down or start them up as required but this still
looks too weak to me. What we want in fact is to have all listeners up and
running but only one receiving the messages. If that one goes down for
whatever reasons then another one sitting there will become active and
start processing messages.
Initially we considered using a topic rather than a queue but this comes
with other issues like below:
we cannot control the source of our messages
the high volume of messages we have would put us in in trouble with our
going down subscribers that have to be durable and such when coming up
back will have to deal with lots of pending messages
the input queues are already part of a cluster and changing all the
infrastructure would require a lot of work
Anyway in my view it has to be an existing pattern to accommodate
situations like this. Any help, suggestion would be greatly appreciated.
The solution does not have to be a specific MQ one, any idea is welcome.
Thanks in advance

No comments:

Post a Comment