Stung by HornetQ: The Revenge

Following on from my previous post I've been spending some more time with HornetQ and have discovered a few more gotchas:

4. Stuck By Default

There's some advice here that says...

"Probably the most common messaging anti-pattern we see is users who create a new connection/session/producer for every message they send or every message they consume. This is a poor use of resources... Always re-use them"

...couple that with other advice that says...

"Please note the default value for address-full-policy [when the send buffer is full] is to PAGE [out to disk]"

And you might think the Right Thing To Do is set up a single connection/session/producer and send all your messages to the queue. But if you're doing this in a transaction (most Web applications are) you'd be wrong. Why? Because there's some conflicting advice that says...

"By default, HornetQ does not page messages - this must be explicitly configured to activate it"

And a setting in the JBoss/HornetQ integration (hornetq-configuration.xml) that says...

<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>

So by default JBoss will get stuck if you try sending more than 10MB of messages in a single transacted producer (ie. before your MDBs can start consuming them). 10MB is not a lot. For me it was about 1,000 messages of about 5,000 characters each (a Unicode XML string).

Here are my suggestions:

1. HornetQ should treat its JBoss JMS integration as more of a first class citizen. It should be a primary use case, rather than relegated to a chapter at the back of the user guide. Why? Because most people that just dip into the User Guide are going to be doing so from a JBoss JMS mindset. So if you say something like "Please note the default value is to PAGE" then you need to also say, immediately afterwards, " (except on JBoss where the default value is to BLOCK)"


2. BLOCKing is a poor default value. Either make it fail (so the user gets an error) or make it PAGE (so the user gets an error when their disk is full). At least then the developer knows where to look. But blocking just results in the queue being 'stuck' - with no clue to the developer who has barely heard of their underlying JMS implementation, let alone blocking versus paging and <address-full-policy>

5. Stuck By Bugs

HornetQ is pretty new and there are a few bugs that can cause your JMS messages to get stuck. There's the fact that MDBs will rollback/retry indefinitely, that messages with different JMS priorities may get forgotten, that messages can be forwarded to dead cluster nodes.

When you have several different bugs interacting to produce an overall symptom (ie. a stuck queue) it can be very hard to separate them to understand their underlying causes. This causes a lot of pain!

6. Stuck By Birth

This one isn't really HornetQ's fault, but its behaviour seems different to JBoss Messaging. If your MDB uses @EJB injection you really need to setup <depends ... > blocks in your jboss.xml...

<message-driven>
   <ejb-name>LongRunningProcessConsumerBean</ejb-name>
   <destination-jndi-name>queue/avant-ss.long-running-processes</destination-jndi-name>
   <!-- Stop MDB consuming too early -->
   <depends>jboss.j2ee:ear=avant-ss-app.ear,jar=avant-ss-dev-ejb.jar,name=LongRunningProcessBean,service=EJB3</depends>
   <depends>jboss.j2ee:ear=avant-ss-app.ear,jar=avant-ss-dev-ejb.jar,name=BusinessRulesBean,service=EJB3</depends></message-driven>

...because HornetQ starts up and starts consuming very early. This is particularly bad because these errors go quietly into the boot.log, not the regular system.log, so you don't realise that your MDBs crashed on startup.