Stung by HornetQ

I've recently been upgrading our JBoss 5.1.0.GA and JBoss Messaging based application to JBoss 6.0.0.Final and HornetQ. The end result has been positive: HornetQ is very fast. But there were a few gotchas I thought I'd share:

1. No More Database, Lots More Backups

HornetQ eschews the JBoss Messaging approach of storing messages in a shared database. Instead it uses the file system directly (and, on Linux, accelerated AIO). But note this doesn't mean a shared file system. It means each cluster node has its own file system, with peer-to-peer communication to distribute messages between them.

The advantage of this approach is you no longer have a Single Point of Failure (ie. the database). But it's quite a shift in mindset if you're used to thinking of your database as a heavy, fault-tolerant kind-of-thing and your nodes as lightweight, throwaway kind-of-things. Because now, each node stores a portion of your messages. So if you lose a node you lose some messages.

The recommended solution to this is to have backup nodes. Whereas before you might have one big backup instance for your database, and no backups for your nodes, now you need a backup for each node. See this forum thread. Confusingly, when talking about backup nodes you may choose to use a shared file. But this is only for sharing between the 'backup' and 'live' instance of a single node.

2. hornetq-jms.xml Is A Red Herring

Out of the box, JBoss 6.0.0.Final includes a server/all/deploy/hornetq/hornetq-jms.xml that looks like this:

<configuration xmlns="urn:hornetq">

   <connection-factory name="NettyConnectionFactory">
      <connectors>
         <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
         <entry name="/ConnectionFactory"/>
         <entry name="/XAConnectionFactory"/>
      </entries>
   </connection-factory>
   
   <connection-factory name="NettyThroughputConnectionFactory">
      <connectors>
         <connector-ref connector-name="netty-throughput"/>
      </connectors>
      <entries>
         <entry name="/ThroughputConnectionFactory"/>
         <entry name="/XAThroughputConnectionFactory"/>
      </entries>
   </connection-factory>
   
   <connection-factory name="InVMConnectionFactory">
      <connectors>
         <connector-ref connector-name="in-vm"/>
      </connectors>
      <entries>
         <entry name="java:/ConnectionFactory"/>
         <entry name="java:/XAConnectionFactory"/>
      </entries>
   </connection-factory>

   <queue name="DLQ">
      <entry name="/queue/DLQ"/>
   </queue>
   
   <queue name="ExpiryQueue">
      <entry name="/queue/ExpiryQueue"/>
   </queue>

</configuration>

To my mind this is very confusing, because if you're using JMS producers and MDBs all those <connection-factory> configurations aren't used! I would recommend deleting them, for 3 reasons:

1. You see the /ConnectionFactory JNDI reference in there and think you should start using ic.lookup("/ConnectionFactory") in your code. If you do, you'll get non-transacted queue sessions, duplicated messages, lost messages, and all sorts of other weirdness. Stick with ic.lookup("java:JmsXA")


2. If using java:JmsXA, those <connection-factory> configurations don't apply! You need to look instead in server/all/deploy/hornetq/jms-ds.xml. This is using InVMConnectorFactory implicitly, but you can explicitly configure it


3. If using MDBs, again those <connection-factory> configurations don't apply! MDBs are configured in server/all/deploy/jms-ra.rar/META-INF/ra.xml. They use InVMConnectorFactory by default.

This last point is a real doozey. Because if you want to simulate a heavyweight store of messages with lightweight consumers who read from it (like JBoss Messaging) you're going to want to use <consumer-window-size>0</consumer-window-size>. You'll find lots of examples on the Web of putting this in hornetq-jms.xml. But consumer-window-size won't work there, because MDBs don't use hornetq-jms.xml. Instead you need this in ra.xml:

<config-property>
   <description>The consumer window size</description>
   <config-property-name>ConsumerWindowSize</config-property-name>
   <config-property-type>java.lang.Integer</config-property-type>
   <config-property-value>0</config-property-value>
</config-property>

3. hornetq-configuration.xml Is Noisy

I'm a 'break it to learn it' kind of guy. So to my mind the connectors and acceptors section of hornetq-configuration.xml is a bit noisy. Here's what you can reduce it to:

   <connectors>
      <!-- Node to node communication -->
      <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host" value="${jboss.bind.address:localhost}"/>
         <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
      </connector>
   </connectors>

   <acceptors>
      <!-- Node to node communication -->
      <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host" value="${jboss.bind.address:localhost}"/>
         <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
      </acceptor>
      <!-- jms-ds.xml produces to, and ra.xml consumes from, InVMConnectorFactory by default -->
      <acceptor name="in-vm">
         <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>
         <param key="server-id" value="0"/>
      </acceptor>
   </acceptors>

Your Mileage May Vary

Of course, these tips aren't for everyone. But if you're like me they may save you a few hours banging your head against the desk!