Wednesday, June 1, 2011

Stung by HornetQ

I've recently been upgrading our JBoss 5.1.0.GA and JBoss Messaging based application to JBoss 6.0.0.Final and HornetQ. The end result has been positive: HornetQ is very fast. But there were a few gotchas I thought I'd share:

1. No More Database, Lots More Backups

HornetQ eschews the JBoss Messaging approach of storing messages in a shared database. Instead it uses the file system directly (and, on Linux, accelerated AIO). But note this doesn't mean a shared file system. It means each cluster node has its own file system, with peer-to-peer communication to distribute messages between them.

The advantage of this approach is you no longer have a Single Point of Failure (ie. the database). But it's quite a shift in mindset if you're used to thinking of your database as a heavy, fault-tolerant kind-of-thing and your nodes as lightweight, throwaway kind-of-things. Because now, each node stores a portion of your messages. So if you lose a node you lose some messages.

The recommended solution to this is to have backup nodes. Whereas before you might have one big backup instance for your database, and no backups for your nodes, now you need a backup for each node. See this forum thread. Confusingly, when talking about backup nodes you may choose to use a shared file. But this is only for sharing between the 'backup' and 'live' instance of a single node.

2. hornetq-jms.xml Is A Red Herring

Out of the box, JBoss 6.0.0.Final includes a server/all/deploy/hornetq/hornetq-jms.xml that looks like this:

<configuration xmlns="urn:hornetq">

   <connection-factory name="NettyConnectionFactory">
      <connectors>
         <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
         <entry name="/ConnectionFactory"/>
         <entry name="/XAConnectionFactory"/>
      </entries>
   </connection-factory>
   
   <connection-factory name="NettyThroughputConnectionFactory">
      <connectors>
         <connector-ref connector-name="netty-throughput"/>
      </connectors>
      <entries>
         <entry name="/ThroughputConnectionFactory"/>
         <entry name="/XAThroughputConnectionFactory"/>
      </entries>
   </connection-factory>
   
   <connection-factory name="InVMConnectionFactory">
      <connectors>
         <connector-ref connector-name="in-vm"/>
      </connectors>
      <entries>
         <entry name="java:/ConnectionFactory"/>
         <entry name="java:/XAConnectionFactory"/>
      </entries>
   </connection-factory>

   <queue name="DLQ">
      <entry name="/queue/DLQ"/>
   </queue>
   
   <queue name="ExpiryQueue">
      <entry name="/queue/ExpiryQueue"/>
   </queue>

</configuration>

To my mind this is very confusing, because if you're using JMS producers and MDBs all those <connection-factory> configurations aren't used! I would recommend deleting them, for 3 reasons:

  1. You see the /ConnectionFactory JNDI reference in there and think you should start using ic.lookup("/ConnectionFactory") in your code. If you do, you'll get non-transacted queue sessions, duplicated messages, lost messages, and all sorts of other weirdness. Stick with ic.lookup("java:JmsXA")

  2. If using java:JmsXA, those <connection-factory> configurations don't apply! You need to look instead in server/all/deploy/hornetq/jms-ds.xml. This is using InVMConnectorFactory implicitly, but you can explicitly configure it

  3. If using MDBs, again those <connection-factory> configurations don't apply! MDBs are configured in server/all/deploy/jms-ra.rar/META-INF/ra.xml. They use InVMConnectorFactory by default.
This last point is a real doozey. Because if you want to simulate a heavyweight store of messages with lightweight consumers who read from it (like JBoss Messaging) you're going to want to use <consumer-window-size>0</consumer-window-size>. You'll find lots of examples on the Web of putting this in hornetq-jms.xml. But consumer-window-size won't work there, because MDBs don't use hornetq-jms.xml. Instead you need this in ra.xml:

<config-property>
   <description>The consumer window size</description>
   <config-property-name>ConsumerWindowSize</config-property-name>
   <config-property-type>java.lang.Integer</config-property-type>
   <config-property-value>0</config-property-value>
</config-property>

3. hornetq-configuration.xml Is Noisy

I'm a 'break it to learn it' kind of guy. So to my mind the connectors and acceptors section of hornetq-configuration.xml is a bit noisy. Here's what you can reduce it to:

   <connectors>
      <!-- Node to node communication -->
      <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host" value="${jboss.bind.address:localhost}"/>
         <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
      </connector>
   </connectors>

   <acceptors>
      <!-- Node to node communication -->
      <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host" value="${jboss.bind.address:localhost}"/>
         <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
      </acceptor>
      <!-- jms-ds.xml produces to, and ra.xml consumes from, InVMConnectorFactory by default -->
      <acceptor name="in-vm">
         <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>
         <param key="server-id" value="0"/>
      </acceptor>
   </acceptors>


Your Mileage May Vary

Of course, these tips aren't for everyone. But if you're like me they may save you a few hours banging your head against the desk!

5 comments:

Clebert Suconic said...

Richard,


Every Message System I know is the same in regard to the resource adapter / pooled connection factories. You are not really complaining about HornetQ here, you are complaining about the integration between a message system and an application server, which I agree is confusing.. but this is not our "exclusive feature". Since that is true that JMS 2.0 has raised concerns about better integrating jms20 on JEE.

Having said that: MDBs are also not part of HornetQ. They are part of the application server, hence you have to configure the pooled connection factories outside of HornetQ. (which is what you did at jms-ds.xml)


Hope that clarifies it.

Richard said...

Clebert,

Thanks for your response! Your community support is outstanding.

My apologies for complaining. This actually seemed less confusing under JBoss Messaging because you set PrefetchSize=0 in connection-factories-service.xml, not ra.xml.

I guess my feedback would be: why are there any connection-factories declared in JBoss' hornetq-jms.xml? It seems we can remove them for JmsXA/MDB use cases? If they have to be there for other use cases, could we at least put an XML comment where to look for JmsXA (jms-ds.xml) and MDBs (ra.xml)?

Clebert Suconic said...

hornetq connection factories is for external use, or if you want more performance and requiring more performance and not having to be bound to XA usages and MDBs.


I honestly think the jms specification lacks a better integragion with EE, and I'm taking actions to make it smoother.

One thing we are doing with AS7 is making a distinction between connection factories and pooled connection factories (through RA). So, this will be clear on the upcoming AS7 release.


Having said that, AS7 integration looks good so far with HornetQ.

BTW: no need to apologize for complaining. It makes sense. I was just mentioning the distinction and the integration with AS fact. Which we are improving by: Making AS7 better and by the whole community where jms20 is getting up to speed now.

Patrick Refondini said...

Excellent post ! It briefly summarise many of the questions that I faced while using JBOSS AS 5.1.0 with HornetQ 2.0.0.
It was especially noticeable when trying to start configuring connections to remote HornetQ server.

I am looking forward to testing JBOSS AS 6 with HornetQ as default JMS provider.

Anonymous said...

Thank you. That helped me with 2.2.5 and 4.2.3 integration.