HtmlUnit vs HttpUnit

Today I bid a fond farewell to HttpUnit.

I've been using HttpUnit for my black-box testing for about 3 years, and I really like it. However its JavaScript support just hasn't kept up with our needs, and it seems HtmlUnit has a much more active community around it.

I converted about 6,000 lines of test scripts in about 3 days. I thought the HtmlUnit folks (if not the HttpUnit folks) might be interested in what I experienced.

First, the FUD
HttpUnit is a great project, and very similar to HtmlUnit.

Blog entries like this (from an HtmlUnit guy) paint an inaccurate picture saying that HttpUnit is 'fairly low-level, modeling web interactions at something approaching the HTTP request and response level' whilst HtmlUnit is 'more high-level than HttpUnit’s, modeling web interaction in terms of the documents and interface elements which the user interacts with'. It then gives an HttpUnit example using requests and responses, and an HtmlUnit example using forms and input controls.

But the examples are not comparing apples to apples. HttpUnit does forms (and input controls, and tables, and JavaScript) too - and in almost exactly the same way as HtmlUnit. I'm not saying this is deliberate deception, but I think such a comparison is unfair and may even be detrimental to HtmlUnit because developers may be more reluctant to 'make the switch' if they perceive the APIs are very different.

In fact, the API methods are almost 1-to-1 identical. In coverting about 6,000 lines of code here's what I found:

• With HttpUnit: 6,198 lines
• With HtmlUnit: 6,285 lines

That's less than a 1% difference - not really a difference of 'high level' versus 'low level'.

Next, the Good
The HtmlUnit API definitely feels nicer.

• There's some neat 'public <I> I getInputByName' code that saves a lot of casting - I've stolen this idea for the next release of Metawidget
• I love the asText and asXml methods which do a lot of parsing for you
• HttpUnit used to silently submit forms using 'null' if the button you asked it to submit didn't exist (eg. form.submit( form.getSubmitButton( 'not-there' ))). HtmlUnit doesn't do this
• You can set file upload boxes just like regular text boxes. HttpUnit required you to wade through some form.getRequest goo
• I love that things like HtmlTextInput are implemented as direct extensions of the internal DOM, rather than some parallel heirarchy

Finally, the Bad
There are some things from the HttpUnit API I missed:

• Locating anchors in HtmlUnit is very fiddly. You have page.getAnchorByName and page.getAnchorByHref. But this is black box testing - it's meant to test the 'user experience'. And the user never gets to see either an anchor's name or its href. HttpUnit had a response.getLinkWith method that located an anchor by its 'innerHTML' or 'what the user actually sees'. This would be very helpful?
• Choosing options from select boxes in HtmlUnit is similarly fiddly. You have select.setSelectedAttribute(optionValue) but again this is keying off something the user never sees. I'd really like a select.setSelectedAttribute(innerHTML) so I can simulate choosing what the user chooses?
• HtmlUnit warns that use of the script type 'text/javascript' is obsolete, and maybe it is (as of about 2007). But that's a pretty recent change. If you try <script/> in the W3C validator it still suggests using 'text/javascript', and older browsers will want to see it. So this seems a very noisy warning to have on by default?
• There doesn't seem a good equivalent to HttpUnit's form.getParameterNames?

Overall, though, there's really no bad: HtmlUnit is the better product. Certainly its JavaScript support seems much better. I guess the only bad is that interest around HttpUnit has waned and it's therefore no longer a close competitor. Competiton is good, incentive to improve is good, and could have only made both products better.

Many thanks to both the HttpUnit and the HtmlUnit teams for all their hard work and contributions to the community!

Declarative UI: Metawidget v0.7

Version 0.7 of Metawidget, the declarative UI, is now available. This release includes:

• Pluggable action bindings for GWT and Swing


• Pluggable Swing validation (including JGoodies Validator)


• MigLayout support


• Scala support


• OSGi support


• Upgraded support for Android (1.0 R2) and Seam (2.1.1.GA)
  
• Fluent API for configuring Inspectors programmatically

Special thanks to Stefan Ackermann, Ivaylo Kovatchev and Renato Garcia for their help with this release!

As always, the best place to start is the Reference Documentation:

http://metawidget.org/doc/reference/en/pdf/metawidget.pdf

Your continued feedback is invaluable to us. Please download it and let us know what you think.

UrlEncodedQueryString now Open Sourced

I just Open Sourced my UrlEncodedQueryString implementation over at java.net.

This is a project I worked on with Sun in response to this RFE, but that ultimately never made it into the JDK. I still use it extensively in my own projects, however, and find it very useful.

A recent posting in the RFE's comments section prompted me to dust it off and give it a good home. Enjoy!

Overview: Represents a www-form-urlencoded query string

An instance of this class represents a query string encoded using the www-form-urlencoded encoding scheme, as defined by HTML 4.01 Specification: application/x-www-form-urlencoded, and HTML 4.01 Specification: Ampersands in URI attribute values. This is a common encoding scheme of the query component of a URI, though the RFC 2396 URI specification itself does not define a specific format for the query component.

This class provides static methods for creating UrlEncodedQueryString instances by parsing URI and string forms. Methods for creating, retrieving, updating and deleting the parameters on a query string, and methods for applying the query string back to an existing URI.

Encoding and decoding

UrlEncodedQueryString automatically encodes and decodes parameter names and values to and from www-form-urlencoded encoding by using java.net.URLEncoder and java.net.URLDecoder, which follow the HTML 4.01 Specification: Non-ASCII characters in URI attribute values recommendation.

Multivalued parameters

Often, parameter names are unique across the name/value pairs of a www-form-urlencoded query string. However, it is permitted for the same parameter name to appear in multiple name/value pairs, denoting that a single parameter has multiple values. This less common use case can lead to ambiguity when adding parameters - is the 'add' a 'replace' (of an existing parameter, if one with the same name already exists) or an 'append' (potentially creating a multivalued parameter, if one with the same name already exists)?

This requirement significantly shapes the UrlEncodedQueryString API. In particular there are:

• set methods for setting a parameter, potentially replacing an existing value


• append methods for adding a parameter, potentially creating a multivalued parameter


• get methods for returning a single value, even if the parameter has multiple values


• getValues methods for returning multiple values

Retrieving parameters

UrlEncodedQueryString can be used to parse and retrieve parameters from a query string by passing either a URI or a query string to its constructor:

URI uri = new URI("http://java.sun.com?forum=2");
UrlEncodedQueryString queryString = new UrlEncodedQueryString(uri);
System.out.println(queryString.get("forum"));

Modifying parameters

UrlEncodedQueryString can be used to set, append or remove parameters from a query string:

URI uri = new URI("/forum/article.jsp?id=2&para=4");
UrlEncodedQueryString queryString = new UrlEncodedQueryString(uri);
queryString.set("id", 3);
queryString.remove("para");
System.out.println(queryString);

When modifying parameters, the ordering of existing parameters is maintained. Parameters are set and removed in-place, while appended parameters are added to the end of the query string.

Applying the Query

UrlEncodedQueryString can be used to apply a modified query string back to a URI, creating a new URI:

URI uri = new URI("/forum/article.jsp?id=2");
UrlEncodedQueryString queryString = new UrlEncodedQueryString(uri);
queryString.set("id", 3);
uri = queryString.apply(uri);

When reconstructing query strings, there are two valid separator parameters defined by the W3C (ampersand "&" and semicolon ";"), with ampersand being the most common. The apply and toString methods both default to using an ampersand, with overloaded forms for using a semicolon.

Thread Safety

This implementation is not synchronized. If multiple threads access a query string concurrently, and at least one of the threads modifies the query string, it must be synchronized externally. This is typically accomplished by synchronizing on some object that naturally encapsulates the query string.

JavaDoc

For more information, see the JavaDoc.

It's the Length Of The Compile/Debug Cycle, Stupid (with apologies to Bill Clinton)

More than anything, I blog this to remind myself of this, because I seem to forget it more often than I should:

"The single most important factor in the quality of an individual's software development is the length of the compile/debug cycle"

I don't know how many times I think to myself 'I could set up a proper environment in which to develop and test this change, but it's such a little thing I'll just fudge it'. Like I use the Ant build instead of the IDE build. Or I redeploy the EAR instead of setting up hot class replacement.

And every single time the 'little thing' requires far more compile/debug cycles than I expected, so the job either:

• takes way longer than it should
• gets done sloppily because you can't stomach umpteen more compile/debug cycles to tweak every last little pixel alignment

And every single time I wish I'd spent that little bit of extra time, upfront, to set up my environment efficiently.

This principle extends to the build/upload cycle too. If I can reduce the amount of stuff I have to re-upload to the server when I make a change, I can reduce the amount of time I have to get distracted and go surf slashdot, thereby 'context switching' my brain and slowing me down even more.

Anything that reduces that compile/debug cycle is gold. We've all known this for a long time, but I think we all often forget it again too.

Gah!

New Features in Eclipse 3.5

In an attempt to attract younger developers, the next version of Eclipse is to include Unlockable Achievements:

Beta testers are encouraged to submit their own Achievement ideas. Suggestions include:

• Anally Retentive: remove all warnings from your code. Yes, even those ones about serialVersionUID


• Saintly Patience: use BigDecimal for more than 5 minutes without wishing for operator overloading in Java


• Ant Fan: start an Ant build and watch it finish without visiting Slashdot in between

Update: this blog post was, of course, a joke. However it appears that 3 years later Microsoft has actually gone and done it!

Better Than Free: Metawidget v0.65

There was a time, years ago, when just having a $0 price tag on a piece of software would have sparked people's interest. Nowadays, the success of Open Source has meant that free is not enough.

For a new software project to be noticed, it doesn't just have to be free: it has to be well documented, well tested, have plenty of working examples, an attractive Web site and high quality code. And even then the competition is fierce: developers are busy people, and have very limited time to evaluate new technologies.

So the big feature in the new release of Metawidget, aside from a number of upgrades and enhancements (such as Commons Validator support), is a Live Demo that lets you immediately get in and start playing and coding with Metawidget. I'm hoping it's a great way to let people try it without having to download, unzip and read through the usual distribution.

The Live Demo uses an enhanced version of the Groovy 1.6 Console Applet to give you a fully working Java-like scripting environment. Everything is set up for you just to click Run. You can then fiddle with the code, even import your own business model classes on to the CLASSPATH to see how Metawidget renders them.

You can find the Live Demo here:

http://metawidget.org/live-demo

Also, special thanks to Gerardo Diazcorujo and Ryan Cornia for their help testing this release. Your continued feedback is invaluable to us. Please let us know what you think.

A Unified Theory

I just had an interesting exchange with the guys from the OpenXava project. We discussed differences in our two approaches, as well as those of other UI generation projects, and what it would take to unify them all under a JSR one day. I think we're a long way from that day, primarily because UI generation isn't particularly 'mainstream' yet (at least, not in the sense of ORM). Still, it's often said in physics that even though we don't know what the Unified Theory is, we know something about what features it must have.

Can we say something similar about UI generation? I'll list here all those features I think are being explored, either by Metawidget, OpenXava or one of the other projects, and see if we can update this page over the years to form consensus.

Static or Runtime

Should the generation happen statically or at runtime? If runtime, how do you allow customisation? If statically, how do you allow re-running the generation when the domain model changes (without losing any customisations)?

Static: n/a
Runtime: Metawidget, Naked Objects, Woko

Modeling Language

Should the generator have its own modeling language, which developers use to describe the UI, or should it try and derive the UI automatically? Do modeling languages introduce error-prone duplication? Is automatic derivation too inflexible? Is there enough metadata to drive automatic derivation, or do we have to 'guess and fill in the gaps'?

Yes: n/a
No: Metawidget, Naked Objects, Woko

Production or Prototype

Should we expect UI generation to be able to be used in production applications, or only during a prototyping phase?

Prototype: n/a
Production: Metawidget, Naked Objects, Woko

Customisation

What sort of customisations of the generated UI are important? Graphics? Layouts? How should we facilitate them?

Pluggable: Metawidget, Woko
Search-based: Naked Objects

Bounds of Generation

Should we try to automatically generate the whole UI, or just pieces of it? Is generating the whole UI flexible enough? Is just generating pieces useful enough?

Whole UI: Naked Objects, Woko
Just pieces: Metawidget

Multiple Platforms

Is supporting multiple platforms (eg. desktop, web, mobile) important?

Yes: Naked Objects, Metawidget
No: Woko

Consistency

Given the same domain model, should we try and produce a consistent UI across all platforms? Does this risk a 'lowest common denominator'? Does tailoring uniquely to each platform introduce too much work for the developer?

Yes: n/a
No: Metawidget, Naked Objects

Diverse Architectures

Should the generator care about diverse architectures? Is mandating the technology stack of the application too restrictive? Or should we try to enforce 'good coding' that way? Does supporting multiple versions of everything introduce too much complexity? Is the ability to retrofit existing applications an important goal?

Yes: Metawidget
No: Naked Objects, Woko

Third Party Components

Should we support third-party UI components? What if they are not available on all platforms (eg. desktop, web)?

Support third-party: Metawidget
No explicit support: Naked Objects, Woko

Non-Domain Model Objects

Should we support modelling objects that are not strictly part of the domain? 'Solution space' objects, such as search screens that are user-centric (even role-centric) and not persisted to long-term storage?

Support non-domain model objects: Metawidget
No explicit support: Naked Objects

---

Convergence

Following conversations (some below, some in newer blog entries) with various parties, we note some convergence with future releases of products. Specifically:

Naked Objects: 4+ supports deriving metadata from different sources. Newer viewers may allow generating just pieces of the UI, and also modelling non-domain objects. Future versions want to support third-party components.

Woko: 2+ will allow more diverse architectures.