Ricky's technical blog

Refactor: Lazy Initialisation to Eager

2010-05-15T19:52:00.005+01:00

A field is being initialised when used, rather than when the object is created.

Extract the initialisation into a separate method that returns the value to store in the field, move the assignment to the field declaration, and inline any private use of the getter.

becomes

Motivation

The lazy initialisation is no longer necessary, and harms readability. Additionally, incorrect lazy initialisation can cause problems for concurrent programs.

Mechanics

Where the field is assigned, redeclare a variable shadowing the original field. Rename that variable including its uses, and at the end of the block assign the new variable to the field:

Extract a method for the operations on the new variable. This should be static, at least during the refactor, to prevent accidental operations on other fields. Other fields' values should be passed in explicitly as parameters. This helps to prevent initialisation-order problems (where y gets initialised before x, but while initialising y, we read x and get a big fat null).

Get rid of the == null check, and make the field final:

Using types to help refactor

2010-04-28T13:48:00.006+01:00

In some real code, I changed an int that was used to index into an array of ViewerPanels, so that the ViewerPanel was passed around instead of the int. I missed some cases in the first pass, so I had a think about how to make sure I don't do that again.

The cases I missed were PropertyChangeListeners. If you're not familiar with that type, you can use it with a PropertyChangeSupport to listen for changes in values. It has Object in its API, rather than using generics. I don't particularly like the idea of listening for changes, particularly if the listeners then end up interfering with the objects they're listening to, but that's what I have in existing code, so I need to get on with it for now.

The tests didn't pick it up, I found it by coming across a firePropertyChange and wondering what the listening end now looked like. The tests could be better, sure.

Here follows four versions of a simpler code sample. The refactor we want to do in this case is just to go from double to float. Version 2 shows what happens when one forgets to check the PropertyChangeListeners. Version 3 steps back to Version 1 but alters it to use a type-safe version of PropertyChangeListener. Version 4 does the refactor again, but this time the compiler forces the ChangeListener to be altered, because we're no longer based on PropertyChangeListener.

The below code is executable.

JNI + applets = pain!

2010-01-18T21:35:00.002Z

I mostly work on GUI software for controlling and viewing video from security cameras. The software is written in Java, and mostly written by people who are no longer with the company. The usual app I work on is a Swing application, but there is also an applet, which displays just a viewer; the controls for it are HTML buttons around the applet.

We use MPEG-4 (and are adding H.264) extensively, and Java's support for that is, well, non-existent, so we use ffmpeg, via JNI bindings generated by SWIG.

If I was starting again, and for some reason still thought using Java was a good idea, I'd try xuggler out. Xuggler appears to provide the kind of wrapping library around ffmpeg that I would hope to evolve our code towards. But even Xuggler wouldn't fix these recent issues.

The first problem, which was mostly solved before I got involved, is that you cannot load a native library directly from your JAR file. You need to extract it (typically into java.io.tmpdir), and then load it. Not too hard, but it does start to feel icky especially when you consider that an abnormal termination of the browser will cause the file to be left taking up space until manually deleted.

The second problem is the strange interaction between classloaders and JNI - even within trusted code, two classloaders in the same JVM cannot each load the same library. I'm sure somebody somewhere thinks this is a feature rather than a bug. Our code does nothing fancy at all with classloaders; it just gets affected by the fact that two applets executed by the same browser instance will share a JVM. This is still true after Java 6 Update 10, in which the JVM that applets run in is no longer the same process as the browser. If the codebase that you specify to the applet tag is different between two instantiations of the applet, then each instantiation occurs in a different classloader, and the JVM prevents the second instantiation from loading the same library (from the same filename).

This wasn't too bad, and we changed our web pages so that the codebase was the same in each instantiation. As it happened, we weren't loading anything other than the applet jar itself, so that was quite okay.

The applets and web pages are hosted on our image servers, which are units that can deliver data from and to multiple cameras. So then when one browses from an applet hosted on one image server to an applet hosted on another, despite the codebase in the applet tag being the same, two classloaders are used, and the JVM prevents the second applet from loading its native library.

The obvious solution to this is to, when writing the native library to disk, choose a unique name. But that's where the third problem comes in..

Most of our users run Windows, and on Windows if you try to delete a .dll that is in use the OS prevents that. This time, however, the relevant users are all using Solaris, and Solaris allows the .so to be deleted even while it's in use. Our library-extraction code would try to delete libffmpeg-1.so, and if it succeeded (which on Windows would be because the library was not in use), it would then use that as the filename to give the library extracted from the jar file, and then load it. As you can probably see, this works well on Windows, and like a wooden saw elsewhere, because the JVM then believes two classloaders are trying to load the same library.

The fourth problem is that the applet is self-contained; we have a separate build for Windows, Linux, OS X and Solaris, so JavaScript in the web pages has to detect the OS. Detecting the OS is ok, but we support Solaris Intel and Solaris Sparc, and there's no way of detecting those (please correct me!) from JavaScript. We have a mechanism for dealing with this, but I'm not happy with it.

I do plan on making the applet no longer self-contained, so that we distribute one platform-independent section, which detects the OS it's on and loads the correct native library, but really, this is just a case where Java doesn't come with batteries included. I realise applets are not even on the backburner anymore; they've been behind the stove getting mouldy for years, but this does seem like something that should be a lot more straightforward.

An approach that looks promising is that of applet-launcher, which wraps your applet up in another applet that handles all the library loading. I couldn't try this easily as I only control the Java code, not our web pages - it would have required the addition of lots of .getSubApplet() calls in the JavaScript code.

Deleting code, what first?

2009-12-08T23:29:00.003Z

I have about 200kloc of Java code to work with, and I often stumble across bits that are more complex than they need to be, costing me time when I'm trying to solve something else.

Within reason, having a smaller body of code is an improvement. Here are my guidelines, mainly for myself, to shrinking it.

1. Use IDEA's analysis tools to find unused code, and delete it. If it's public-facing API, and you know of no code that uses it, it's probably broken anyway. Deprecate it now and delete it at some later date. If you deleted it and somebody was using it, they'll complain at you, you'll reinstate it and include at least a comment that team X at company Y is using it, and at best an automated test that the code does what team X requires.

Watch out for main methods when you delete code - there are a few ad-hoc utilities in our codebase.

Always ensure that deprecated code is only called by deprecated code, i.e., that deprecation forms a clique.

2. Look at 'throws' clauses. If one low-level method has, say, 'throws IOException' in its signature, and all callers either pass it on to their caller or catch and do something dumb with it like logging, then you may as well do that within the low-level method, or rethrow it as a RuntimeException. This can remove a surprising amount of bullshit code.

3. Look for non-final static variables, a reasonable indicator of mutable singletons. Try to stamp them out. While this might actually increase the amount of code you have, it will do wonders for the maintainability. I did this today and found a potential bug (which I've yet to prove). The worst case is that you find out why you can't stamp a particular one out - you'll have learned something useful about the application's architecture and flaws along the way.

4. Use IDEA's "field could be local variable" inspection to simplify classes. As a general rule, a class with fewer fields is simpler.

5. Private getters and setters that do nothing beyond this.foo = foo; and return foo; are useless; inline them.

6. Search for repeated code; the IDEA Ultimate Edition can do this, but the Community Edition cannot. Unfortunately, it can only find them, not suggest a solution, and in my experience it doesn't get the ordering right; we had 15 almost-identical classes that it didn't even notice. It's still very useful though.

Even if you can't think of a decent abstraction, get rid of the repetition anyway. If needed, a better name or better way of doing it will be easy to use later. It's far easier to refactor 1 copy than 10 copies of the same code.

After a discussion at work, we decided that duplicated code has a cost proportional to its age multiplied by its size multiplied by the number of repetitions. That seems a reasonable rule; the cost is paid when using it or removing it.

7. Try to replace setters with constructor parameters, until it gets unwieldy or simply impossible as the objects get mutated after initialisation. When it gets unwieldy because the parameter list gets too long, consider splitting the class up.

8. Immutables remove the need for defensive copying. Lots of awful code is dedicated to copying values so that they can be passed to third parties. If you can make those values immutable this gets a lot easier.

9. Immutables remove the need for synchronization. Lock-free programming is far simpler if you can get away with it. Most programmers, myself included, get locks wrong, and the problems grow exponentially. That said, I don't have a general answer for what to do instead; I'm still at the stage where I need to work it out per-case.

10. Delete commented-out code. It's revision-controlled; if anyone actually ever remembers about it they can easily restore it. It's usually traces of previous experiments, and isn't of any real use. Similarly, delete the IDE-generated crap, like "Created on Mar 13 2005". You might want to keep the @author info, unless it's obviously useless, like @author Administrator, but even that can be of limited value years later when the author has left the company.

11. This isn't really about code, but: commit little and often. It's easy to do 3 refactors in one go and screw the third up, then have to revert and repeat the first 2. It's easier if you committed the first two. Similarly, if you find that a commit caused a regression, it's easier to work out what change caused it if each commit is small. That's not to say you should commit known broken code, at least not to anything other than an experimental branch.

In short: commits are game save points, not presentations.

My co-worker wouldn't get it

2009-11-30T20:38:00.002Z

I'm continually unimpressed by a certain form of argument that I've seen in many places, but especially in the continuing discussions about closures in Java.

It goes like this: Feature X is great, and will make lots of code simpler, but there will be a lot of worse code because of it. Sometimes co-workers are mentioned, sometimes contractors who have to maintain the code without having months to learn its innards.

I think this boils down to two issues:

1. *I* would write bad code with Feature X.

2. I'm worried that I'll have to deal with bad code that uses Feature X.

Any healthy programming environment will not freeze code that is known to be bad in some way. It can be fixed, and lessons can be learned. If you seriously worry about your co-workers, watch their commits. If you worry about yourself, write some non-critical code that uses the feature a lot; perhaps a prototype, some unit tests or something unrelated to work.

With some exceptions, most programmers will write bad code before they learn how to write good code. Let them do so, and help them to get past it.

My co-workers would get it.

100 Bugs (ok, tickets) in 100 Days

2009-10-27T16:12:00.005Z

I've now closed 100 tickets in the last 100 calendar days at work, which might not mean a lot, but it's been a personal target of mine to get to this point, so I'll celebrate it by, um, blogging!

Which language shall we learn?

2009-09-26T09:46:00.006+01:00

Hi Jose,

We decided, while drinking an overly-priced red wine the other night, that I'd help you to learn how to program, but without making it sound complicated. So, I thought I'd show you a few different language syntaxes and let you choose.

You can do anything you want with any of these languages, so feel free to pick the prettiest or whichever makes you laugh, or use whatever other criteria you feel like.

I chose the top-10 current mainstream languages, with the exception of PHP, which is too focused on one kind of task, and added Scala, Haskell and Erlang, because I like them.


1.
import java.util.Scanner;

public class Main {
  public static void main(String[] args) {
    Scanner scanner = new Scanner(System.in);
    System.out.println("Please enter your name.");  
    String line = scanner.nextLine();
    System.out.println("Nice to meet you, " + line + ".");
  }
}

2.
#include <stdio.h>

int main(int argc, char **argv) {
  char name[256];

  printf("Please enter your name.\n");
  scanf("%s", name);
  printf("Nice to meet you, %s.\n", name);
  return 0;
}

3.
#include <iostream>
using namespace std;

int main() {
  string name;
  cout << "Please enter your name.\n");
  getline(cin, name);
  cout << "Nice to meet you, " << name << ".\n";
  return 0;
}

4.
Module example
  Public Sub Main()
    Dim line As String
    Console.WriteLine("Please enter your name.")
    line = Console.ReadLine()
    Console.WriteLine("Nice to meet you, " + line + ".")
  End Sub
End Module

5.
#!/usr/bin/perl -w
use strict;
print "Please enter your name.\n";
my $name = <>;
chomp $name;
print "Nice to meet you, $name.\n";

6.
using System;
class Hello {
  static void Main() {
    Console.WriteLine("Please enter your name.\n");
    var name = Console.ReadLine();
    Console.WriteLine("Nice to meet you, " + name + ".\n");
  }
}

7.
name = raw_input("Please enter your name: ")
print "Nice to meet you, ", name, "."

8.
importPackage(java.util);
scanner = new Scanner(System['in']);
System.out.println("Please enter your name.");
name = scanner.nextLine();
System.out.println("Nice to meet you, " + line + ".");

9.
puts "Please enter your name."
name = gets.chomp
puts "Nice to meet you, " + name

10.
main = do
  putStrLn "Please enter your name."
  name <- getLine
  putStrLn ("Nice to meet you, " ++ name ++ ".")

11.
val name = Console.readLine("Please enter your name.")
println("Nice to meet you, " + name + ".")

12.
Name = get_line("Please enter your name.").
put_chars("Nice to meet you, " ++ Name).

Making Methods Testable

2009-09-26T08:05:00.001+01:00

I'm going through my old drafts and deleting or publishing them. This one I no longer agree with, I'd rather rewrite the methods to be side-effect free. And the 'magic' I referred to is probably mocks.

"OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things." -- Alan Kay.

Messaging seems to imply behaviour. From wikipedia: "in object-oriented programming languages such as Smalltalk or Java, a message is sent to an object, specifying a request for action."

It seems fairly trivial that, in, say, Java, Math.cos(double) is not object-oriented. It doesn't take any objects, it doesn't return any objects. Even if double was an object type, it wouldn't be object-oriented. One reason is that it isn't late-bound; the other is that it isn't a message that demands action, it's just a function call.

It's certainly possible to imagine an action that has no effects other than returning a value; I don't dispute that; but it does seem overblown to say "sending the message 'cos' to the Math object", instead of saying that Math has a function called cos, and we're calling that. In fact, function is a much better fit for this, because Math.cos is as close as we can get to a mathematical function. Whenever I say function from now on I'll mean a mathematical function.

Java actually has two implementations of all the maths functions, e.g., Math.cos and StrictMath.cos. We could make it so that which one we used was dynamically configurable (again, I'll cover late binding later), but that doesn't make it any more of an action. It's still a function. I assert that most of what we consider an object's behaviour can be validly and usefully thought of as functions over that object's data. In a simple way, you can see this is true; if an object has state X, and you call method x() on it, that changes the object to be in state Y, and you later return the object to state X and call the method again, it will do the same thing. The big difference between methods and functions seems to be that you can't see what side effects a method causes, without knowing the code.

Methods that have real effects on the world, or change objects, are really actions. Testing these is harder than testing functions. They're also harder to reason about and actually unnecessary.

I can think of two ways of testing them, the standard one of which is to simulate the environment and test for evidence of the action. This has the problem that any unwanted action isn't easily detected, except in the most carefully controlled environments. For example, you might set up a chroot environment for automatically testing a program that manipulates files, and not notice that it erases /etc/passwd (because you never use the chrooted filesystem again), or you might test what a method sends to System.out but neglect to also check System.err.

Another way of testing actions would be to first look at the code differently. Code doesn't do anything. It describes things. A program that writes data to a file doesn't really; it requests that data be written to a file. Intercept all interactions with the outside world. If someone wants to write to System.err, they do it through your testing code. The implementation is a little difficult to conceive; perhaps in Java it has to be done through aspects, or DI, if it is to be reasonably brief.

Once you have all the effects that an action produces, you can see if they match your expectations. There is a complication though; you could end up simulating too much of your environment. If a method asks how big a file is, you need to provide an answer. Ideally you would give a different value for each test, e.g., some exception saying the file is inaccessible, lengths 0, 1, and any other lengths that seem relevant to the method.

What we've actually done is to cocoon the method; we've now made it into a function! The perfect test suite for a method makes the method into a function. Well, that's all well and good for I/O, etc., but how about when we're testing a mutable object - it might have changed more than is usual to test for. E.g.:

public void broken()
{
    setX(5);  // we're happy with this line.
    setY(10); // but this one seems wrong.  We want the test to fail while this line is present.
}

.Let's magically jump in the way and grab a list of changes to the object that this method causes. We know x has been changed and y has been changed, so the test fails. In fact, for this case we never need to execute the actions, because we can see that they fail without executing them. For more interesting cases, we can again simulate the effect. Besides my using this technique to demonstrate a point, it could be used to test real running objects without changing them.

So again, the ideal test suite for a method makes it look like a function. If we adapt our code to explicitly state what effects it should cause, instead of going off and causing them, we then actually have functions in our code. Here's the above broken(), but fixed..

public List<Action> fixed()
{
    return asList(new AssignAction("x",5),new AssignAction("y",10));
}

The code is significantly uglier and doesn't use static typing, but it's now actually a function. Because I've adapted it to return stuff instead of do stuff, it is now far easier to test. Or perhaps all I've done is to make message passing even more explicit. Well, message passing involves actions, and to some extent we can invoke the above actions and observe their results without changing anything (recurse through the list constructing a chain of values, rather like SICP's discussion of chained environments).

It's now clearly far easier to reason about the method, because you can trivially observe all of its side-effects. You could even decide which ones to allow, externally to the code. In the usual case that you want to execute the effects immediately, you can still do that.

Is it actually type inference we need, or just a better syntax for type declarations?

2009-09-19T23:36:00.004+01:00

I've looked a little at Typed Scheme recently, because the main problem I have with Lisp is that it's untyped, but I lack the imagination and skill to make a typed version.

Typed Scheme has local type inference, like C#'s var, but other than that, if you want it typed, you write it typed. The types go before the declarations, like in Haskell, which provoked a surprising thought, namely the title of this post.

In Scala, Java and C#, type inference always seems desirable. Let's look at how you'd declare a method that takes a list, an int and returns an element of that list in each language:

Scala: def method[A](list: List[A], num: Int): A // the : Float would normally be inferred if left out.

Java: A <A> method(List<A> list, int num)

C#: A method<A>(List<A> list, int num)

Those aren't amazingly bad, but compare them to Python:

def method(list, num)

or Scheme:

(define (method the-list num))

The untyped versions are undoubtedly prettier, though far less informative. And compare now to Haskell:

method list num = ...

Haskell's version uses type inference; the types of list and num will be inferred by their usage in the function. But lots of Haskell programmers will write the type explicitly for all functions. I think they do this because in Haskell the syntax for explicit types is better than in Scala, C# and Java:

method :: [a] -> Int -> a
method list num = ...

You can read that first line as "method takes a list of some type we'll call a, an Int and returns an a."

The specific way in which the Haskell (and Typed Scheme) mechanisms are better is that the type information isn't mixed up in the syntax for naming parameters (and after Scala and C# add optional parameters, that too). It'll never happen for any, but I'd like to see the following syntaxes become possible:

Scala:
method[A] : (List[A], Int) => A
def method(list, num) ..

Java/C#:
A method<A> (List<A>, int)
method(list, num) { .. }

The programming language I'll one day write currently looks a bit like Typed Scheme. Like Typed Scheme, it has local variable type inference, but not function definition type inference. I realise that supporting one but not the other is a bit inconsistent, so I know I have more thinking to do, but Typed Scheme (and Haskell) made me realise not to imitate Scala, C# or Java's type declaration syntax.

Victor James Clarkson De Luca, Version 1.0

2009-01-16T19:55:00.006Z

After 9 months of hard work and alcohol avoidance, Josefina and I are proud to announce that we have managed a stable release of Victor James Clarkson De Luca. Version 1.0 was released on January 5th 2009, before you got to work.

There are no known bugs, as yet. Here's a screenshot:

The Typeclass Pattern

2009-01-02T11:06:00.003Z

The Typeclass Pattern

This pattern applies to C# and Java, and possibly some other typed languages. It doesn't apply to C++, which does generics via templates.

A simple problem to demonstrate it with is a generic Pythagoras method that takes two numbers a and b and computes sqrt(a * a + b * b) for them. In both C# and Java this can't be written in the most straightforward way:

public T Pythagoras<T>(T a, T b)
{
    return Math.sqrt(a * a + b * b);
}

There's more than one reason, but the 'innermost' is that T doesn't have the operator *, and there's no way to restrict T so that it does.

The equivalent C++ will work, because C++ generates code rather than using constraints.

So the missing information in Pythagoras is:

* How to multiply two Ts. (let's assume T * T gives T)

* How to add two Ts. (let's assume T + T gives T)

* How to find the square root of a T.


interface Num<T>
{
    T Add(T one, T two);
    T Multiply(T one, T two);
    T Sqrt(T t);
}

Now Pythagoras can be written with an extra parameter, a Num:

public T Pythagoras<T>(T a, T b, Num<T> num)
{
    return num.Sqrt(num.Add(num.Multiply(a, a), num.Multiply(b, b)));
}

Clearly, some language support wouldn't go amiss for this, at least in the arithmetic case. What's more of a problem is that your method gets a 'surprising' extra parameter, the Num. It's one of those things that makes sense in isolation, but is really an unnecessary detail when reading the code to gain an understanding of it. Scala offers an interesting solution to that.

In Scala, you might write Pythagoras as:

def pythagoras[T](a: T, b: T)(implicit num: Num[T]) = num.sqrt(num.add(num.multiply(a, a), num.multiply(b, b)))

Then calling it would look like:

pythagoras(3, 5)

The second parameter list for pythagoras is an 'implicit' parameter list, meaning that in compilation the compiler looks for a Num[T] in scope that it can use for num. You can specify it explicitly, e.g., pythagoras(3, 5)(Num.integer), or you can just import Num.integer at some point.

As with all patterns, though, all it takes is a language to support it directly, and then the pattern disappears from user code. In Haskell, Num is a built in type class and Int, Double, etc. are types that are instances, or members, of Num. +, -, *, / are defined as methods that are part of the Num typeclass. Let's omit the sqrt part for a moment:

Prelude> let pythagorasSquared a b = a * a + b * b
Prelude> pythagorasSquared 3 4
25
Prelude> :type pythagorasSquared
pythagorasSquared :: (Num t) => t -> t -> t

I hope the first two commands I gave to the Haskell interpreter are self-explanatory. The third asks what type pythagorasSquared is. That's like asking what signature a method has.

pythagorasSquared is a function with one type parameter called t. t is an instance, or member, of the Num type class. pythagorasSquared takes two values of this type t, and returns a value of the same type. E.g., given two Doubles it will yield a Double. This usually works internally by passing around a Num implicitly, like in the Scala solution, but can work quite like the C++ code generation way, depending entirely on the compiler.

The reason that I omitted the sqrt is that in Haskell, sqrt is defined on Floating, another type class, rather than Num (giving the sqrt of an Int as an Int is perhaps not useful).

At some point, some readers will have stopped understanding the terms, but if you grasped the concept, this wasn't a waste of time!

Java Just Died (no closures in Java 7)

2008-12-11T12:05:00.002Z

So it's confirmed. Despite James Gosling wanting closures, despite 3 working closure prototype compilers, despite every other JVM language supporting closures, Java 7 will not have closures.

while (true)
    print("new Runnable() { public void run() { ")

Frankly, I'm pissed off. I'm sure Java dying will kill the JVM too, ultimately.

In the meantime, there's Scala. But the Java community will probably prefer Groovy, because it has learned from Java that types are hard.

(my source: http://twitter.com/theplanetarium via Ismael Juma)

Binary Reading and Writing Combinators for Java

2008-11-23T13:20:00.002Z

Binary4J is a combinator library for reading and writing arbitrary file and stream formats. Comment on it here.

My Scala Coding Style

2008-09-19T10:08:00.003+01:00

This was originally a posting to the scala-tools mailing list, in response to someone asking how to make the Eclipse plugin format using braces on the next line and spaces instead of tabs.

I used the braces-on-next-line rule, lining up the { to the previous line rule for years, and tabs and no spaces, for years. I was happy with it. I used an 80-character line length limit, which made coding in a text terminal easy, and tabs equivalent to 8 characters (though the actual tab width didn't really matter as I only used indentation to signify nesting depth, not to line up individual characters). My code was very structured.

If you look at my blog's very early days, you will find a post somewhere saying to avoid anonymous classes in Java. This is exactly the opposite of what I'd say now (now I'd say avoid Java :) ). It dawned on me after a while that the reason I didn't like anonymous classes was that my coding style made them a real pain in the backside to use. Specifically, if you have one anonymous class inside another with the above coding style, you end up splitting most lines of actual code.

Then I learned Lisp.

I realised that my overly structured Java code penalised me for nesting. So nesting must be bad, or the coding style must be bad.

Most human languages are highly recursive - within one sentence you can set up phrases, talk about what would happen in the future, and discuss two possible futures or even possible pasts. Only one language is known of that isn't highly recursive - the Piraha language. In Piraha the culture discourages talking about the future, or any event that you haven't seen personally, or been told about by another. A researcher who lived with them and learned their language tried telling them about Jesus (yes, there are God-botherers in linguistics too). One Piraha asked what Jesus looked like - and when the researcher said he didn't know, as Jesus lived 2,000 years ago, the Piraha wasn't interested anymore. This actually caused the researcher to start questioning his own religion, but that's a little beside the point.

I imagine that if you asked a Piraha to design a computer programming language (or a suitable analogy to one - perhaps they wouldn't be interested in computers), I imagine it would be limited to a nesting level of 2, by virtue of a 25-space tab and an 80-character line limit. And if they invented braces, I'm sure they would be lined up.

This has parallels in our society too. The more fluent amongst us can happily deal with long sentences, discussing multiple futures. The less fluent (perhaps speakers of a foreign language learning English) will generally prefer shorter sentences, discussing only the present.

In summary, pick a coding style that doesn't punish nesting, unless you want to make the language 'unnaturally' statement-orientated. I have taken this to an extreme - I use one space for indentation, and never place opening OR closing parens on their own lines. I think this is a direct result of learning Lisp, but it took a long time after learning Lisp for me to change.

Plus, silly Scala makes putting the brace on the next line fail to parse sometimes anyway.

Implementing the Builder pattern in Java without repeating code

2008-09-10T01:37:00.002+01:00

When writing some Java wrappers around some CGI requests at work, I began with a normal implementation of the builder design pattern, but when I realised I was going to have to do this for about 50 CGI requests, and some of them were nested (CGI requests with query parameters to be sent on to a further CGI request on another machine), and that many of the parameters had interesting constraints, I realised that while the API might be fine, the implementation we were looking at, Josh Bloch's, encouraged repetition of logic.

Anyway, here's an example to get us started. The problem:

Person person = new Person("John", "Jackson", 1979, 11, 10, "AC2193");

We wanted something more like:

Person person = new Person.Builder()
    .forename("John")
    .surname("Jackson")
    .yearOfBirth(1979)
    .monthOfBirth(11)
    .dateOfBirth(10)
    .nationalInsuranceNumber("AC2193").build();

To keep the code short, we'll use a simpler class as an example, a Person consisting of name and age. As you'll see, even this can be large enough to be interesting.

public class Person
{
    private final String name;
    private final int age;

    private Person(Builder builder)
    {
        this.builder = builder;
    }

    public static class Builder
    {
        private String name;
        private int age;

        public Builder name(String name)
        {
            this.name = name;
            return this;
        }

        public Builder age(int age)
        {
            this.age = age;
            return this;
        }

        public Person build()
        {
            return new Person(this);
        }
    }

    public String getName()
    {
        return name;
    }

    public int getAge()
    {
        return age;
    }
}

Even for 2 parameters, this is quite a lot of code, though thus far there isn't really any logic to be duplciated. Now let's look at a really simple constraint, that values cannot be set twice.

The most obvious way of trying this would be to have a boolean alongside each field in the builder, e.g.:

private String name;
private boolean nameAlreadySet;

private int age;
private boolean ageAlreadySet;

And then in the name(String) and age(int) methods in the Builder you would check the value of that boolean, and throw an IllegalStateException if the boolean had already been set. This is clearly a repetition, which can lead to copy-and-paste errors or just make things hard to change.

In object-orientated programming the usual way of handling this would be to package the field with its boolean in an object and call it, say, MaxOnce. There is a good reason not to go down this path, though, it's difficult to chain MaxOnce with other such types, for example when we want BoundedInt, which prevents values outside a certain range, to work with MaxOnce. So we have a problem that the classes don't work together well. Time for another approach.

It would help if MaxOnce and BoundedInt were more like filters that data gets passed through (or not, if the data is invalid). Enter Parameter.

private final Parameter<String> name = maxOnce("name", null);

private final Parameter<Integer> age = bound(maxOnce("age", 0), 0, Integer.MAX_VALUE);

Notice how bound and maxOnce are chained together in the age parameter It's easy to see how you might write other filters. Here's a largely useless example:

Parameter<Integer> number = not(5, bound(maxOnce(0, "a number from one to ten, but not five"), 1, 10));

For a Parameter that has no default, it might be handy to store the value as an Option, rather than use null, or an empty String or some other sentinel. In another case we want to store the value as a TreeMap (for sparse arrays, mentioned later). So generally we'd like to be able to specify an input type and an output type for a Parameter.

private final Parameter<String, Option<String>> name = maxOnce("name");

private final Parameter<Integer, Option<Integer>> age = bound(maxOnce("age"), 0, Integer.MAX_VALUE);

Note that bound and maxOnce work together for the age parameter, as two filters.

In a few cases, the Parameters that we use are allowed to take multiple indexed values. They are effectively sparse arrays. The Parameter's input type is a Pair<Integer, T> and the output type is a TreeMap<Integer, T> - each incoming Pair gets added to the TreeMap.

private final Parameter<Pair<Integer, String>, TreeMap<Integer, String>> = ...;

We can see that the Parameter is more a declaration than it is an actual value. Then it's quite handy that, actually, Parameter holds no mutable state - we store that in a Map<Parameter, Object> but with slightly better types, wrapped up as a GenericBuilder, and we don't modify that Map, we copy it when we add values, like CopyOnWriteArrayList does in the Java libraries.

Here's the original Person class with a new implementation:

public class Person
{
    private final GenericBuilder finalValues;

    private static final Parameter<String, Option<String>> nameParam = param("name", "The name of the person", Conversion.<String>identity());

    private static final Parameter<Integer, Option<Integer>> ageParam = notNegative(param("age", "The age of the person", Conversion.stringToInt));

    private Person(GenericBuilder finalValues)
    {
        if (realBuider.isDefault(nameParam) || realBuilder.isDefault(ageParam))
            throw new IllegalStateException();

        this.finalValues = finalValues;
    }

    public static final class Builder
    {
        private GenericBuilder realBuilder = new GenericBuilder();

        public Builder name(String name)
        {
            realBuilder = realBuilder.with(nameParam, name);
            return this;
        }

        public Builder age(int age)
        {
            realBuilder = realBuilder.with(ageParam, age);
            return this;
        }

        public Person build()
        {
            return new Person(realBuilder);
        }
    }
      
    public String getName()
    {
        return finalValues.get(nameParam);
    }

    public int getAge()
    {
        return finalValues.get(ageParam);
    }
}

There are a couple of extra bells and whistles in the real code, such as building and parsing URLs containing the parameters. I have another post taking this one step further (using Parameters from code generation) in the pipeline.

An IRC Bot in Haskell, 20% code, 80% GRR

2008-09-07T16:55:00.002+01:00

I decided to look at Haskell more seriously, after mainly using it to learn functional programming and, well, as a posh calculator. So when I came across Don Stewart's little tutorial on writing an IRC bot, I followed it, but with one use case in mind:

In the #scala channel, often it's handy to display a URL to a bug by its index, particularly as the URLs are a little tricky to remember.

Don's code was excellent, very readable, and on my little Asus EEE, all I had to do besides install ghc6 was to install ghc6-network-dev - for some reason the Xandros packages are quite split up.

All worked fine - I tested it in a private channel, then I wanted to make it work on a remote machine, which Eugene Ciurana lets me use. Eugene is mainly a Java programmer, so he doesn't have Haskell installed at all, and as I write this, it's around 9am on a Sunday his time - I'm not going to bother him!

So I wondered about installing ghc as a user, and downloaded the 64-bit binary distribution for ghc-6.8.3. After extracting it I realised I couldn't see a bin/ directory or similar, so I checked the README file, which pointed me at the INSTALL file. So I typed ./configure --prefix=/home/ricky/ghc/, only to find out that gcc was broken on the machine.

Eugene also lets me use another machine, a 32-bit one, but on there I got this error from ./configure:

"checking for path to top of build tree... utils/pwd/pwd: /lib/i686/libc.so.6: version `GLIBC_2.3' not found (required by utils/pwd/pwd)"

dmwit and Heffalump (no, really) from #haskell suggested I try using ghc 6.8.2 instead of 6.8.3, which I am going to try shortly, but I thought of another solution. My little Asus EEE runs ghc (albeit version 6.6) well enough, so I compiled my file. After using gcc on and off for years, I was surprised that ghc 3.hs didn't do anything useful - #haskell suggested ghc --make 3.hs. This produced an executable file called 3, which I promptly ssh'd to both of Eugene's machines. On both it failed because libgmp wasn't present. So I was advised to try ghc --make 3.hs -optl-static - which seemed to almost kill my EEE. But a minute and a half later my EEE returned to its usual speed, last.fm started playing again, and I had a somewhat larger file called 3, which worked on the target machine!

So, if you wander into #scala now and type ~trac 100 it will give you a link to Scala bug report 100.

I spent more than 4 times as long on the config as I did on writing (well, mainly copying, though I copied without a clipboard) the code. Hopefully that will drop next time!

Optional Values in Java

2008-07-28T08:24:00.001+01:00

If you take some Java code and write psuedocode representing it, you'll probably find that you don't bother with null checks and you don't bother with getters and setters. Sure, in psuedocode you're lazy, but it's more than that - null is usually wrong, so much so that intentional uses of null look like sloppy code.

In fact, if you're writing an API, you probably want to keep null out of your interactions with your users - you want to make sure they realise their mistake if they give you null and you don't want to give them null, lest they forget to check it. But there are actual times when you need some way of representing an optional value.

One particularly popular approach is to use sentinel values - let's say "" for Strings, Double.NaN for doubles, -1 for ints. Now everywhere you read the value you need to check for the sentinel, or be sure that not checking for it won't cause you problems.

Another approach is to use an empty list to represent no value, and a list of 1 element otherwise. Again you need to check whether the list is empty before getting the result out.

You could make a class that might hold a value, that has methods called hasValue() and getValue(). Again, requires a check.

In all these you need to remember to check before you get the value - not much of an improvement over using null directly.

If I categorise some code including null checks (no, not nunchucks), then we'll have something to toy with:

1. foreach

if (x != null) {
 doStuffWith(x);
}

2. map

String s;
if (x == null) {
 s = null;
}
else {
 s = x.toString();
}

3. fold

int length;
if (s == null) {
 length = 0;
}
else {
 length = s.length();
}

Those were some strange names I gave to these categories! Let's tackle foreach first: Think of a value that might be null as a collection containing 0 or 1 elements - foreach would be a loop that runs 0 or 1 times to do something with the value.

map is a mapping from a domain containing null, to a co-domain containing null, - for example, mapping from rectangular coordinates to polar coordinates should probably yield null for a null input, if it doesn't throw an exception.

fold is a more manageable name for a 'catamorphism', which is a transformation that tends to yield a simpler value than the collection it's applied to (which seems the opposite of a fold in origami). In the case of a possibly-null value, the result is simpler because the result is (usually) a not null value.

Being responsible non-repetitive Java programmers, we'd like to encapsulate our possibly-null value plus the checks into an object with three methods, foreach, map and fold, rather than repeating them everywhere:

interface Optional<T> {
 void foreach(Task<T> task);
  R map(Conversion<T,R> conversion);
  R fold(R theDefault, Conversion<T,R> conversion);
}

(you might really want to make Optional Iterable so that you get Java's foreach loop, rather than providing foreach, as an implementation detail).

In the same way that java.util.Collections.sort can take a Comparator, each of these methods takes in an object that has a method that gets called if and when it needs to be.

interface Task<T> { void execute(T value); }
interface Conversion<T,R> { R convert(T value); }

Let's look at how we can convert the earlier null-using code to code using Optional.

1. foreach

x.foreach(doStuff);

2. map

String s=x.map(toString);

3. fold

int length=x.fold(0,length);

Of course, the likelihood is that you're not lucky enough to already have doStuff stored as a Task, toString stored as a Conversion and length stored as a Conversion, so perhaps you'd use an anonymous class to provide those. Unfortunately the syntax for anonymous classes bloats the code too much to be readable in a blog (or an IDE).

It would be useful to have good syntax for using foreach, map and fold in Java, so that there was at last an attractive alternative to null. For now we'll have to settle for attractive semantics rather than attractive syntax though.

I think this is beautiful because it provides a level of abstraction that gets you further from a potential source of bugs, makes your code more expressive about what it accepts, and lets you do in objects what otherwise would be repetitive.

A complete implementation of Optional is available in Functional Java under the name Option. There, Task is called E, and Conversion is called F. Option is most widely known as Maybe, from Haskell.

May your nulls rest in peace.

Designing an Object

2008-07-21T08:20:00.001+01:00

If there may exist an object with a method that is not appropriate at all points in the existence of the object, then the object or the method are flawed.

A class encapsulating a compile phase in an IDE might have a blocking or non-blocking execute() method, plus a getErrorMessages(). There is an obvious protocol in using this class - instantiate, call execute(), call getErrorMessages(). It's not particularly hard to use, though it's also not hard to get wrong. Even if you decide not to help those users who don't bother to learn the protocol, it's worth thinking about whether that protocol should even exist, and what the alternative is.

Many readers would probably, when prompted at least, make execute() return the error messages (or a Future for them), which solves the problem quite well. If you wouldn't, keep adding phases plus methods only appropriate for each phase, to one class that grows and grows, until you end up agreeing or changing career :) Anyway, in this case it's clear that the object was flawed by doing two things one after the other - executing the compile phase and delivering results.

I bet most people could train themselves to spot this flaw and remove it, and if anyone only goes that far as a result of reading this post I'll be happy. But most people are probably quite happy with another flaw, java.util.Iterator.next(), which is only allowed when hasNext() returns true, in most Iterator implementations. But moving next() or hasNext() onto another object doesn't really work for Iterator. For a long time I was unhappy with Iterator, but didn't really have a solution, despite trying a couple of things out.

The biggest use of Iterator directly in Java for many years was in what has since been replaced by a foreach loop. There are some detractors of, well, anything new, but generally the foreach loop was really well received by the Java community. It provides a higher-level interface than the Iterator gives us. We can write a lot of code using the foreach loop that would have been more verbose and awkward to get right using Iterator directly. But foreach is only one abstraction; there are some more that are higher still than it, and don't (but can) depend on Iterator. If you don't know what those abstractions are I really think you should take the time to learn about map, filter and reduce, and the more general but less usable parent of those, fold. But this isn't a post about those, so I'll return to the topic at hand.

Iterator has been shown to be flawed, though flawed in a way that is acceptable to most of us and in a way we're used to, and in a way that seems non-trivial to solve (without knowing about map, filter, reduce and fold!). You might not be in a position to, or even want to, replace Iterator, but at least you should know not to copy its design, or bind yourself unnecessarily to it. You're now either armed with a simple way of deciding between two API designs, or you're about to tell me why I'm wrong.

Happy coding.

Programming in a natural order

2008-06-26T13:34:00.004+01:00

Whenever I'm doing the same thing over and over, a little background thread in my brain starts to wonder if there's a better way. Sometimes I tell it to shut up and let me code, other times I listen. In this case it took me a long time to listen, so it must have been pretty persistent. The first time I noticed it was while writing some lisp. Here goes the usual overly trivial example:

I write (+ 3 4) then realise I want to multiply the result by 5. I have to go back to the start of the expression and change it to (* (+ 3 4) 5). I could easily blame lisp's prefix syntax and maybe define some reader macro so I can write something like (+ 3 4) >> (* 5). But that would solve one case and not really fix the overall problem. Here's the obligatory less trivial example:

I write (+ 3 x) then realise that x needs to be a lambda parameter, so I go back and write (lambda (x) (+ 3 x)). I think it's difficult to make the above >> reader macro work with this.
(+ 3 x) >> (lambda (x)) might work, but it's starting to get hard to read. It doesn't seem as natural as it did for the previous case.

But, as I said, it took me a long time to take notice of this background thread, and in fact when I did I was writing Scala, not lisp. When in lisp I just wrote the code anyway, and it didn't harm me noticably. I never wrote the above reader macro, and as I've never written one, chances are it's unwritable anyway.

The case when writing Scala was where I was looking at an expression, then realised that one of its inputs would be a collection, not a single value. A closer-to-reality example - you have a Client, and a Client has a manager, the contact there that you speak with. So you have:
def sendSpam(client: Client) = emailer.send(marketingTripe, client.manager.emailAddress). Then later you change Client.manager to Client.managers, and change its type from Manager to Iterable[Manager]. Now the emailer protests (at compile time), so you have some options:

1. Change the emailer to make it accept an Iterable[Client]. This is actually quite reasonable, and what I did in the real code this mimics.

2. Change the code to:
client.managers foreach (manager => emailer.send(marketingTripe, manager.emailAddress)).

3. Listen to the background thread in your brain, and blog about an automatic transform from the original version to option 2. Today I'm choosing option 3, as you can tell. To avoid ambiguity, I'll introduce some syntax to put around 'managers':
emailer.send(marketingTripe, ^(managers).emailAddress) will be transformed to the code from option 2 above (though perhaps without a real variable name).

At this point, you might be seriously glad that I am not a committer on the compiler for the language that you use every day. But I think the above might actually be a good idea, and might even be a better syntax for a map with a lambda than what we're used to. The reasoning is: it's preferable to write code in a natural order, rather than the order that the language forces you to.

A Cross-Language Generics Trick - Java, Scala and C#

2008-06-25T00:01:00.007+01:00

Given a Pair<T, U> type in Java, Scala or C#, such as Map.Entry, Tuple2 or KeyValuePair respectively, you can construct type-checked variadic heterogenous containers that you can write general methods to operate on.

Let's write a Pair interface for Java and C#:

interface Pair<T, U> {
 T _1();
 U _2(); }

For Scala we'll use Tuple2, which has _1 and _2 as well. You could use Map.Entry and KeyValuePair in Java and C# respectively, but they seem to have extra semantic information in their names. I know some readers will be crying out for more semantic information than _1 and _2, but I hope they bear with me a moment.

Eliding the implementation, one could have a line of code like the following easily:

Java: Pair<String, Integer> pair = Pairs.pair("hello", 5);
Scala: val pair=("hello", 5)
C#: val pair = Pairs.Pair("hello", 5);

I expect that's fine with most people. For C# you'd probably change 'pair' to 'Pair' for the method name. Then, to start introducing the trick:

Java: Pair<Double, Pair<String, Integer>> withDouble = Pairs.pair(3.0, pair);
Scala: val withDouble = (3.0, pair)
C#: var withDouble = Pairs.Pair(3.0, pair);

You can see that the type in the Java code starts to look a little messy; this is no accident. Explicit static typing makes us more likely to choose less expressive types. Anyway, we can add a method 'prepend' to the Pair type, which doesn't modify anything, but returns a new Pair consisting of a data item on the left and the original Pair on the right. So we get:

Java: Pair<Double, Pair<String, Integer>> pair = Pairs.pair("hello", 5).prepend(3.0);
Scala: val pair = Pairs.pair("hello", 5) prepend 3.0
C#: var pair = Pairs.Pair("hello", 5).Prepend(3.0);


So prepend must be an interesting method, because it looks like you can use it to add more type parameters to something.  Clearly you can't, I'm just chaining Pairs, but it makes a nice effect.  So far not very useful; I'll get to that.  First let's implement prepend:Java:
 public class Pair<T, U> { ...
  public <V> Pair<V, Pair<T, U>> prepend(V v) {
   return pair(v, this); } }
Scala:
 implicit def Tuple2WithPrepend[T, U](tuple: (T, U)) = new {
  def prepend[V](v: V) = (v, tuple) }
C#:
 public class Pair<T, U> { ...
  public Pair<V, Pair<T, U>> Prepend<V>(V v) {
   return pair(v, this); } }
The nice part about this way of building up Pairs is that you can write methods to handle them instead of writing one per Pair arity.  Specifically, you could gather up parameters for an immutable class then instantiate it in one go.  In fact, that's what I do in a prototype for a JDBC wrapper.  To wet the tastebuds (sorry, only Java for this one):List<QuestionInfo> questions=select(conn).asString("question").asString("correct").asString("wrong").as(question).from("questions").toList();
The idea is that the above runs the SQL query: select question, correct, wrong from questions and constructs a QuestionInfo for each result, putting the result into a list.

The surprising thing is probably that there's no reflection or casting going on at all.  Each asString (well, after the first one really) builds up more in a chain of generic types, then the .as(question) deconstructs them again.  question is actually an F<Pair<String, Pair<String, String>>, QuestionInfo>, which means it's a function that takes 3 Strings and returns a QuestionInfo, roughly.

The above code comes from a working test case I published here.

It turns out that someone else had this idea way way way before I did, and made something professional out of it, though only some of that appears to be statically type-checked.

I hope that what I've showed here proves useful to you, and if you are my team leader and I pointed you at this page, remember that you saw it on the Internet, it's real, so you have to let me write it in our project.



So You Like For Loops? Zip It Up!
2008-04-26T02:06:00.005+01:00
I've written this little for loop many a time, it finds all duplicates in a sorted list of integers:List<Integer> nums=Arrays.asList(1,2,2,3,4,6,6,7,8,8);

List<Integer> dups=new ArrayList<Integer>();
int prev=nums.get(0);
for (Integer i: nums.subList(1))
{
    if (prev==i)
        dups.add(i);
    prev=i;
}Now go and read your email or something, and come back to read just the for loop (and the line above it).  You have to trace through time in your head to work out what it does.  The code doesn't say what it means, but that's ok, right, because it works..  hmm.

Let's make up a type called Pair<X,Y>, and quickly imagine that List now has this nice method, zip.  Here's the above list, zipped with a sublist of it starting from 1.
System.out.println(nums.zip(nums.subList(1)));
output: ArrayList((1,2),(2,2),(2,3),(3,4),(4,6),(6,6),(6,7),(7,8),(8,8));
Can you see how that relates to the original list?  I hope so.  Given this data, how would you find duplicates?  Well, you'd look for any Pairs where the X and Y value are the same, easy.  We'll call that zipped list 'zipped', of type List<Pair<Integer, Integer>>:List<Integer> dups=new ArrayList<Integer>();
for (Pair<Integer, Integer> pair: zipped)
    if (pair.x().equals(pair.y()))
        dups.add(pair.x());
This for loop is a common pattern now, it's a 'collector'.  At each iteration we have a/the list of dups, and the current pair.  If the current pair has equal elements we modify the list of dups to make it have one more.  Really, it seems like it should be a general operation, filter:List<Integer> dups=zipped.filter(new Predicate<Pair<Integer, Integer>>()
{
    public boolean invoke(Pair<Integer, Integer> pair)
    {
        return pair.x().equals(pair.y());
    }
}).map(new Function<Pair<Integer,Integer>,Integer>()
{
    public Integer invoke(Pair<Integer, Integer> pair)
    {
        return pair.x();
    }
});
It's kind of better but the syntax is getting in the way.  Hold up your closure glasses to the screen, or if you don't have any, I've kindly repeated the code but using the proposed Java 7 closures syntax:List<Integer> dups=zipped.filter( { Pair<Integer, Integer> pair => pair.x().equals(pair.y()) } ).map( { Pair<Integer, Integer> pair => pair.x() } );
I hope that helped you read the previous code.  Let's go one step further with these glasses, they are now magically type inference glasses.  We don't need to specify the type of 'pair' because it's blindingly obvious from the type of zipped.  We don't need to specify the type of dups because it's blindingly obvious from the return type of filter:val dups=zipped.filter( { pair => pair.x().equals(pair.y()) } ).map( { pair => pair.x() } );
It looks to me like the braces in there are a bit redundant, the () after x and y are just annoying, and .equals is a Java design error that our glasses can correct:val dups=nums.zip(nums.tail).filter(pair => pair.x==pair.y).map(pair => pair.x)
Und viz zis Scala zee transfurmaschun vill be complete! (parody of a parody)

Update: Thanks to David MacIver, who spotted a mistake, I had to add 'map' to each of the non-for-loop examples, and in updating it, I've stopped short of what might be the Scala norm - in a closure such as (x => x+2), where the closure parameter is only used once, you can write (_+2).  So above you'd write blahblahblah.map(_.x) instead of blahblahblah.map(pair => pair.x).



Implementing OOP in Java
2008-03-02T15:00:00.003Z
Object-Oriented Programming is all about late binding.  Java has some support for it via polymorphism, but it is not as late as it could be.  Java makes some attempt to guarantee things statically, e.g., method calls are guaranteed to correspond to an actual method at runtime.  This has a couple of implications:

1. Some of the dynamism that languages like Smalltalk, Python, Ruby and Groovy have is lost.

2. In attempting to reach that dynamism, programmers end up doing things like: throw new UnsupportedOperationException, because their interfaces are too big, and having smaller interfaces would increase the number, and hence perceived complexity, of APIs (yes, this really is the reason java.util has 'optional methods' in interfaces).

Static typing, particularly when it's done really well, is incredibly useful.  So useful that it can replace lots of unit tests, or even make unit testing even easier to write.  It's actually possible to guarantee that code cannot have a runtime error, if you write it with that in mind.  However, some code really needs to be dynamic, at least temporarily, because it's being written by an amateur, or needs to be swapped in or out at runtime, or is an idea that might not match up with your language's typesystem just yet.  In that kind of code, guaranteeing that there are no errors is actually an anti-goal.

So, let's implement this idea of OOP, in Java.  Some languages have a symbol type that is quite appropriate for this, but Java doesn't, so we'll use strings.  To avoid annoying namespace collisions, we'll call our Object type Dyn, short for Dynamic.  I chose this short name to not distract from the meaning of Dyn-using code.  A Dyn is an object that has a method, ap, that takes a String and zero or more Dyns as parameters, and returns a Dyn.  (Some conversion methods to Java's static types might be useful too)
public abstract class Dyn
{
    public abstract Dyn ap(String name,Dyn... args);
}
And an implementation, just enough to try it out:public static final Dyn identity=new Dyn()
{
    public Dyn ap(String name,Dyn... args)
    {
        return this;
    }
};
The above Dyn is a bit silly, whatever message you send to it, it returns itself, and that's all.  Here's one that holds a Java int and when the message "sqr" is passed to it, returns a Dyn holding the square of that int:public static final Dyn squarer(final int i)
{
    return new Dyn()
    {
        public Dyn ap(String name,Dyn... args)
        {
            return name.equals("sqr") ? squarer(i*i) : identity;
        }
    };
}Here I'm using the identity object to represent an error/missing method, which seems broken, but right now we can't actually observe that because there's no way to print a Dyn or convert it back to a Java object, so it doesn't really matter.  I could add a toString() to the implementation to be able to see the results.  I could write Rails for it, with all the method_missing goodness involved.

My question to you who have read this far is, if Dyn was made more a part of a static language instead of a clumsily-added library as shown here, would you be tempted away from Ruby et al?  Let's, for this post, reserve the [] brackets to say 'in here be dynamic code'; any time we don't want the typechecker to look at our code, we put it in there, so we might write something like: Animal animal=new Dog(); [ animal.woof(); ]  You could write all your code between [ and ], and gradually move it outside when you know how to make it type check, hence incrementally getting better reliability and performance (but reduced flexibility).

Note that this isn't the same thing as adding explicit static types to a dynamic program.  Good static typing doesn't need explicit types everywhere, thanks to type inference.

Would this be an attractive option?  Does such a language already exist?



When would you choose to repeat types?
2008-02-20T19:31:00.004Z
I was surprised to see that some developers would prefer to repeat code with different
names rather than using abstractions.

In other words they name types by the way they're used instead of what they are.

I always like to consider the extremes of always applying a rule or never applying it.
If we made every use its own type we'd have lots of conversion functions to write 
to avoid code duplication (but of course those functions would be duplicates).

The other extreme would be only naming a type by what it is so you'd never have 
 NameAndAge and AddressAndYear, you'd only have StringAndInt (or Tuple[String, Int])
and the meaning would only be shown by the use.

Tony Morris generally seems to agree, but he would give names to types when the
 typesystem cannot express important things about the type, such as associativity.
AssocF[Int] instead of F[Int, Int, Int]
I've seen people prefer names for types in all cases, so they would never want 
tuples or function types in their languages.

I think these programmers have a fear of abstraction. What do you think?



A Trivial Display of How Scala's Type System Beats Java's
2008-01-21T18:33:00.000Z
This article shows that Scala lets you add methods to objects, depending on their generic types.  In Scheme, and other lisps and languages influenced by lisp, there is a construct called a cons, which is a pair of values.  Linked lists can be built up by 'consing' pairs together.  The first part of a cons can be called the 'car' (if used as a list, this is the head), and the second the 'cdr' (if used as a list, this is the tail).  So if you have a cons with a cons as its second element, you can think of that as a compound data structure with three elements.  E.g.:

(cons 1 (cons 2 3))

The first value of the second part of that, 2, the head of the tail, the car of the cdr, is known as the cadr.  Let's write a Cons class in Java and Scala and then look at how to add the cadr.

Consider this Java code:
class Cons<A,D>
{
    public final A car;
    public final D cdr;

    public Cons(A car,D cdr)
    {
        this.car=car;
        this.cdr=cdr;
    }
}
equivalent to this Scala:case class Cons[A,D](car: A,cdr: D)(yes, that's all)

Now let's think about cadr.  If a Cons has a car of type A and a cdr of type D, what type is the cadr?  It's the type of the car of the cdr, which we haven't got a type parameter for.    We could try this:

class Cons<A,D extends Cons<E,F>> but there's a recursion problem here -- F must extend Cons<E,F>>, not going to work. Plus, even if that worked, we could no longer use Cons for simple pairs.

We can make a static method elsewhere:
public static <A,D,E> D cadr(Cons<A,Cons<D,E>> cons)
{
    return cons.cdr.car;
}
That works fine, but it makes cadr somewhat a second-class citizen.  It's a shame we can't do pair.cadr.  In steps Scala.implicit def addCadr[A,D,E](cons: Cons[A,Cons[D,E]])=new { def cadr=cons.cdr.car }

Cons(1,Cons(2,3)).cadr gives 2addCadr, when in scope (it can be imported if it's not written where we need it) defines an implicit conversion from Cons[A,Cons[D,E]] to an anonymous class that implements cadr.  So any time we ask for cadr the compiler (not the runtime) looks at our Cons type, doesn't see cadr and looks for any implicit conversions that result in a type that implements cadr.  In this case it finds one.

The above is probably not a very practical use, let's think of some.  You can make a List[Char] have the same methods as a String, for convenience.  You can make a List[Double] have a total method.

(The worst thing about Java generics is how annoying they are to type in blogger!)



In Defence of (0/:l)(_+_) in Scala
2008-01-07T04:11:00.000Z
A few days ago, Doug posted a rather angry-sounding exit note about Scala, based on two and a half months of 'an in-depth look at the Scala language'.  He pointed out some code as an example of how write-only Scala is:

(0/:l)(_+_)

I had no clue what this meant (the first part anyway) at first.  Zero Slash Colon One?  Is that a new band?  I typed it into my Scala interpreter, and while I was typing it, I realised that the One was an Ell.  Let's change the name now.  l is a bad name.  Call it x, xs, or kiwiFruit, but not l:

(0/:list)(_+_)

I then had a bit more of a clue.  From the Scala book, I remember reading that operator names that end with a colon are right-associative.  In other words, a+b is shorthand for a.+(b), but a+:b is shorthand for b.+:(a).  So the above code can be rewritten as:

(list./:(0))(_+_)

This doesn't look a lot clearer, but we can now look up the /: method in the Scaladocs.  It's not on List, it's on one of List's traits, Iterable.  It's a fold.  Here's a longer way of writing the code (and in a way I'd have understood immediately):

list.foldLeft(0)(_+_)

If you're still left bewildered by this version, let's go a bit further.

(_+_) is an anonymous function that takes two values and adds them with the + operator.  Let's make up two names and roll them into place.  x gets rolled into the first _, y into the second:

list.foldLeft(0)(x, y => x+y)

And if anyone's still not following, what this does is to sum all the elements of list.  If the list is List(2,3,4) it looks like 0+2+3+4.  Some languages/tools call it reduce (notably MapReduce/Hadoop), some call it inject (Ruby.  Any others?).  Some languages make you use a for loop, which I assume must be to stop you from getting too cocky about how good your code looks.

So, 1 minute after not understanding the original code, I understood it.  The blog in question got mentioned on the Scala mailing list, and Martin Odersky, Scala's inventor, apologised, kind of:
That was my fault. I included it because I liked it, and that for two reasons:

1. (z /: xs) (op) looks like an abbreviation of a left leaning tree
with a `z' on the lower left end (at least to me). I.e. something like

      op
    /    \
   op    x1
  /  \
 z   x0

That's the tree I always draw when I explain fold left.

2. (z /: xs) has the operands in the ``right'' order.I can see both points, and I'll still use foldLeft.  I would not balk at someone else's code using /:.  It took me 1 minute to understand, and now I can happily read folds written that way.

In discussions with other Scala programmers, I tried to say that the time taken to learn things isn't as important as how useful they are once learned, but I couldn't find a good way to say it.  David MacIver, in an unrelated post to the mailing list, said what I intended to, but much better:

"Optimising your notation to not confuse people in the first 10 minutes of seeing it but to hinder readability ever after is a really bad mistake."

If only I could get him to stop arguing with Ian Clarke and write stuff I want to read!

If I used folds a lot, and perhaps I will someday, I would quite happily use the /: operator.  Once I've internalised its meaning I can just get on with reading the names that I chose for things, rather than reading what the language forces me to have.  Of course, to most of us, myself included, I'm not that used to folds, but if I considered folding to be just as primitive as +, I'd much rather write /: than foldLeft, just as I'd rather write + than plus.

The rest of Doug's blog really surprised me; I don't know how two people can spend the same amount of time in a programming language's community and get such different results.

My only explanation is that I am comfortable with Haskell, and a long-time Java programmer, so I probably have an advantage when it comes to Scala, which is largely Java minus some bad things, plus some good things from Haskell and many other places.

And while it's in my text buffer:

Don't assume that Scala is only useful for writing web apps, desktop apps, object orientated programming, functional programming, scripting, encoding mathematical properties and concurrency just because that's all that's been discussed on the mailing list this week.



Make Scala Your Language for 2008
2007-12-29T01:38:00.000Z
Scala's a statically-typed language based on Java, but with features that make it comparable to Ruby, Groovy, Haskell, Python, Erlang and Smalltalk.  It's pronounced "Skah-la", rather than "Skay-la", it has closures, gets rid of Java's controversial checked exceptions, and is almost perfectly interoperable with Java APIs.

There's an official book nearing completion (available now in PDF form), and some very clever people are using Scala (and some not so clever ones, like me).  One of the core developers is called Lex Spoon, which has to be a plus for any language.  Scala's been cooking, and by the time you finish reading this and procrastinating about whether to use it, it'll be ready.

It is one of those languages where boilerplate isn't welcome, yet is statically typed and supports both OOP and functional programming without blinking.  Scala programs can use Java APIs effortlessly, and Scala turns out to be better than Java for testing Java code (despite some integration concerns raised by Ola Bini)!

So how can you use it?  Obviously, install it, then you can launch its interpreter (you can use it as a scripting language or a regular compile-to-bytecode JVM language [though the difference is an elaborate illusion]).  Here I'll launch the interpreter with the Google Translator API on the classpath:
$ wget -q http://google-api-translate-java.googlecode.com/files/google-api-transla
te-java-0.26.jar
$ scala -classpath google-api-translate-java-0.26.jar
Welcome to Scala version 2.6.0-final.
Type in expressions to have them evaluated.
Type :help for more information.

scala> import com.google.api.translate.{Language,Translate}
import com.google.api.translate.{Language, Translate}

scala> import Translate.translate
import Translate.translate

scala> import Language.{ENGLISH,SPANISH}
import Language.{ENGLISH, SPANISH}

scala> translate("bastante facil",SPANISH,ENGLISH)
res0: java.lang.String = Fairly easy
So it's pretty handy for trying out APIs, even built-in ones.  Let's look at replacing all backslashes in a String with forward slashes, presumably to insert the resulting code into our Java program.scala> "blah\\blah\\".replaceAll("\\","/")
java.util.regex.PatternSyntaxException: Unexpected internal error near index 1
\
 ^
        at java.util.regex.Pattern.error(Pattern.java:1700)
        at java.util.regex.Pattern.compile(Pattern.java:1453)
        at java.util.regex.Pattern.(Pattern.java:1130)
        at java.util.regex.Pattern.compile(Pattern.java:822)
        at java.lang.String.replaceAll(String.java:2190)
        at .(:4)
        at...
scala> "blah\\blah\\".replaceAll("\\\\","/")
res2: java.lang.String = blah/blah/
As you can see it's useful for prototyping little bits of Java too.  But Scala in this case has a better way, too.  Strings delimited with """ do not need any escaping (and can be multiline):scala> """blah\blah\""".replaceAll("""\\""","/")
res4: java.lang.String = blah/blah/Now the only escaping necessary is what Java's regex implementation requires.

Scala's method call syntax can be used without punctuation in some cases.  x.y(z) can be written x y z, and x.y() can be written x.y.  It also has implicit conversions, so if you define a conversion from type X to type Y, it looks as though X has all Y's methods.  The type Int has a conversion to RichInt, and RichInt has a to(Int) method, so I can do:
scala> 1.to(10)
res0: Range = Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
or even better:scala> 1 to 10
res1: Range = Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
The lack of punctuation makes some things really attractive:scala> (1 to 10) ++ (20 to 30) map (_.doubleValue) map Math.sqrt filter (x => x-
x.intValue>0.5) map (x => x*x) map Math.round
res60: Seq[Long] = Array(3, 7, 8, 21, 22, 23, 24)This code takes the range 1 to 10, the range 20 to 30, and concatenates them together, then gives another range with the same values as the first but as doubles, then another with the same values as the second, but square rooted, then it gives another range with only those square roots whose fractional parts are greater than 0.5, then another range with the remaining values squared, then another with rounded values of those.

You can read the code pretty much like a bash pipeline.  Here's the same thing how I imagine a bash programmer would like it:
concat $(range 1 10) $(range 20 30) | map doubleValue | map Math.sqrt | filter x-x.intValue>0.5 | map x*x | map Math.roundAs with bash, each new Range does not stomp over the memory of the previous one.

Let's insert the punctuation again to see how it looks with Java-style punctuation:
scala> 1.to(10).++(20.to(30)).map((_.doubleValue)).map(Math.sqrt).filter((x =>
 x-x.intValue>0.5)).map((x => x*x)).map(Math.round)Eek!  I think it's safe to say we wouldn't write such elegant code so often if we had to write (and read) the punctuation!

This use of methods as if they were infix operators is really powerful; so powerful that it is used for what we normally call infix operators.  3+4 is just 3.+(4) (operator precedence rules are preserved though).

That's enough for now.



A Functional Way of Testing OOP Programs
2007-12-16T16:08:00.000Z
A message in OOP implies effects on entities[1], rather than mathematical functions.  If you use mathematical functions in your code, how often they're evaluated isn't important:    print cos 0*cos 0 is equivalent to:
    let x=cos 0 in print x*xIf you instead say that cos 0 is a message you send to an object, then you can't make that optimisation without knowing how the code you're calling works, because you'd be eliminating a message.  cos 0 as a message may cause effects that you can't easily see as a caller; collapsing two messages to one can introduce different behaviour.

However, most of the time that you send an object a message it doesn't appear to perform an action, it just returns you some value.  I'll name a method called for its value a function, and a method called for its effects simply a method.

If you can intercept effects that methods cause, then the method no longer causes effects, but describes them.  In other words, if you notice an action and control whether it really happens, you've made the method appear to be a function, and you've made it easier to test, because you can observe all the actions that happen.

Such an interceptor, a body of code that intercepts effects as described could use metaprogramming of some sort, perhaps by changing classes directly at runtime, perhaps through compile-time techniques such as macros.  However implemented, it would apply an automated transformation to the innards of methods.  Let's see what we'd want that to generate, by writing the result of the transformation ourselves.

The interceptors in the following code vet, log or reject effects.  I've made an interceptor return an interceptor on each call so that interceptors themselves can be implemented using return values rather than state changes.  The code in this article is something a bit like Java, so that implementation details of a particular language don't get in the way.
public Interceptor writeSomeTextToFile(interceptor,text,file)
{
    (interceptor,val out)=interceptor.new FileStream(file)
    interceptor=interceptor.write(out,text)
    return interceptor.close(out)
}
It looks doable, but pretty ugly.  One part of it can be improved.  If we change Interceptor so that it can has a type parameter, we can get rid of the tuple return that interceptor.create gave:public Interceptor[Nothing] writeSomeTextToFile
    (Interceptor[Nothing] interceptor,text,file)
{
    Interceptor[FileStream] out=interceptor.new FileStream(file)
    Interceptor[Nothing] two=out.write(text)
    return two.close(out)
}It's really up to the Interceptor now what it does with that code.  It could run the effects there and then, store them and never execute them, and our code would be none the wiser, because we haven't seen any mechanism for getting values out of the Interceptor.

The code processor to add interceptors would be pretty handy.  Let's say we have an annotation that instructs some build tool or macro to do that, so now our source code looks like:
@WithInterceptor
public void writeSomeTextToFile(text,file)
{
    val out=new FileStream(file)
    out.write(text)
    out.close()
}
Our unit test can look like:val passed={
    val ceptor=new LoggingInterceptor()

    return ceptor.invoke(writeSomeTextToFile,"hello","/etc/passwd")
                 .matches(list(creating(FileStream,"/etc/passwd"),
                               writing(FileStream,"hello"),
                               closing(FileStream)))
}It's now clearly far easier to reason about and test the method, because you can trivially observe all of its side-effects.  You could even decide which ones to allow, externally to the code, to implement a sandbox.  In the usual case that you want to execute the effects immediately, you can still do that.

This is a very long-winded way of showing that methods are functions in disguise.  Allowing methods to have difficult-to-notice side-effects makes them harder to reason about.  It's harder to write tests for them, it's harder to think about them.

This interceptor technique could be applied to existing code, to compare the effects that a 1,000 line method has, to the effects that a refactored version of it has, in the same way we often write unit tests that compare returned values.  It seems to make such a good regression test framework that I'd be very surprised if it didn't already exist for most mainstream languages.

The interceptor technique is very heavily based on monads (and may even just be a monad).  Haskell programmers, the biggest monad users today, even have special syntax for the interceptor chaining; the translation I mentioned is built into their compiler.  In fact, they can do all the things OO programmers do, but they make it harder to have unwanted side effects.  To my knowledge though, Haskell's IO monad is largely implemented as a compiler hack, so it's hard to write the same tests for side effects that I've showed in this article.

[1] "in object-oriented programming languages such as Smalltalk or Java, a message is sent to an object, specifying a request for action." -- http://en.wikipedia.org/wiki/Message



Java 7 Example - Writing Your Own Foreach
2007-11-01T13:17:00.000Z
One of the promises of closures was that if Java 5 had closures instead of foreach, you could have implemented foreach as a method.  Let's put that to the test with the Java 7 prototype.

Firstly, the 'control invocation syntax', which is a little subjective, hasn't been implemented in the prototype, so anything I show here is certainly less than foreach could be with closures.

Here's a simple attempt:
public static <T> void foreach(T[] ts,{T=>void} block)
{
    for (int a=0;a<ts.length;a++)
        block.invoke(ts[a]);
}I can call this like so:

foreach(new String[]{"hello","world"},{String s=>System.out.println(s);});

I found that I kept repeating one error in testing out this prototype, namely forgetting the ; for a statement in a {something=>void} closure.

Anyway, the above doesn't work with continue, break or return.  Those are not bound yet by the closures prototype.  No matter, let's roll our own (ignoring return; I don't know of a way to apply the following technique to return).

Not only can we pass a closure to a method, but the method can pass a closure to our closure!  No, I've not yet gone mad:

public static <T> void foreach(T[] ts,{T,{=>void},{=>void}=>void} block)

Perhaps the T,{=>void},{=>void} is better abstracted into a ForeachControl class or something.  Imagining that it was, the T field would be called 't', the first {=>void} would be called 'brake' (as in break, but avoiding collisions with the keyword), and the second {=>void} would be called 'cont' (as in continue).

For now, I'm quite happy to use the above.  I'll just reiterate it:

{T,{=>void},{=>void}=>void} is a function type that takes a T, and two {=>void}s and has no return value.  A {=>void} is a function type that takes nothing and returns nothing, analogous to Runnable.  Here's some example usage:

String[] input={"fish","print","fingers","don't print"};

foreach(input,{String s,{=>void} cont,{=>void} brake=>
        if (s.startsWith("fish"))
                cont.invoke();

        System.out.println(s);

        if (s.startsWith("fingers"))
                brake.invoke();
});Ok, that kind of looks like a foreach statement now, plus some baggage for loop control.  Actually that's as far as I can go in the current prototype.  Let's implement foreach then.

cont and brake both need to 'send a message' to the foreach method, without allowing the closure to finish.  Without convoluting the usage code above, I can do that by making cont and brake throw exceptions, which is very similar to how continue and break are planned to be supported in closures, except I'm doing it in-language.
public static <T> void foreach(T[] ts,{T,{=>void},{=>void}=>void} block)
{
    class Continue extends RuntimeException { }
    class Break extends RuntimeException { }

    for (int a=0;a<ts.length;a++)
    {
        try
        {
            block.invoke(ts[a],{=>throw new Continue();},{=>throw new Break();});
        }
        catch (Continue c)
        {
            continue;
        }
        catch (Break b)
        {
            break;
        }
    }
}Look at the block.invoke line.  I'm passing two closures to block.invoke - one that throws a Continue and one that throws a Break.  Other than that, this method is pretty simple.

As I've said elsewhere, even when/if the control invocation syntax appears, you won't be able to implement foreach exactly as in Java 5, because int cannot be a type parameter, and the closures spec doesn't specify any boxing between int and Integer for type parameters.

Even if you never use this code (I won't!), you can see at a small scale the power of thinking in closures, especially for implementing language features.  This is one of the reasons why Smalltalk was such a small language - you could implement much of what Java programmers think of as language as library methods.  Lisp and FORTH are small languages (at least conceptually - ignore Common Lisp!) for similar reasons.  Java 7 is aiming in the same direction.



Java 7 Example - Pattern Matching
2007-10-31T18:27:00.000Z
One way you can measure a programming language is by taking a feature, either one that it has, or not, and seeing how close you can get to implementing that feature without writing your own parser, etc.  Let's take a look at (and copy!) one of Scala's (and many other languages') features; pattern matching:animal match {
    case Dog() => "woof"
    case Cat() => "meow"
}It's very similar to a switch statement, and the above really doesn't show how flexible pattern matching is, but it will suffice for now.  The usual way of writing the above code in Java would be with some nested instanceof checks and casts, or by adding a speak() method to the Animal type hierarchy.

It's not always practical to add methods to types directly in Java.. e.g., if you're dealing with classes you don't control.  And instanceof is fallible - you don't want to have: if (animal instanceof Car) pass compile-time checks, but it does.  So let's see how close we can get to the above pattern matching using the Java 7 prototype.  I hope you can do better than me!

Here's what I'd ideally want in Java:
Animal animal=Math.random()<0.33 ? new Cat() : Math.random()<0.5 ? new Dog() : new Horse();

System.out.println(match(animal,
                        {Cat c=>"meow"},
                        {Dog d=>d.name+" says woof"},
                        {Horse h=>"neigh"}));What type would match take as parameters?  For now, let's imagine that we're always dealing with Animals, and always returning Strings.

public String match(Animal animal,{? extends Animal=>String}... cases)

That's not legal - {? is an "illegal start of type" according to the compiler.  Neal said that because the things on the left hand side of a closure automatically have ? super in their types, I can't also have ? extends.  If I replace it with the interface that javac generates it compiles fine:

public String match(Animal animal,OO<? extends Animal,String,null>... cases)

(the null part there specifies the exception types that may be thrown - in a function type you can omit that)

We can't actually call that method without getting at least a warning from the compiler, thanks to varargs using arrays instead of generics.  I'm sure that must have made sense to someone at the time.  If I convert it to List<OO<? extends Animal,String,null>> cases, I still have a problem.  Inside match, I don't know what type ? is, so I can't do anything with it.  Next trick then:
class Case<T>
{
    public final Class<T> clazz;
    public final {T=>String} function;

    public Case(Class<T> clazz,{T=>String} function)
    {
        this.clazz=clazz;
        this.function=function;
    }
}
Now I could have List<Case<? extends Animal>> as the parameter type.  Here's some sample usage then:System.out.println(match(animal,
                         new ArrayList<Case<? extends Animal>>(){{
                             add(new Case<Cat>(Cat.class,{Cat c=>"meow"}));
                             add(new Case<Dog>(Dog.class,{Dog d=>"woof"}));
                             add(new Case<Horse>(Horse.class,{h=>"neigh"}));
                         }}));Note that I'm taking advantage of a little trick for creating lists inline by creating an anonymous subclass of ArrayList and providing a static initialiser.  Verbose?  Yes.  Ugly?  Yes.  But enough about me.  That code will be gone in a moment.

Look at each line.  I've got Cat three times in the same line.  Ugh.  In fact if I make match generic, so that it can return something other than String, then I'll have to add a type parameter to Case for that, and then those lines become even worse.  There must be a better way.

I can use a fluent interface/embedded DSL, or, erm, in real words, "readable code" by making match(animal) return a matcher, and putting a method on it, add, that has a type parameter, U, and takes a Class<U> and a U=>String.  In fact, let's go the whole generic hog and make it take a Class<U> and a U=>R, where R is the return type we want.  When we've finished adding cases, we call done(), which returns R.

Usage:
System.out.println(match(animal)
        .add(Cat.class,{Cat c=>"meow"})
        .add(Dog.class,{Dog d=>"woof"})
        .add(Horse.class,{Horse h=>"neigh"})
        .done());
One step at a time then.  match is a simple wrapper method around a type called Matcher, that lets us avoid writing new Matcher<Animal,String>:public static <T,R> Matcher<T,R> match(T t)
{
    return new Matcher<T,R>(t);
}
Matcher is a class that stores the T passed into its constructor, and an R to return, which is initially null.  It has a method, add, that has a type parameter, U, which extends T.  Having U extends T guarantees I don't add an invalid case, such as one that tests whether an animal is a Car.
class Matcher<T,R>
{
    public final T t;
    public R r;

    public Matcher(T t)
    {
        this.t=t;
    }

    public <U extends T> Matcher<T,R> add(Class<U> aCase,{U=>R} f)
    {
        if (aCase.isInstance(t))
            r=f.invoke(aCase.cast(t));

        return this;
    }

    public R done()
    {
        return r;
    }
}The type parameter U is declared to extend T so that you cannot add a case that is impossible, e.g., .add(Car.class,{Car c=>"beep"}) (which makes no sense unless a Car is an Animal).

Get this code, as one compilable/runnable file.



Java 7 Prototype - Surprises and Expectations
2007-10-30T01:50:00.000Z
In my first week of testing the prototype, I found quite a lot of bugs, and some more subjective defects.

Neal Gafter seems amazingly fast at fixing bugs, so there's probably not much point reporting any bugs on this blog.  I'll run through some surprises instead.  If you want to see the bugs I found, take a look in the test/ directory in the prototype.  Neal put some of my bug reports in as Clarkson2.java etc.  It looks like he uses some automated testing on those, so if he breaks one he should know before I report it again!

If there's anything in this code that you don't understand, just ask.  I mainly wrote it to test the prototype, so I wasn't overly concerned with writing good code.  Most of these code samples are from bigger programs that I chopped down whenever I found a bug.

And now to the meat.

1.  Definite assignment rules make it difficult to write recursive closures.

(1)

{int=>int} factorial={int x=>x<2 ? x : x*factorial.invoke(x-1)};

The above only works if factorial is a field, not a local variable, because factorial is not 'definitely assigned' on the right hand side of the =.  I've been playing with Scala a bit recently (which has given me some good fuel for testing this prototype with), and it doesn't have this restriction.  After a conversation with David MacIver, a Scala programmer rather than a dabbler like myself, we came to the conclusion that it's possible to cause a NullPointerException in similar code to the above in Scala.

Lo and behold, Neal replied with much the same argument - if the definite assignment rules were relaxed, the following code could fail, or even do undefined things:

(2)
{int=>int} factorial=f({int x=>x<2 ? x : x*factorial.invoke(x-1)});

If f() invoked the closure, then factorial would be accessed before it had been assigned a value.  In Scala that is guaranteed to give a NullPointerException, because all variables are assigned to null/0/false before being assigned their real value.  In Java, the verifier would not load the code, because it would possibly use values that had been in memory before.

However, for the actual case above (not (2), (1)), it's easy to see that the closure is not invoked before factorial has been assigned a value.  I think this idiom is useful enough to make a special case for, but then I'm not involved in writing either the code or the specification so I can't really estimate that task.

The obvious workaround would be:

{int=>int} factorial=null;

factorial={int x=>x<2 ? x : x*factorial.invoke(x-1)};

..except that local variable capture is not supported yet.  The current workaround is to make factorial a field instead of a local variable.

If I ever get my head around combinators properly I think I'll have a better workaround.

2.  Java 5's foreach statement cannot take a closure.

This code fails:

for (final String s2: {=>asList(args).iterator()});

This code works:

Iterable it={=>asList(args).iterator()};

for (final String s2: it);

The only difference is that I explicitly made closure conversion happen.  Typecasting would not fix it (ClassCastException).  This is apparently as specified, as a foreach is only specified to take an Iterable or an array.  It's not a huge issue, and I can't think of any use cases, but it's still a surprise.

3.  Autoboxing doesn't apply to function types.

{Integer=>Integer} f={int x=>x*2}; is a compile error.

{int=>int} g={Integer x=>x*2}; is a compile error.  This doesn't look insane on its own, but because generic type parameters cannot be primitive types, it has repercussions.  I can't write a generic map function and use it like this:

Iterable<Integer> mapped=map(asList(1,5,11,26),{int x=>x*x});

I have to use the more verbose and more indirect-looking {Integer x=>x*x}.

There's no way of writing a single generic method that converts a primitive function to a wrapper function, again because generic type parameters cannot be primitive types.  I could write 8 individual methods, e.g.:
public static {Integer=>Integer} lift({int=>int} f)
{
 return {Integer i=>f.invoke(i)};
}etc.

However, there's again no way of generalising those to make them work with {int,int=>int}, etc.  There is no subtyping relationship between different function types, except the covariance and contravariance rules, quickly demonstrated here:

{Number=>Number} f={Object o=>5};

The above compiles because any Number input is obviously also an Object, and because the output, 5, is obviously a Number (autoboxing taken for granted).

The lack of autoboxing also means that sometimes you'll have to explicitly supply the generic types.  Here's an example:
class Main
{
        public static void main(String[] args)
        {
                int i=id().invoke(5);
        }

        public static <T> {T=>T} id()
        {
                return {T t=>t};
        }
}
This code won't compile, because the compiler judges the <T> in the call to id() from main to be int, which cannot be a type parameter.  Neal says this isn't a bug.  This version compiles:class Main
{
        public static void main(String[] args)
        {
                int i=Main.<Integer>id().invoke(5);
        }

        public static <T> {T=>T} id()
        {
                return {T t=>t};
        }
}I think this is a bug; it seems that inference should pick Integer instead of int in the first case.

Another implication of the lack of autoboxing is that when the closure invocation syntax arrives, that Neal promised would let you reimplement the foreach loop yourself as a method, you won't be able to.  I expect this will work, if you write your own foreach method:
foreach(Integer i: asList(3,4,5))
 System.out.println(i*i);
but this won't:foreach(int i: asList(3,4,5))
 System.out.println(i*i);
4.  Closures cannot implement generic methods.

This one is much easier to describe in code than words.  Given:
interface Identity
{
 <T> T id(T t);
}
..it is impossible to implement this with a closure.  I think Haskell allows something like this.  Scala doesn't.  A use for this would be in implementing some kind of Either type, where an Either<X,Y> may hold an X or a Y, and you don't want to actually have to test which it is, just call a method in Either, supplying a function to run if it holds an X, and a function to run if it holds a Y.  The return type of that method should be generic.

Stay tuned for some examples that actually do work with the prototype, at least one of which is actually interesting!



Life in the old Java yet - Closures Prototype
2007-10-22T10:50:00.000+01:00
I get home from eating some overpriced underportioned seafood, and drinking a few stouts, phone my girlfriend and check my email.  Nothing out of the ordinary, nothing interesting.. but wait!  An email from Neal Gafter with a closures prototype!  Wow!

So I download it, follow the instructions and it fails.  Hmm.  I'll try on Linux.  javac -version works.  I write my first working Java program with closures.  I write my second!  I mention it in an IRC channel or two, excitedly!  I remember that I'm still on the phone and have no idea what she said for the past 5 minutes, so I lower my laptop lid.  I get off the phone as quickly as possible, which is to say we talk for another half hour, which is to say she talks for another half hour.

An hour or two after that I send Neal a list of about 5 or 6 problems with the prototype, but go to bed happy, especially having drunk a Spitfire.  A beer, not a WWII aeroplane.

First thing in the morning, he's fixed the problems, so I download again.  The same problems are there.  Hmm.  Oh, right, the filename included the date.  What's today's date (it was 2am today that I sent the emails)?  2007-10-22.  But Neal's in America, and they're backward, so 2007-10-21.  Ok, got the right file now.  Still fails in Windows, and at least one of my reported failures still fails but for a different reason.  I start playing again, and some more code fails for the same reason as the first, so I give up for now.

There's much more fun in breaking a prototype compiler than a production one!

Ok, new day, and I'm at my girlfriend's house for the evening.  Neal's just sent another prototype, and it still doesn't work in Cygwin.  I can't get my laptop online to connect to Eugene Ciurana's Linux box that I was testing on, so I start looking at the bin/javac script.  I see and fix the not-on-Windows problem.  It's passing a full path to lib/javac.jar to javac, and that includes /cygdrive/c, Cygwin paths, which Java doesn't know about.  I find a fresh batch of bugs and surprises.  After finding the same bug twice from unrelated pieces of code (and getting pestered to watch a film) I stop trying.  Neal confirms the bugs, but nothing new arrives the next day.

Over the next few days I find more and more bugs, and Neal fixes them all.  There are a few bugs that turn out to be features; they're actually in the spec that way.

Now that I have a fresh set of bugs waiting to be emailed to Neal, he's just published the closures prototype.  You can find it over at his blog.

I'll shortly publish some surprises from playing with the prototype, then some examples of what you can (and can't) do with it.  In a less commentative style..



IDEA 7 - A Release Too Early?
2007-10-16T16:16:00.001+01:00
I downloaded the new IDEA 7 release today, and pondered buying the upgrade licence, but went with the 30 day trial first.  Here's a little video I made of an editing bug that I noticed immediately.

(If anybody can figure out how to get a .swf into a blogger page, please tell me how.  Blogger just mangled any HTML I used to do it.)

Another niggle is that, including the milestone releases, the inspections that use @Nullable and @NotNull seem not to be present, and with no mention of why.  Hopefully they're now just in some plugin that I don't happen to have installed.  That's a bit of a show-stopper for me, I changed my coding style to use those annotations.  I'd have to convert to FindBugs' equivalent to continue using that coding style.



Why Java Needs Closures (It Already Has Them)
2007-10-10T10:57:00.000+01:00
Some bloggers and people I've talked with think of closures in Java as something unnecessary - e.g., "why must we copy C#?".  As it happens, most languages have closures, including Java.

A closure is an executable block of code that can refer to free variables from the enclosing scope.  Here's an example in Java:

public void example()
{
        final double use=Math.random()*10000;

        SwingUtilities.invokeLater(new Runnable()
        {
                public void run()
                {
                        System.out.println(use);
                }
        });
}
It's not a great example of how closures are useful.  The instance of that anonymous class can reference the variable 'use'.  In fact, there's a leak in the implementation that gets into the language - 'use' has to be final.  That doesn't stop the anonymous class from being a closure.  Let's consider how life would be if Java didn't have closures:public void example()
{
        double use=Math.random()*1000;

        class MyRunnable implements Runnable
        {
                private double use;

                public MyRunnable(double use)
                {
                        this.use=use;
                }

                public void run()
                {
                        System.out.println(use);
                }
        }

        SwingUtilities.invokeLater(new MyRunnable(use));
}
In fact, that's not far off what the first code sample gets compiled to.  In Java 1.0, inner/nested/local/anonymous classes didn't exist, so you'd have to take the above MyRunnable, and put it in a separate file, but the code would be the same otherwise.  If you were going to complain about Java getting closures, Java 1.1 was the time to do it!

The above two code samples work in today's Java, and some of you might even favour the second, because it's more explicit - it matches the bytecode more closely.

The way I think is that you should, at least as a thought experiment, take anything like "favour explicit code over abstract code" to an extreme, to test it out.  The most explicit you can be in programming is to write assembly code, and none of us wants to do that.  Let's flip it around.  The most abstract you can be in programming is to write in Lisp, and none of us wants to do that.  Both of those statements have holes in them, of course, there are some people who love writing assembly, and there are some who love writing in Lisp.  I'm one of the latter.

As it turns out, abstraction is so prevalent in programming that most of us program in languages that are closer to Lisp than they are to assembly.  C is the lowest-level programming language that I know, and even it has huge abstractions over the underlying assembly.

Java programmers are already programming on an abstraction, the Java Virtual Machine, before they even worry about syntax.  It seems that we already favour abstract code over explicit code.  Abstract doesn't mean imprecise, or vague, it's another way of saying "general".  We can use abstractions to stop repetition.

Consider the above code samples as templates of some kind:
        final $TYPE $VAR=$VAL;

        SwingUtilities.invokeLater(new Runnable()
        {
                public void run()
                {
                        $ACTION($VAR);
                }
        });
and        $TYPE $VAR=$VAL;

        class $BLAH implements Runnable
        {
                private $TYPE $VAR;

                public MyRunnable($TYPE $VAR)
                {
                        this.$VAR=$VAR;
                }

                public void run()
                {
                        $ACTION($VAR);
                }
        }

        SwingUtilities.invokeLater(new $BLAH($VAR));The first template has 4 parameters, $TYPE, $VAR, $VAL, $ACTION.  It mentions $VAR twice, the others once each.

The second has 5 parameters, $TYPE, $VAR, $VAL, $ACTION, $BLAH.  It mentions $TYPE 3 times, $VAR 6 times, $VAL once and $ACTION once.  We're always taught not to repeat ourselves in programming, and we can easily see that the second template is more repetitive than the first.  A less well-known rule is to avoid unnecessary names.  MyRunnable is a name that's defined once and used once - a bad sign.

Extra repetition means extra scope for errors.  You might change 'use' between calling invokeLater and invokeLater actually happening - now you've got a sync problem.  This is because you had to copy your variables yourself.  Admittedly, thanks to the 'final' restriction, Java doesn't help much there, but at least it stops broken code from compiling.

Let's briefly return to the restriction that the local variables that anonymous classes capture must be final.  Does that stop anonymous classes from being closures?  Changing the values of variables is kind of an optional feature in programming languages.  Mathematics seems to have managed without x++ for many years.  Java's anonymous classes place some restrictions on what kind of variables are considered free, but that doesn't stop them from being closures.  Haskell definitely has closures, and definitely doesn't have mutable variables.

Ok, with all that out of the way, we now know that Java has closures, and they aren't disappearing anytime soon.  Let's take a moment to laugh at the blog posts (actually most of them are anonymous comments on blogs) saying that Java doesn't need closures.

There.  Now let's briefly examine why Java needs better closures, by returning to our template.  This time we're going to read it with boilerplate glasses:
        boilerplate double use=Math.random()*10000;
        SwingUtilities.invokeLater(boilerplate System.out.println(use));
 What we'd like to do now is to make that boilerplate disappear.  Scala has an interesting way of doing that:        val use=Math.random*10000
        invokeLater(System.out.println(use))
Scala methods can have lazy parameters, that is, parameters that are not evaluated at call time, but are evaluated when the method wants to.  You can use that to write your own if method, e.g.:        myIf(Math.random<5,System.out.println("Still here"),System.exit(0))Of course, myIf is just an interesting result, and not actually useful, but in the case of invokeLater it is useful; it gets rid of a lot of our boilerplate.

Java 7 closures let us do pretty much the same thing:
        double use=Math.random()*10000;

        SwingUtilities.invokeLater({=> System.out.println(use));
There is another syntax for the last line:        SwingUtilities.invokeLater()
        {
                System.out.println(use);
        }
We still have a little boilerplate with both these syntaxes, but not enough to need the boilerplate glasses.  Sadly, despite it being technically possible for IDEs to become boilerplate glasses, they haven't done so.  Folding an anonymous class in IDEA looks like:        new Runnable(){...}
They've folded the wrong thing.  Duh.  It should look more like:        new ...{System.out.println(use);}Some people think that Java doesn't need better closures because Java as a language is already broken in many ways.  Often Scala is quoted as the next Java.  I've tried Scala out - it's very impressive, and similar enough to Java to not have too many surprises.  So in one sense I agree, but I also think that Java should have better closures to move the status quo of programmers up a notch.  Programmers should consciously favour abstraction over boilerplate, instead of the current situation, where many use abstractions all the time but are reluctant to invent their own.

Being free of boilerplate lets you think in different ways.  My post on point-free programming shows one route you could take - there are many others.



Point-free Programming in Java 7 - Beyond Closures
2007-09-14T18:04:00.000+01:00
Here is an introduction to point-free programming, using Haskell, Scheme, Java 5 and the proposed Java 7 for examples, and a couple of ideas for how the Java 7 proposal could improve to make point-free programming more practical.

I've grown quite accustomed to programming in lambda style now, in Common Lisp, Scheme and Haskell.  Here's a quick example of how that looks:

Scheme: (lambda (x y) (+ x y))

Haskell: \x y -> x+y

Java 5: 
new FunctionIII()
{
    public int invoke(int x,int y)
    {
        return x+y;
    }
}
Java 7: {int x,int y => x+y}

Not too bad.  You could use this to fold (or reduce) across a list.  To fold is to apply a function on a starting number and an element of a list, and then to apply the same function on the returned value and the next element of the list, until the end of the list.  You could use this to compute the sum of all the elements of a list:

Scheme: (foldl (lambda (x y) (+ x y)) 0 (list 3 4 5 6 7))

Haskell: foldl (\x y -> x+y) 0 [3,4,5,6,7]

Java 5: 
foldl(new FunctionIII()
{
    public int invoke(int x,int y)
    {
        return x+y;
    }
},0,asList(3,4,5,6,7));
Java 7: foldl({int x,int y => x+y},0,asList(3,4,5,6,7));

One point to note here is that you could, in Scheme, write this as (+ 3 4 5 6 7), because Scheme's + function takes any number of arguments, even 0 (whereupon it returns 0).  So this is a contrived example using non-optimal code.

It's not idiomatic Haskell either.  For a start, you can write sum [3,4,5,6,7], and avoid foldl altogether.  Bear with me for a moment though.  In Haskell, you can shorten \x y -> x+y to simply (+), which means 'the function called +'.  In Scheme you don't even need the parentheses.  +, used in a 'value position', which is anything other than the first in a list, refers to the function +, rather than calling it.  So you can do:

Scheme: (foldl + 0 '(3 4 5 6 7))

Haskell: foldl (+) 0 [3,4,5,6,7]

This is the simplest example I can think of of pointfree style, which is where you remove lambda expressions but without introducing mutable variables.  Here's a better example:

I want to multiply all elements of a list by 2.  Here's the lambda style:

Scheme: (map (lambda (x) (* x 2)) (list 3 4 5 6 7))

Haskell: map (\x -> x*2) [3,4,5,6,7]

Java 5: 
map(new FunctionII()
{
    public int invoke(int x)
    {
        return x*2;
    }
},asList(3,4,5,6,7));
Java 7: map({int x => x*2},asList(3,4,5,6,7));

But these lambda expressions don't really express my intent that well.  I want to map the function 2* across the list.  Haskell has a convenient syntax for this:

Haskell: map (2*) [3,4,5,6,7]

What I've done there is to specify the * function, with its first argument filled in.  This works because all functions in Haskell are automatically curried.  If I type this at the REPL:
:type (*)
Output: (*) :: (Num a) => a -> a -> a
This looks a bit confusing if you're not familiar with it.  What it means is that (*) is a function with a type parameter, a, which in Java terms would look like <a extends Number>.  It takes one parameter of type a, and returns a function that (takes one parameter of type a, and returns a value of type a).  Parentheses used for disambiguation there, this turns out to be hard to write in English.:type (2*)
Output: (2*) :: (Num t) => t -> t
That means that (2*) is a function with a type parameter, t, which is an instance of the Num typeclass (the same as 'a' in the first example).  It takes one parameter of type t, and returns a value of type t.

So I can call (2*) as an ordinary function, adding a value (this is called taking a section, and is a way of partially applying a function).  The following expressions all have the same output, 8:
(2*) 4
(*2) 4
(*) 2 4
((*) 2) 4
This seems like an interesting but not-so-useful result of automatic currying at first, but you can actually rewrite any lambda expression in this style.  Sometimes the code will be easier to read than a lambda expression, sometimes not.  Here's a more complex example:map ((+10) . (*2)) [3..6]

The . operator means function composition, like in mathematics - f(g(x)) would be written as (f . g) x.  You can compose functions before you have the values to give them, so f . g has a meaning.  Read ((+10) . (*2)) as: "multiply by 2 then add 10".  Look ma, no variables!

Scheme doesn't have any support for this, as its procedures are not automatically curried, but you can make something similar happen:

(map (compose (plus 10) (multiply-by 2)) (list 3 4 5 6))

plus and multiply-by need defining in an odd way:
(define plus
    (lambda (x)
      (lambda (y)
        (+ x y))))
(define multiply-by (lambda (x) (lambda (y) (* x y))))

You can write a general curry procedure in Scheme, but as there's no portable way of discovering how many args a procedure takes, you have to supply the arity.  Plus, some procedures are of variable arity.

(curry 2 +) would return a procedure that takes a value, and returns a procedure that takes a value and returns the sum of the two values, used like this:

(((curry 2 +) 4) 5) ;; output: 9

Java 5:
public static final FunctionIO<FunctionII> plus=new FunctionIO<FunctionII>()
{
    public FunctionII invoke(final int x)
    {
        return new FunctionII()
        {
            public int invoke(int y)
            {
                return x+y;
            }
        };
    }
};For brevity I'll leave multiply-by out..  it's the same but with + changed to *!

Java 7:

public static final {int => {int => int}} plus={int x => {int y => x+y}};

public static final {int => {int => int}} multiplyBy={int x => {int y => x*y}};

map(compose(plus.invoke(10),multiplyBy.invoke(2)),asList(3,4,5,6));

Because both Scheme and Java make you curry explicitly, they discourage point-free programming (although it's still possible, and quite useful).  I'd like to see all function types in Java 7 be automatically curried.  In other words:

{int,int => int} should be the same as {int => {int => int}}.

This doesn't have to have any negative impacts at runtime, as demonstrated by Haskell's great performance.

To make it really useful, it should also be applicable to operators, so that #+ gives a {int,int => int} and #+.invoke(2) gives a {int => int}.  (where #+ is a possible syntax for taking a reference to +).

Sometimes you'll want to provide the second argument rather than the first.  Haskell's solution to this is a function called flip:

(flip (/)) 3 6 -- gives 2, the result of 6/3.  I could use it in this possible future of Java with #/:

flip(#/).invoke(3) would return a {int => int}, that, when you pass it a number, it returns that number divided by 3.  Equivalent to Haskell's (/3).

As you do more and more in point-free style, you get more and more used to it, as one would expect, and some constructs that seem unnatural at first seem normal later.  I'm still a novice, but I'm having fun.



Stop Sitting On The Type Fence
2007-08-25T19:57:00.000+01:00
Static typing is not just about preventing simple bugs, and dynamic typing is not just about writing millions of tests that you wouldn't have to write with static typing.

Programmers of languages like Java, C#, C, C++, etc., middle-of-the-road statically typed languages with some dynamic features, have some intuitions, built up from years of experience at getting the best out of their language, but don't often stop to see what contortions their language makes them go through, or that they are limited in what their compiler tells them about their code.

They often think that rigorous static typing is about preventing bugs that a few unit tests would solve, or that dynamic typing is about writing tests that static typing would solve.

Static Typing is Not About Low-Hanging Bugs

Many years back, clever mathematicians proved that functions express types.  If you have an expressive enough type system, you only need to give the input and output types for a function, and you have its implementation.  Consider the identity function, written in Scheme like this:
    (lambda (x) x)
Don't worry if lambdas are unfamiliar to you.  No, actually, do worry.  Lambdas underpin most of computer science and certainly all of programming, even if that's not obvious.  If, as I did, you gained a Computer Science degree without ever hearing about lambdas, ask for your money back, and spend it on a copy of The Structure and Interpretation of Computer Programs (SICP) for you, all your friends and family.  It's also available in PDF form for no money, in case you're unsuccessful in getting your money back from your University.

Anyway, a lambda expression is analogous to an anonymous function in many languages, and even an anonymous class (with one method) in Java.  The particular lambda expression above is a function that takes a value, and returns that same value.

In Haskell, the syntax isn't much different:
    \x -> xThe \ is Haskell's approximation to the Greek letter lambda, the bit before -> is the parameter list, and the bit after it is the result.  Scheme doesn't care about telling you the types of things, but Haskell does.  The type of \x -> x is t -> t, which means it's a function that takes a t (which can be anything) and returns a t (which is the same as the first t).  All fairly obvious.

Given the above, there's only one logical implementation of t -> t, and that's \x -> x.  You could make silly ones, where you store x in a local variable, and then return it later, but they're all equivalent.  Or perhaps the implementation could look up the type of x, and grab something similarly-typed from a cache.  All a bit silly though.  Perhaps if all types in a language had a clone function, then you could provide a different implementation, but without that, or something similar, 

What you can see from this is that, at least for simple functions, you only need the types to derive the implementation.  Does this scale up to things like \x y -> x/y, etc.?  Yes, as it happens, though you then need the concept of dependent types, which is types that depend on runtime values.  They can still be checked at compile time.  In other words, types and values are isomorphic to each other (isomorphic is a mathematical way of saying 'equivalent').

What does the Java version of the identity function look like?  Oddly, it depends on whether you write it how Scheme works, or how Haskell works.  In Scheme there are no compile time types (or you can say there is exactly one), and the closest that Java comes to that is Object:
    public static Object identity(Object x)
    {
            return x;
    }
If you write it like Haskell, then you need to make sure that the return type is the same as the input type, with no silent upcasting (to Object):    public static <T> T identity(T x)
    {
            return x;
    }There is quite a difference between the two pieces of code, which is particularly disturbing, because the Haskell and Scheme versions look so similar to each other.  In fact, there's a project called Liskell, which gives Haskell lispy syntax.  There, the two would probably be the same.  This is a first hint that Java gets in the way of thinking about static typing, by making it very verbose, in particular by making untyped code look different to typed code.

Haskell is also guilty of this, partly by design, and partly because of some deficiencies in its type system.  That's the reason that the side of the fence I sit on is the dynamically typed one, but I keep my eye on the static typing people, because they come up with great ideas, and secretly I'd like to be able to use their type systems on some of my code.

So static typing isn't just about making sure that you don't add two credit card numbers together, it's about expressing code as close as possibly to mathematics.  Mathematicians tend to be very good at reducing problems to their smallest representations, and coming up with very general solutions, so there can be no doubt that this approach will (and already has) produced amazing results.

Haskell's type system is certainly imperfect, and so are its programmers - this is clear because of the presence of its unit testing system, QuickCheck.  If the type system was perfect, and the programmers using it never took shortcuts, there would be no need to run unit tests.  It would be evident by the compiler succeeding, that the code could not crash, or produce any results outside the expected range.

If you look for more and more sophisticated type systems, you will probably come across Coq at some point, which is a proof engine that can generate executable code.  There you deal with logic, with set theory, all things that are the pinnacle of static typing, and which my Computer Science degree, and probably yours, did not cover.

Go there.  Have a look.  Even if you decide that static typing is not for you, or that you'll get by with a lesser static typing system such as Haskell's or Java's, you can learn a very general way of thinking that will apply across a lot of programming, in the same way that understanding lambdas will help you when you see C# delegates, JavaScript's anonymous functions, etc.

Dynamic Typing is Not Just About Writing Tests

Many static typing proponents, even Haskellers, have the opinion that while Lisp, Python, Ruby, JavaScript, et al, might have some nice syntax, the fact that they are devoid of static types makes them ultimately useless.  You can't easily get the compiler to tell you if you try to do something stupid.  For example, 3/"hello world" will be accepted by the language despite it being obvious (read: provable) that a runtime problem will occur.

Many Java programmers assume that you have to guess at the types being used in a dynamic program, or provide excessive documentation.  The greater point is that what will work will work.  Dynamic languages get out of your way so that you can explore a problem.  Whereas in a statically typed language you have to make the solution consistent before you can try it out, you can try out an incomplete function very easily in dynamically-typed languages.
    (lambda (x)
      (if (< x 0)
          (do-this x)
          (do-that x)))I can call the above lambda on negative numbers, even if do-that hasn't been written yet (assuming do-this has).  The runtime doesn't generally complain unless it has to.

Once you have explored a problem, and you understand it, you probably have a nearly-working program, so you're pretty close to having a working one.  You get no guarantees that the program will work for all possible inputs, and there isn't even a general way of restricting inputs (other than adding checks at runtime).

Dynamic programmers tend to think in the language they are using - that is, they write code while they're thinking, and test functions out to see whether they work for the cases they're interested in.  If I write a function that adds two numbers together, I'm not even remotely interested in what happens if someone passes strings to it, because I'm not writing it for that use case.

This way of thinking keeps the code focused on the task in hand.  If the code starts to get hard to read, or repetitive, the programmer will come up with an abstraction.  E.g., beginner Java programmers often try this:
    if (x==3 || 5 || 10)
which doesn't compile, because the compiler sees 3 expressions that should all be booleans, and the 2nd and 3rd are ints instead.  The programmer gets told they're wrong, and they should fix it to be:    if (x==3 || x==5 || x==10)
That's clearly garbage, because x== is repeated.  The programmer has just adopted a poorer way of thinking to suit a language, instead of adapting the language.    if (x in (3,5,10))
would be great, as would:    if ((3,5,10) contains x)
, or even, as valid Java:    if (asList(3,5,10).contains(x))
Of course, this latter example is a little ugly.  Translated directly to Scheme:    (if (contains (list 3 5 10) x)
If this was used a lot, a Scheme programmer might write:    (if (in? x 3 5 10)A Java programmer could do that too, but they would need to know more of the language to be able to do it.  Therefore, the Java programmer is 'corrected', instead of shown how to write their idiom in Java.

Because the language gets out of the way, this kind of code is easy to write.  That brings about the worry that because you can write code to be as good as you like, you can also write incredibly bad code, and this is completely true.

The remarkable thing is that the language doesn't try to judge whether your code is good or bad - it just runs the code.  That's great, because it lets you learn from your own mistakes, and even better, it lets you come up with things that the language designers didn't think of.

It's become quite clear in recent years that Object-Orientated Programming as Java and C++ do it is not the pinnacle of software engineering - but you can't escape Java's OO system.  You can't really implement multiple dispatch for two separate codebases in one central place unless you go outside the language (e.g., reflection, bytecode weaving).  If it turns out that Haskell's typeclasses aren't the best way of expressing things, you can't remove them from the language, because existing code depends on them - the compiler always needs to understand them - and probably you will still have to use them because the language's central concepts depend on them.

By the language not telling you that you're wrong, it lets you be right in ways that the language designer might not have envisaged.  For example, the code snippet (+ "hello" "world") looks wrong, because + is not defined on strings.  But in a language like Scheme, + is only a variable, so I can write:
    (let ((+ string-append))
      (+ "hello" "world")) and it's no longer wrong (arguably a bit stupid though).

So how about those tests?  Well, if you look at Java code:
    public Integer add(Integer a,Integer b)
    {
        return a+b;
    }Both a and b can possibly be null, so what happens if null is passed?  Er, you get a NullPointerException.  Some programmers will specify that in documentation for the method, or just write that null is not allowed.  The same method in Scheme would be callable with any values at all, though it would fail.  That changes the attitude of calling programmers.  Instead of looking at the half-hearted type signature, and seeing that null is a possible value, the programmer is more likely to look at the implementation.

In the first half of this entry, I said that types imply implementations - well, the reverse is true!  Implementations imply types.  You don't need a type signature to see what values can be passed to (lambda (a b) (+ a b)), you can infer the type signature from the implementation.  In this case, you can pass anything to that lambda that you can pass to the primitive + procedure, whose type signature actually depends on the implementation of Scheme you're using, but it accepts at least those types in the specification.

So Scheme programmers aren't likely to write tests to see what happens if you pass credit cards to a function expecting addresses, because that's never going to be a use case.  They informally restrict the inputs.  They're emulating what a static type checker does, but mentally.  That lets them organically approach a solution to a problem, instead of having to design it beforehand.

It's certainly true that this mental processing is error-prone, and bugs get into source code because of this.  Then, many programmers are tempted to write swathes of tests, in an informal attempt to prove that their code is correct.

The bad part of this is that they aren't benefitting from the static typing camp's results now.  They have a solution, but they can't use the machine to prove it.  They'll end up writing the same kinds of tests over and over.

All this is why I think that, despite him talking about middle-of-the-road
type checkers like Java's, Gilad Bracha was on the ball with his presentation about adding pluggable type checkers to dynamic languages.  I thoroughly expect that it will be some time before a great one exists, and largely because too many people are sitting on the fence, instead of knocking it down.

If you ignore mediocre languages, there isn't a great deal of difference between how statically typed programmers think and how dynamically typed programmers think.  We all like lambdas, we all like side-effect free code, we all dislike crashing programs.  So the next time you see a proposal to add static typing to your favourite language, or to add some new dynamic feature to your static language (including 'get-out' clauses like Haskell's unsafePerformIO), bear in mind that you're just witnessing a step on the road to convergence.

If the next big language has a mandatory static type system, then it will either be a crap one, or Computer Science degrees will suddenly contain a lot more mathematics.  That won't happen.  The next big language will be crap, or dynamic.



Objects To Functions 1 - Stateless Classes
2007-08-21T13:59:00.000+01:00
Before I had really heard of functional programming, I had started to notice that it made a lot of sense to make certain objects 'value objects', i.e., create them with all the data they need and then never change them, which lets you pass them around without worrying about who changes them.

This sounds like low-hanging fruit, I mean, of course you can easily keep track of changes, just like C++ programmers always manage to track who frees an object..  Ah.  You probably see where I'm going now.  So, if I don't mutate an object, I can pass it around without thinking about the consequences.  That's why it turns out to be better to share Strings rather than StringBuffers in Java.  Of course, you can treat a StringBuffer as an immutable value, just by never mutating it, but as it is intended to be mutated, you generally don't share it, at least not without thinking very carefully.

Let's look at a very simple stateful class:
public final class Point
{
    public double x,y;
}I'm sure some readers will be complaining about the lack of get/set methods now - I'll happily explain why I don't use them if you ask in a comment.  Others may complain about final, I'll happily explain that too.

If you instantiate a Point and pass it around, some programmers will see that it's intended to be mutable, and store a copy of it (or take pains never to store it, but just use its values once), and other programmers will take the attitude of, well, if you're giving out a mutable object you'd damn well better expect it to be mutated*.  The latter sounds like an unreasonable attitude, but it's why IDEA places warnings on methods that return the values of fields that hold arrays and collections.  IDEA puts the onus on the class giving out the mutable object, rather than on the user of that object - it assumes that the latter programmer is the most likely one.

* this latter programmer often also has the attitude that while it's ok for him to mutate the object, nobody else should!

It's quite easy to make Point immutable, you can provide a constructor that sets x and y, and then make x and y final.  You don't absolutely have to do this - all it really takes to stop a data structure from being modified is to stop modifying it - you don't have to put up barriers to modification to do that.  Still, in Java modifying things is the norm, so it might be a good idea for now.  If you get to a stage where mutation is only done with great care then you might not need the barrier.
public final class Point
{
    public final double x,y;

    public Point(double x,double y)
    {
       this.x=x;
       this.y=y;
    }
}So, how does this relate to functions?  A Point is now a function, as in mathematics.  If you create a Point, it will always give you the same value for x and the same value for y.  You can think about it as a function that takes the symbol x or the symbol y and always gives the same result.  That is, objects can be or represent functions just as well as methods can.  Math.abs represents a function, because it always gives the same output for the same input, and so does point.x.

Now I can treat a Point as a long-lived value and never worry whether it will be mutated, just as I don't worry about it being garbage collected.  In effect, I've taken time out of its semantics.

You might be thinking that Point is crying out to be an interface, because programming to the implementation is inflexible.  However, I'd like all clients of Point to be able to rely on .x and .y returning the same thing, forever, and not have to worry about different implementations doing strange things.  Instead I would suggest that if you wanted a different implementation, you don't call it Point, and don't try to make it extend or implement Point.  Clearly it would be of a different type, e.g., MutablePoint, or DebuggablePoint, Quaternion, etc. - and you wouldn't want to use it in places that expect a Point.

There are cases that make this way of programming tricky, and solutions to each:

1.  Collections.  It would be pretty expensive to copy a collection instead of changing it, so in many cases you may as well change it in place, which loses you the time-independence, and stops the collection from behaving as a function.  There are some concessions you could make - e.g., only allowing adds, not removes (then any successful accesses will always succeed - you won't break assumptions).  For the more general case, there is one kind of collection in particular that works well as a function - a linked list.  Java's LinkedList class doesn't give you access to the individual nodes of the list, which is what we need for this to work, so you'd need to write your own.
class Link<T>
{
    public final T item;
    public final Link<T> next;

    public Link(T item,Link<T> next)
    {
        this.item=item;
        this.next=next;
    }
}

Link<String> list=new Link<String>("hello",null);If you feel the need to add to the list, you don't need to change anything, just call new Link<String>("world",list) and use that value instead from now on.  Then each list retains its properties as a function - it will always return the same value if you ask it the same question.

If you want to iterate over such a linked list and do something, say, finding the maximum of a Link<Integer>, you could do it like so:

public int max(Link<Integer> list)
{
    int max=Integer.MIN_VALUE;

    while (list!=null)
    {
        max=Math.max(max,list.item);
        list=list.next;
    }

    return list;
}

This works, and from the outside you can't see any problem, but on the inside you can see that time is important.  That is, the list objects always work as functions, but because the variables list and max change values, they don't work as functions - you have to think about time whenever you think of them.  It would be easier if you only had to look at the method parameters to see the values, instead of having to trace changes in your head.  This is a small-scale version of the bigger problems caused by things that change.  We can probably deal with this version with no real problems, but let's follow through and see what happens when we apply the large-scale thinking to the small scale.

Because a list is a recursive data structure, it turns out to be quite easy to rewrite max as a recursive method:
public int max(Link<Integer> list)
{
    return new Object()
    {
        public int max(Link<Integer> remaining,int current)
        {
            return remaining==null ? current : max(remaining,Math.max(current,remaining.item));
        }
    }.max(list,Integer.MIN_VALUE);
}Woah!  I'm abusing inner classes or something!  What's going on?

Java doesn't let you define a method inside a method, but it lets you define a method inside a class, and a class inside a method, so that's what I've done.  If I was going to write it in more idiomatic Java, I'd do this:

public int max(Link<Integer> list)
{
    return maxImpl(list,Integer.MIN_VALUE);
}

private int maxImpl(Link<Integer> remaining,int current)
{
    return remaining==null ? current : maxImpl(remaining.next,Math.max(current,remaining.item));
}
Now that you've understood that these two idioms are actually equivalent to each other, you can read the first one without any confusion.  Just understand that if I could declare a method inside a method directly, I would have done.  There is a problem with this code, in that it throws a StackOverflowError for medium-sized collections, on the Sun JVM, but not IBM's, so I am only using it as an intermediate case for this blog post, before reaching a final conclusion.  Anyway, let's move on and create a similar method that gives the minimum for a collection.

public int min(Link<Integer> list)
{
    return new Object()
    {
        public int min(Link<Integer> remaining,int current)
        {
            return remaining==null ? current : min(remaining,Math.min(current,remaining.item));
        }
    }.min(list,Integer.MAX_VALUE);
}

I copied and pasted the code, then changed it, which is a good sign that there's some thinking left to be done.  I can make more of it the same by renaming the method inside the anonymous class to 'invoke' or something equally generic.  I can move the starting value to the method header - it might not make sense to everyone that the minimum of an empty list is Integer.MAX_VALUE, let's allow the user to decide what value that should be:

public int min(Link<Integer> list,int minimum)
{
    return new Object()
    {
        public int invoke(Link<Integer> remaining,int current)
        {
            return remaining==null ? current : invoke(remaining.next,Math.min(current,remaining.item));
        }
    }.invoke(list,current);
}

There's no reason now not to use the outer method as the target for recursion instead of creating an inner one, so if the anonymous class annoyed you, it goes away now:

public int min(Link<Integer> list,int minimum)
{
    return list==null ? minimum : min(list.next,Math.min(current,remaining.item));
}

Wow, that's short.  It's also fairly simple.  It says, for an empty list, return the given minimum.  For a non-empty list, return ourselves invoked with a smaller list, giving the new invocation a different minimum.  Notice that nowhere here have we changed any variables.  Still, this throws a StackOverflowError too.  Two more steps and we'll get rid of it.

The only parts that are specific to minima now are the name of the method, and Math.min.  A more general version of what we're doing here is reducing a list to a single value.  There are lots and lots of reasons we might do that.  Asking whether a list contains a certain item is a reduction.
interface Reducer<T,R>
{
    R reduce(T left,R right);
}

public <T,R> R reduce(Link<T> list,R accumulator,Reducer<T,R> reducer)
{
    return list==null ? accumulator : reduce(list.next,reducer.reduce(accumulator,list.item),reducer);
}
A Reducer is something that knows how to reduce 2 values to one.  It doesn't necessarily have to mean two items from the list, just an accumulator and one item from the list.  Let's have some imaginary syntax meaning that if I type Math.min with no parentheses, I get a Reducer<Integer,Integer>:

int min=reduce(list,0,Math.min);

That says find me the minimum value in that list, using Math.min to do so.  Unfortunately, Java doesn't support such syntax, so this is what you really end up with:

int min=reduce(list,0,new Reducer<Integer,Integer>()
{
    public Integer reduce(Integer left,Integer right)
    {
        return Math.min(left,right);
    }
});

Unfortunately, this is pretty annoying - so annoying that if you use it more than once, you'll want to wrap the reducer up in a static field or method somewhere, so you end up with:

int min=reduce(list,0,Maths.minRef);

I'm English, we say Maths instead of Math, which turns out to be a handy conflict resolution here.

We still have the StackOverflowError problem, but we can resolve it by rewriting reduce in the 'old' way, with changes to variables.  Note that StackOverflowError is really a fault with Java, because at the point where reduce calls itself, there is no need to retain any of reduce's data on the stack.  Java could make it work so that reduce overwrites its own parameters and then does a goto to get to the top.  Unfortunately, it doesn't, so we have to do that ourselves (except the goto is some loop or other):

public <T,R> R reduce(Link<T> list,R accumulator,Reducer<T,R> reducer)
{
    while (list!=null)
    {
        accumulator=reducer.reduce(accumulator,list.item);
        list=list.next;
    }

    return accumulator;
}So even though we've ultimately had to make a concession for the technical problems of being in a JVM, we've got something pretty reusable out of the thought experiment.  Of course this is nothing new.  I actually wouldn't at this time recommend using linked lists in this way in Java, but closures in Java 7 and a solution to the problem of tail-call elimination might make this much more attractive.

2.  Cyclic data.

If class A needs to hold a B, and B needs to hold an A, there's no way to instantiate them both at the same time.  There are clever tricks that make them nearly instantiated at the same time, but you can't make the respective fields final in each.  A common solution is to get rid of the cycle by creating a third object, C, that holds A and B, supplying the instances of each other as necessary.

Unfortunately, the name for C is often hard to think of, which is a reasonable sign (I'm contradicting another of my blog posts here) that C is a bad thing to have at all.  I'd suggest looking at the individual case and deciding what to do.  Sometimes it's just that A and B should be merged, or that some functionality should be moved from one class to another.  In other cases, it might turn out that some methods can be made static, in which case there's no need to hold an instance.

3.  Data that will change over time.

Essentially a philosophical question, it is possible to argue that something that is different before a certain time than after it hasn't changed, that there are actually two objects.  Monads are a way of treating immutable objects as if they can change, by making the new value replace the old value seamlessly, which effectively solves the philosophical question by letting you write code nearly the same way regardless of the answer.  Using Java, though, I'd probably suggest that if data changes over time, you may as well use mutable data.  Perhaps Java 7 will make monads more attractive.


Remaining Nameless
2007-07-18T19:36:00.000+01:00
In college, I came across some budding Pascal programmers with
hundreds of lines something like this:
    gotoxy(12,13);
    write('|');
    gotoxy(12,14);
    write('|');

That's right, they were hand-coding menu systems in ASCII.  I actually
didn't spot this as being such a great problem at the time, because I
couldn't see past how ridiculous it was not to make a procedure
called, say, printxy, to do the above.
   printxy(12,13,'|');
   printxy(12,14,'|');

I got a couple of these students to do that, but they didn't really
know why.  After all, their editor had copy and paste, and everyone
else was doing the same as them.  Probably their teacher was, too,
but I never looked into that.

What happened when they realised that all their '|'s were off by one?
They edited each line of code.

Some of those students are now successful (in that they make money) IT
professionals (in that they make money), so I can only assume that
they learned something in the intervening years.

The rest of education, in the meantime, has changed a little.
Students now come along with the word 'reuse' on their tongues.  It's
a magical thing, and they joke that it means copying and pasting, but
they know it doesn't.  They can't do it though - the novices anyway.
They still copy and paste, like those of my time, sometimes guiltily.
Mainly they don't actually understand the alternatives.  Even more
experienced programmers will struggle to avoid copying and pasting in
some situations, and introduce false abstractions to do that.
Languages are partially judged on how much repetition they force you
into.

Going back to the Pascal example before - printxy wasn't a
particularly bad name, but imagine that you're a bit of a weak
programmer, and you have a small friend group who usually help each
other, the infinite monkeys' approach to assignments.  Your friends
have never heard of printxy before, and look at your code like it was
written by a Martian.  You don't want that, so you'll keep your code
as it is, despite what that weird Ricky suggested at lunchtime.

At this point, the text editor isn't really very useful.  You cannot
keep your friend group and me happy at the same time.  You could
probably write an editor macro that let you type printxy(x,y,'['); and
converted it into the appropriate code, but it would be a one-way
transformation.

What you'd like, because you want to get help from me and from your
peer group (infinite monkeys plus a programmer might equal higher
marks!), is to be able to maintain two views of the same code.

That is, you'd like to type printxy(x,y,'[') and have it converted
into a gotoxy and write call, but have the editor still know that it's
a printxy call, so that you can flick a switch and Ricky can see 'his'
version.

This never happened, because the editor didn't support it.  The only
students who I spoke to who ended up using printxy were smart enough
that they didn't depend on a peer group.  Except Dave, who tried to
depend on me instead.

Suppose that printxy was a particularly bad name, or the
implementation of it was a bit more intricate (this is perfectly
possible - it would make sense to reset the cursor to 0,0 after
writing each bit of text, because on DOS screens the cursor is usually
visible - having it flick around the screen is particularly
distracting).

It'd be good if, whenever people wrote similar code (either through
finger-typing or copy and paste), their environment picked up on it
and made it into an abstraction.  E.g.:
   gotoxy(12,13);
   write('[');
   cursor(0,0);
   gotoxy(12,14);
   wr..

At that point, the environment could start autocompleting the write
and cursor calls, and if the user started to type gotoxy again after
the next write and cursor calls, the environment could complete the
whole three-line form, placing the cursor in the right places to edit
the parts that were different between the first and second times.

Further, the environment could fold the blocks of code, just showing
the first one and then the differences:
   gotoxy(12,13);
   write('[');
   cursor(0,0);
   ..gotoxy(12,14)..
   ..gotoxy(12,15)..

Then if the user starts editing one of the forms, the others should be
edited in parallel, unless the user makes a specific gesture not to,
or manually 'ungroups' a form.  Groups could be shown via highlights
in the margin, just as they are in some environments for an individual
method, etc.

This would allow new programmers to start seeing abstractions appear
before they even know what the word abstraction means, and more
importantly, without even having to give a name to it.  printxy would
be something they applied later.

The missing half-dimension of code

In most programming environments, you're restricted to what I'll call
one and a half dimensions.  The vertical scrollbar is fine, most
people will happily scroll a couple of screens down, but never
sideways.  In most languages, 'arrow-like' code is discouraged as
being hard to read.  It's a bit like humans wandering around in 2
dimensions, not really being able to wander freely in the air.  Code
is the same - take it away from the top-level unit (function, class,
method), etc., and you're just waiting for it to collapse back down.

The more nested the code is, the harder it is to understand, because
you need to take the more polluted namespace into account.  If you
declare x outside a form, declare x1 inside it, then sometimes forget
to use x1, you'll be using the wrong value.  It's only because there
is no independent namespace.

Programming environments could help a little with this by warning you
when you use something from the surrounding namespace, but sometimes
you mean to.  They could use a different colour instead of warning
you.  I think it's worth looking at why arrow-like code happens, to
find a solution.

I think it's because there's nowhere to store unnamed snippets of
code.  Imagine a programming environment, where, when you typed a
nested form, such as a lambda expression (or anonymous class, or
anonymous function), the environment made it appear dislocated from
the code that uses it, possibly connected via dotted lines, etc.  Any
variable that is used from the surrounding namespace can be shown
specially, or as some kind of parameter.  You could drag the snippet
to another block of code, from where it could be used again.  Any
variables that the snippet needs that the first scope provided, can be
either defaulted or new links can be made.  This wouldn't even need
the snippet to have its own names for variables.

Past the user interface layer, Lisp macros with generated (and hidden)
names could easily be used.

You may be wondering why ordinary methods/functions as in most
languages can't be used - they can, but not in all the same ways.  For
example:
for (int a=0;a<100;a++)
{
   if (x[a]<50)
      return;
   print("here");
   if (y[a]<25)
      return;
}

Here the if..return is the repeating part - there's no way of
encapsulating that in a method/function, without changing the code
a lot.

I'm not saying that naming abstractions is bad, but that being forced
to name them is bad.  I plan to come up with an environment for
editing code in this way.  It probably will be based on an existing
language, using generated names that the developer doesn't see.

Slava Pestov pointed me at subtext as an existing implementation of something like what I want.  It looks interesting, but I don't think there's a huge gap between it and what Excel would be if you made cells part of a tree instead of a grid.



On how the lack of macros damages Java code
2007-06-29T17:31:00.000+01:00
Update - commenters pointed out an obvious mistake - I should have kept the sample code here the same as my real code.  I've updated the code samples.

Some time ago I asked why unit tests use assertions instead of returning a boolean.  I didn't get any particularly convincing answers, so I continued returning a boolean, in my home-grown test framework, that's so small it hardly deserves the name 'framework'.

One obvious benefit from using exceptions is that you get the name of the test method and class in the exception output.  In my framework, I had to explicitly supply that.  Each test object had a getName().  return "TestUninstallingDrivers".  Ok, now I have duplication:

public static final UnitTest testUninstallingDrivers=
    new UnitTest()
{
    public boolean test()
    {
        stuff..
        return !card.hasDrivers();
    }

    public String getName()
    {
        return "testUninstallingDrivers";
    }
};


The usual "Java programmer" approach would be to just go back to exceptions, then you'd be able to get your IDE to show you the test method by clicking on the stack trace.  However, there were some reasons not to go back to exceptions.  I'd instead like to generate the getName() implementation based on the field name.  I believe at least 2 out of the 3 major IDEs can help me with some kind of template like this:

public static final UnitTest $x=new UnitTest()
{
    public boolean test()
    {
        //Auto-generated method stub.  To stop this
       //annoying message, go through some dialogs
       //that you never really wanted to anyway.
        return false;
    }

    public String getName()
    {
        return "$x";
    }
}

So, that's all well and good, until I start refactoring, because the IDEs do not carry on linking the two expansions of $x even after the template has been applied.  When I first heard someone rave about IDEA's Live Templates, I assumed they would provide such linking - they do not.

Plus, even if they did, you'd still see the generated code, which isn't really that useful.  The only part of the above code that's relevant to me is the name of the class and the code inside test.  That is, I'd be happy with:

unitTest(testUninstallingDrivers,
{
    stuff..;
    return !card.hasDrivers();
})
..which would generate the above usual Java code.  Of course, should I want to see the expanded template, I could just right-click on the template name or something.

So what I'd really like from IDEA's Live Templates is for the template's name to stay in my source code, and to be expanded for compilation.  But then, if it's staying in my source code, I can't compile without IDEA, or even with IDEA on another computer (unless I successfully manage my project settings).  It would be better if the template could stay in my source code too.  The source for the template.  E.g.:

template unitTest(name,body)
{
    public static final UnitTest <name>=new UnitTest()
    {
       public boolean test()
       {
           <body>
       }

       public String getName()
       {
           return "<name>";
       }
    }
};

This would need a preprocessor before the usual Java compiler.  There are some difficulties, like the <body> expansion above generating two sets of braces, but they can be resolved (a syntax like <@body> to splice in the body).

In case there's anyone out there who hasn't recognised this yet, I'm really talking about Lisp macros, and not even a significantly complex application of them.  I don't see any reason why modern programmers don't use a decent macro system, pretending to be happy with 'Live' Templates, and refactoring everything at a higher level (such as changing the unit test framework to use exceptions) instead of working with the actual source code they like.

For a Lispy implementation of a unit test framework (in 26 lines of Lisp!), see this chapter from Practical Common Lisp.


Book Review - Release It! Design and Deploy Production-Ready Software
2007-06-25T13:50:00.000+01:00
Release It! Design and Deploy Production-Ready Software, by Michael Nygard (Pragmatic Programmers)

You know the drill.  Here's a sample case of how something went wrong in production.  The author needs to pad out the book, let's make the case more convoluted.  Maybe include some code.  A book written in dry technical English, painful to write and worse to read.  That's what I expected when I received this book, except for the 'Pragmatic Bookshelf' logo on the front.  I was pleasantly surprised; none of the above apply.

The book took me quite some time to get through, not because it's padded out, but because it's dense.  Despite being written in plain (and sometimes overly American) English, a lot of information is presented, and very readably.  The author has clearly spent a lot of effort in making sure that the book doesn't patronise, yet does explain its terms clearly.

Even though I'm not exactly the target market for this book, as it is aimed at enterprise developers (the author defines enterprise as systems that cost money whenever they go down), I enjoyed it, and I can apply quite a lot of it to my lowly non-enterprise project.  There are a lot of references to Java in the book, but I don't think that should deter those who don't use Java, as the same points can be applied to other technologies.  Rarely does it go deeply into Java-specific problems.

Stop Waffling, And Tell Me What The Book's About

Stability, capacity and operations.  The stability part covers ways of making sure that errors in software don't spiral out of control.  Even where there are failover subsystems in place, and everything seems isolated from everything else, there may be a hidden route of chaos, such that when one system goes down in a certain way, all the rest do.  The 'Circuit Breaker' pattern will help to stop that.

Capacity has a lot more human factors in it - how many sessions can your server support?  Can you make sessions shorter in the event of a surge, saving RAM but annoying users?  Does your testing environment match the production environment closely enough?  Can you get away with static content for those users who are browsing, not buying?  Do you really need pretty-printed HTML, adding 50K to data transfers?

Operations is about ensuring that the administrators have an easy time of it, that you can have meaningful logs that are useful, that you can inspect (and even tinker with) a running application, throttling backups so that capacity isn't affected, and generally making sure that you can.

A lot of the book may seem intuitive, but it's good to see your intuitions formalised, and taken a few steps further.  Some programmers reading this book will be screaming out things like "use Erlang and this problem disappears", etc., and they'll often be right, but it's useful to understand a problem, even if your environment makes it impossible.  Further, many problems will out in other ways even if an environment prevents the major cause.  So I'd recommend this book to any programmer - learn from Michael's mistakes, and the mistakes of those around him, then hopefully your own mistakes won't be so costly.  I'm glad that I keep the review copy, as I will be looking back through it again.  Plus, I read some of it in the bath, so it's got a few crinkled pages - I don't fancy posting it back to anyone!

The stories about what happened in production are always real, though the names are changed, and the author builds up a real sense of suspense that I've not seen often in technical books.  The problems that he and his colleagues find are often hilarious, but the lessons drawn from them are important.  I've often found, as I said at the start of this review, that stories in technical books are very dry, very boring and just padded out, but the author has enough writing skill, and great raw material, so that hasn't happened.

To give you a flavour of how the book is written, here are a few randomish paragraphs:

Avoid fiddling

Human intervention leads to problems.  Eliminate the need for recurring human intervention.  Your system should run at least for a typical deployment cycle without manual disk cleanups or nightly restarts.
 It seems like common sense, but there are plenty of systems that require routine restarts.  Common sense just isn't that common, it seems.

For example, I've seen badly configured proxy servers start re-requesting a user's last URL over and over again.  I was able to identify the user's session by its cookie and then trace the session back to the registered customer.  Logs showed that the user was legitimate.  For some reason, fifteen minutes after the user's last request, the request started reappearing in the logs.  At first, these requests were coming in every thirty seconds.  They kept accelerating, though.  Ten minutes later, we were getting four or five requests every second.  These requests had the user's identifying cookie but not his session cookie.  So, each request was creating a new session.  It strongly resembled a DDoS attack except that it came from one particular proxy server on one Navy base.
Would you have thought of that in advance?  I wouldn't, I'd have just blocked the user automatically, assuming malice.

Amazon ran into trouble with the Xbox 360, too.  In November 2006, Amazon decided to offer 1,000 units for just $100.  News of the offer spread far and wide.  Not surprisingly, the 1,000 units sold within five minutes.  Unfortunately, nothing else sold during that time, because millions of visitors hammered on their Reload buttons, trying to load the special offer page and score a huge discount on the hot console.
The context here is attacks of self-denial, where systems get attacked by themselves, by a special offer attracting an incredible surge of page requests.

Again, I'd recommend this to any programmer, and to any administrator who deals with bespoke solutions.  Perhaps if all you do as an administrator is install/configure applications for which a support contract exists, and you can afford the time between a problem and the support engineers appearing, then you won't benefit from this book.  But then surely you're only 5 minutes away from being replaced by a shell script..

The one missing part, for me, is a discussion of how to go about designing a system from scratch - all the case studies describe existing systems, and often only a small part of them - I'd like some suggestions on how to write a system that will be able to scale easily - and how to avoid writing systems that won't.


Missing Refactor - Convert to Anonymous Class
2007-06-21T16:10:00.000+01:00
I'm still looking through my source for stray uses of 'private', following on from my recent blog about removing private, favouring other forms of encapsulation.

One way I like to remove private is by converting named implementations of interfaces into anonymous ones.  Here's some real code that I want to change:


public final class EthernetCableIcon implements
  ActionListener
{
  private final MouseInputListener listener;
  private final GUIContext context;
  private final JToggleButton button;

  private EthernetCableIcon(final GUIContext context)
  {
    button=new JToggleButton
      ("Ethernet Cable",EthernetCableHandler.icon);

    this.context=context;
    listener=new MouseInputListener(context,button);

    button.setVerticalAlignment(SwingConstants.CENTER);
    button.setVerticalTextPosition(SwingConstants.BOTTOM);

    button.setHorizontalTextPosition
      (SwingConstants.CENTER);

    button.addActionListener(this);

    button.setToolTipText("Click on this to draw an "+
      "Ethernet Cable, then drag on the display to "+
      "make one appear");
  }

  public static JToggleButton newButton
    (final GUIContext context)
  {
    return new EthernetCableIcon(context).button;
  }

  /**
   * Replaces the NetworkView's mouse listeners with its 
   * own, so that the EthernetCable can be dragged
   * without causing other problems.
  */
  public void actionPerformed(final ActionEvent event)
  {
    final Component view=context.getNetworkContext()
      .networkView;

    if (button.isSelected())
    {
      context.getNetworkContext().toggleListeners.off();

      view.addMouseListener(listener);
      view.addMouseMotionListener(listener);

      view.setCursor(new Cursor(Cursor.HAND_CURSOR));
    }
    else
    {
      view.removeMouseListener(listener);
      view.removeMouseMotionListener(listener);

      context.getNetworkContext().toggleListeners.on();
      view.setCursor(new Cursor(Cursor.DEFAULT_CURSOR));
    }
  }
}


My IDE, IntelliJ IDEA, has lots of refactorings, including from anonymous to nested, from nested to inner, moving classes, etc., but has nothing for making this an anonymous class.  It's used precisely once, from the static method newButton.

I've refactored it by hand, which isn't too bad, but it does make me wonder why I have to.  Eclipse never supported this refactor either.

Post-refactor:

public final class EthernetCableIcon
{
  public static JToggleButton newButton
    (final GUIContext context)
  {
    final JToggleButton button=new JToggleButton
      ("Ethernet Cable", EthernetCableHandler.icon);

    final MouseInputListener listener=new
      MouseInputListener(context, button);

    button.setVerticalAlignment(SwingConstants.CENTER);
    button.setVerticalTextPosition(SwingConstants.BOTTOM);

    button.setHorizontalTextPosition
      (SwingConstants.CENTER);

    button.addActionListener(new ActionListener()
    {
      public void actionPerformed(final ActionEvent e)
      {
        final Component view=context.getNetworkContext()
          .networkView;

        if (button.isSelected())
        {
          context.getNetworkContext()
            .toggleListeners.off();
          
          view.addMouseListener(listener);
          view.addMouseMotionListener(listener);

          view.setCursor(new Cursor(Cursor.HAND_CURSOR));
        }
        else
        {
          view.removeMouseListener(listener);
          view.removeMouseMotionListener(listener);

          context.getNetworkContext().toggleListeners.on();

          view.setCursor
            (new Cursor(Cursor.DEFAULT_CURSOR));
        }
      }
    });

    button.setToolTipText("Click on this to draw an "+
      "Ethernet Cable, then drag on the display to "+
      "make one appear");
    
    return button;
  }
}


Whether it's more readable now probably depends on the reader, but it certainly doesn't need private and hasn't lost anything in the encapsulation.  I find it slightly more comprehensible anyway.


Taking private out of code without losing encapsulation
2007-06-19T00:53:00.001+01:00
When I started programming in Java, I took private and public as fairly necessary access levels.  It's only in the last couple of years that I've realised that you can achieve better encapsulation than private - by making the variable disappear, or at least not have a name in the relevant scope.

As an exercise, I thought I'd take a look at my codebase and see how it is impacted if I get rid of private.  But of course, I didn't want to expose implementation details either, so I did it with the power of thought, a scarce resource.

Low-hanging fruit first:  Private constructors.  Sometimes these are used to prevent instantiation of a utility class.  IDEA actually warns me if I instantiate a utility class, so I don't need the constructor.  It's a no-brainer to delete unused code.

Ok, now private fields with get/set methods.  There are two kinds - the first is the simple problem of having a private field for no reason - the get and set methods just pass through to the variable with no checking.  Make the variable public and get rid of the methods.  IDEA helps me with this - Ctrl-Alt-N inlines a method.

Private fields with more interesting get/set methods.  In my code, so far, I don't have any interesting get methods, just some set methods that do some work after the value has been set.  This isn't an original idea:

Make a Property interface, using generics (no evil casting) with get, set and addPropertyListener.  Provide an implementation as a static factory method.  Now change the private field to be a Property, adding the logic from the set method to the Property via a PropertyListener.  Fix the get and set method so that they delegate to the Property, make sure everything compiles, then inline the get and set methods.  If your get and set methods are more interesting than mine, then perhaps you'll need a more advanced Property type, e.g., addPrecondition, addPostcondition, etc.

Now for the fruit at the top of the tree.  I use a Map to allocate IDs for Cables (it's a network simulator, and the IDs are useful for the event log).  I don't want to expose the Map, but I do want to expose the operation I have on it - cableIDFor(Cable cable).  How can I do that without using private?  The Map has to persist between invocations, so I can't create it inside cableIDFor.

I'll make a field, called cableIDFor.  It will hold an object that has one method.  That method will take in a Cable and return an Integer.  In other words, it's a Function<Cable,Integer>.

public final Function>Cable,Integer> cableIDFor=
    new Function<Cable,Integer>()
{
    final Map<Integer,Cable> cableIDs=hashMap();
 
    public Integer invoke(Cable cable)
    {
        for (final Map.Entry<Integer, Cable> entry: 
                cableIDs.entrySet())
            if (Caster.equalT(entry.getValue(), cable))
                return entry.getKey();

        for (int a=0;;a++)
            if (!cableIDs.containsKey(a))
            {
                cableIDs.put(a, cable);
                return a;
            }
    }
}


You call that like so: cableIDFor.invoke(cable), and of course now you get it as a Function for free, so if you want to pass it to some other method, you can.  Oh, and yes, it's not a pure function, but I know that and treat it accordingly.

Thankfully I only call cableIDFor in one place, so I don't mind that IDEA doesn't provide an automated refactoring for this.  Perhaps the structural search and replace will do it.

One more, a small one.  I have a private field in a class that implements IncomingPacketListener - it's a StringBuffer, used for unit testing only.  Easy solution, make the class anonymous and the name of the StringBuffer variable disappears from relevance.

I haven't lost any encapsulation in any of this, so I'm quite pleased.  This all makes me realise that it isn't a problem that Common Lisp's OO support doesn't include private.  That might actually be a good thing.

I expect a lot of people will be thinking "No!  You're losing flexibility", but that's just not true.  None of this code is external - I'm not actually impacting anything outside my own codebase.  Externally-facing code gets put in separate projects that I normally leave closed so that I don't mess with it by accident.  So, suppose that I decide that one of my public fields does need some validation - I can use IDEA's refactoring to encapsulate it, and then convert it into a Property or leave it as a private field with get/set.  I'm not losing anything.

It's not less readable either.  Because I use bits of functional programming in my code, it's actually fairly useful to be able to refer to a 'method' without executing it, which I can do if I convert it to a field that holds a Function.  I end up with less anonymous classes this way.  Likewise, even for 'normal' programmers, it's useful to be able to refer to a property instead of grabbing its value, for instance, for binding to a graphical object, etc.



Improving Your Visitors, and a Request For More Autoboxing in Java 7
2007-06-13T01:48:00.000+01:00
Prior to Java 5, there was one really common way of implementing the
visitor pattern.  I'll use my network simulator as an example,
defining a visitor that can visit any kind of network component
(a card, a cable, a hub or a computer):
public interface Visitor
{
    void visit(Card card);
    void visit(Hub hub);
    void visit(Cable cable);
    void visit(Computer computer);
}

and an example accept method:
public void accept(Visitor visitor)
{
    visitor.visit(this);
}

For those not initiated with how this works, here's a typical
example:
public static void drawAComponent
       (NetworkComponent component,final Graphics graphics)
{
    component.accept(new Visitor()
    {
        public void visit(Card card)
        {
            graphics.drawRect(10,10,20,20);
        }

        public void visit(Computer computer)...
    });
}

It works quite well, and you can adapt it to make it work over a
collection.  In IPSim there is a tree of components (each component
knows its children, and there is a collection of top-level
components).  So you can do:
network.visitAll(new Visitor(){etc.});

If we want to inspect the object (get a value back from it) rather
than do something, then we end up with code like this:
public static Dimension getSize(NetworkComponent component)
{
    Dimension[] result={null};

    component.accept(new Visitor()
    {
        public void visit(Card card)
        {
            result[0]=new Dimension(50,60);
        }
        etc.
    });

    return result[0];
}

Clearly it would be useful if the visitor could return something
instead of having to set some value.  We could make the Visitor's
methods return an int, but then we'd have to write it again if we
later needed another type.  There are two solutions to this:

1.  Make the methods return Object.  This obviously results in
casting, which we all know is for magicians.  Let's move on to
Java 5.

2.  Make the methods return T, where T is a type parameter on
Visitor.  Ok, easy:
public interface Visitor<T>
{
    T visit(Card card);
    T visit(Cable cable);
    T visit(Hub hub);
    T visit(Computer computer);
}

and the accept method:
public T accept(Visitor<T> visitor)
{
    return visitor.visit(this);
}

The only problem with that is that T is undeclared in the accept
method.  Don't make the mistake of declaring it on the class (class
Card<T> implements NetworkComponent<t>), because then you'll only be
able to use one type of Visitor with each instance you create.
Declare it on the method:
public <T> T accept(Visitor<T> visitor)
{
    return visitor.visit(this);
}

Now you've got something more useful, the above clunky code becomes:
public static Dimension getSize(NetworkComponent component)
{
    return component.accept(new Visitor<Dimension>()
    {
        public Dimension visit(Card card)
        {
            return new Dimension(50,60);
        }
        etc.
    });
}

This becomes much more interesting when you apply it across a
collection.  Instead of writing code that iterates over all the
components and does something, we can write code that generates a
result.  For example, suppose I want to know if any of my components
are a crossover cable.  I want to visit all the components, and
return true if any of them are a crossover cable.  I could store a
list of booleans, to process later, but that would be wasteful and
indirect.  What I want to do is apply the || operator between each
pair of results.
public static boolean areAnyCrossover(Network network)
{
    return network.visitAll(new Visitor<Boolean>()
    {
        public Boolean visit(Cable cable)
        {
            return cable.isCrossover();
        }

        public Boolean visit(Computer computer)
        {
            return false;
        }

        public Boolean visit(Card card)
        {
            return false;
        }

        public Boolean visit(Hub hub)
        {
            return false;
        }
    },new Combinator<Boolean,Boolean>()
    {
        public Boolean combine(Boolean one,Boolean two)
        {
            return one||two;
        }
    });
}

Whew.  This is an implementation (or at least behind the scenes
there is one) of the 'reduce' algorithm, plus a bit of mapping going
on first.  It's very flexible.  There's quite a bit of boilerplate
above though, and as the number of component types grows, the amount
of boilerplate grows and grows.

The usual solution is to make an abstract class with empty (return
null;) methods for each visit method, which you subclass to
customise.  That works pretty well (remember the @Override
annotation!), but I've been avoiding subclassing for some time now,
so I immediately started to wonder whether there was another way to
do this.  I'm not a zealot, if I didn't find another way, I'd just
subclass.  I found another way..
public static boolean areAnyCrossover(Network network)
{
    return network.visitAll(new DefaultVisitor<Boolean>()
        .visitCards(new Function<Card,Boolean>()
        {
            public Boolean invoke(Card card)
            {
                return card.isCrossover();
            }
        })
        .visitAnythingElse(new Function<NetworkComponent,
            Boolean>()
        {
            public Boolean invoke(NetworkComponent any)
            {
                return false;
            }
        }),new Combinator<Boolean,Boolean>()
        {
            public Boolean combine
                (Boolean one,Boolean two)
            {
                return one||two;
            }
        }
    });
}

Ok, it doesn't look brilliant, but it does look a little like
functional programming.  Let's make these anonymous classes look
more like functions:
public static boolean areAnyCrossover(Network network)
{
    return network.visitAll(new DefaultVisitor<Boolean>()
        .visitCards({Card card => card.isCrossover()})
        .visitAnythingElse(
            {NetworkComponent component => false}),
        {Boolean one,Boolean two => one||two});
}

One more improvement would be if it were possible to get at || as a
function.  Let's imagine that #|| works for this.  Let's also
imagine that closures allowed you to elide unused parameters, that
type inference were supported, and that method references were
supported (!):
return network.visitAll(defaultVisitor
    .visitCards(Card#isCrossover())
    .visitAnythingElse({ => false}),#||);

Interestingly, it still looks like Java, and it doesn't get bloated
whenever you add a type.  I don't suppose for an instant that Java 7
will go this far.  For an example of the <T> visitor pattern in
current Java, see the source code for javac.  It's littered with
them, and it's quite nice to work with, but at least for now,
remember your @Override..

And back to normal Java, suppose you implement Visitor<T>, but then
come across those cases where you don't want to return something.
void doSomething(NetworkComponent component)
{
    component.accept(new Visitor<Void>()
    {
        public Void visit(Card card)
        {
            whatever.
            return null;
        }
        etc.
    };
}

I'd like to be able to use my Visitor type with a <void> type
parameter, but I cannot.  Please, if you got this far, and are Neal
Gafter or someone else relevant, consider allowing primitive generic
type parameters.  As far as I can see, this can all be done at the
compiler level, making a Visitor<void> appear as a Visitor<Void> in
bytecode.  At the moment, I often implement two Visitor types for
each hierarchy - one returning void, and one T.  Perhaps that's
overkill, but it reminds me that it's a workaround.


Optional Checked Exceptions - Spitting at the compiler
2007-05-28T15:01:00.000+01:00
In further anecdotal evidence that Gilad was right, Neal Gafter has blogged with a suggestion that checked exceptions could be removed from Java.  He's right, they could.  As far as I can see, all existing code would still compile, and what's more, it would actually work with existing bytecode with no problem, as the runtime has no problem with throwing undeclared checked exceptions.  I've thought for a while that the problem with checked exceptions is that the API that throws them doesn't know whether you will care to deal with them.

First, I'll mention some ways in which closures can help with checked exceptions, then a mechanism for optional checked exceptions:

Reader.close() can throw an IOException, but we don't let it propagate.  Closures to the rescue:

suppressing(IOException.class){ reader.close();}

Making Swing use the native look and feel requires catching about 5 exceptions, but we don't care about any of them:

loggingAndIgnoring(Exception.class){ UIManager.setLookAndFeel..}

Or even better:

whenExceptionsHappen({e => e.printStackTrace();}){ UIManager.setLookAndFeel...}

Suppose that we do want to propagate the exception instead, but we don't want to wrap it in a RuntimeException:

public void readsFromAFile() spits IOException

I'm not seriously suggesting 'spits' as a keyword, feel free to think of your own (I also like 'farts' and 'vomits', but that's because I've had a very busy weekend), but in this context, a method that spits IOException is declaring that it can throw an IOException, but that callers of it do not have to catch the IOException.  That way everyone can be reasonably happy.  API designers can still declare their exceptions using throws, guaranteeing that the client programmer has to think about the exception, but the client programmer can prevent the exception from leaking through their codebase, by using spits.

There is one problem - catching a checked exception from code that does not throw that exception is currently a compile-time error.  Spat exceptions would make that compile-time error need to disappear too.

Challenge: think of a better keyword than 'spits' for this.



You don't need TDD, you need a REPL
2007-05-24T20:49:00.000+01:00
A walkthrough of how TDD can be badly applied to promote bugs, and an alternative.  I'm imagining that my project is in Lisp, for the code samples, and so that I have a decent REPL (read-eval-print-loop, a place where you type code and see what it evaluates to).

The situation described here is only partially based on reality, but you can judge how likely it is for yourself.

4:40pm  Let's consider some code that we completely understand - adding two 2D points together.

TDD says that I should write a test.. ok:

(assert (equal (point+ (point 3 2) (point 4 6)) (point 7 8)))

Then it says that I should make the test fail:
(defun point+ (first second)
  (point 1 2))

Ok, that fails.  Now I should make it pass, and I should do the minimum required for that.
(defun point+ (first second)
  (point 7 8))

4:45 It passes.  I get distracted by the telephone.

4:55 I look back, and I see that my test passes.  Ok, I'm done, let's move on.  None of my other tests use point+, because I try not to write dependent tests.  So in this case, I will find the above bug in integration tests.  But those involve files, and are a bit slower, so I don't bother running those quite so often.

5:00 My boss comes in and asks for a build for today's work, so that he can have a play around with it on his laptop on the train, which sets off in 10 minutes.  I disable the integration tests temporarily so that I can just build it and give it to him, then dump a jar file to his USB pen drive.

5:10 He's on the train.

5:20 I'm distracted by the phone again.  "All the drag and drop seems to have suddenly started going to the same place".  Ok, I say, I'll fix it right now.  So I re-enable all the integration tests, and yes, there's a failure.  I fix it.

5:30 I email another build to my boss, but he's not able to get it until he's at home.  While he was on the train, he gave a broken demo to a potential customer.

You can find a few things wrong with my environment, such as that I don't use continuous integration, I actually answer phone calls while coding, and that I don't keep daily builds around for those times that my boss asks for a build.  However, those are fairly superficial.  The big problem that following TDD caused me was that I felt good about some really bad code.

Without a test, I would have had suspicion for the code, but the test gave me confidence.

That's not to say that testing is a bad thing, but that TDD gave me confidence in broken code.

So let's implement the same thing again, but with a more 'distraction-proof' testing technique:

4:40 Write the function:
(defun point+ (first second)
  (point
    (+ (x first) (x second))
    (+ (y first) (y second))))

4:45 Check that it works for a few values in the REPL:

(point+ (point 1 2) (point 3 4))
> 4 6
(point+ (point 1 4) (point 5 10))
> 6 14

ok, seems fine.

4:50 Get distracted.

4:55 Return to the code, there's no test yet, so I don't trust point+.  Oh, but wait, look at the REPL, it looks like it works.  I run the tests in the REPL again, then write a test case.

(assert (equal (point+ (point 3 4) (point 5 6)) (point 8 10)))

The test case passes, and this time I haven't made incomplete code pass tests.  My boss, on the train, is happy, and so is the potential customer he demoed the software to.  

A REPL helps me to separate code that's part of my project from code that is just being thought about.  It encourages experimentation, and hence thought, but doesn't get in the way should you get distracted and come back.  It also makes you less likely to put broken code into the codebase, because it doesn't make you write experimental code in the production code.


I'm a Twit Too
2007-05-22T17:56:00.000+01:00
After diligently avoiding finding out what twitter was, although it's hard to ignore it if you read technical blogs, two of my friends started using it and I joined them.



It's amusing to see it go down every so often, and to think that Ruby is the culprit.  Still, it goes down more gracefully than javalobby, and it's only lost one post of mine so far, which is a lot better than javaranch, and many blog sites.

I'm going to keep mine technical, mainly on Java and Lisp at the moment, mainly as ideas for blog posts or projects that I'll probably never get around to.  So if you'd like to steal my ideas, do whatever it is that twitter users do, or add the feed to your RSS reader.

twitter ID - ricky_clarkson - http://twitter.com/ricky_clarkson


Using just the JDK and 55 lines, I can find empty 'then' statements in Java code.
2007-05-17T21:04:00.000+01:00
A few hours ago I noticed some comments on Joe Darcy's blog post about annotation processors.

I originally lost interest in annotations very quickly when I realised that annotation processors didn't have access to the full AST.  I fiddled around with ANTLR a little, with CheckStyle, PMD and Findbugs' APIs to see what I could do.  Not a lot, as it turned out.  ANTLR looked promising, but as I recall, the language file for Java 5 was a bit flaky at the time.  The others didn't provide me with complete types, so I couldn't detect things like code that tried to find a StringBuffer in a List of Strings.  I gave up for a while, went off and learned to be productive in Haskell, and Lisp, and stuck my head into closures, which I'm very optimistic about.

I followed some links (about 6 deep) from Joe's blog, and came across the compiler API and compiler tree APIs.  I have used it before, when I got some novice students to write their own compiling Java editor (that was fun), but I hadn't realised that you could get access to the actual compiler types - the standard API is rather thin.  All you need to do is cast the task returned by the Java compiler, into a com.sun.source.util.JavacTask, and you're away.

I'm already mildly familiar with the compiler, having fiddled with its source a month or two back.  I was trying to add a language feature that Stephen Colebourne, Stefan Schulz and I had been discussing, but we dropped the idea before I finished (well, before I really got started).  Today I decided to attack a problem that used to catch me out when reading novices' code - an if statement with a semi-colon on the end, which just means 'if this, do nothing' and it executes the next statement regardless of the condition.

Javac uses the visitor pattern incredibly heavily, to a scale I've never seen anywhere else.  I like it, but it caught me out temporarily.  I don't really write subclasses anymore, I always use an alternative where practical.  So I subclassed TreeScanner, which has nice 'return null;' implementations of every method, just ready for me to override.  Which I didn't.  Though I thought I was.  The @Override annotation saved me when I realised..

Because I have my IDE set to Java version 5, and I wanted to run this code against version 6, I was just editing in vim and running from bash in Cygwin.  No warnings about unused methods there then..

The short-term solution is to make sure I always remember @Override.  The long-term, unlikely solution, is to persuade the javac project lead, sometime after Java 7, to accept my patch that makes these visitors more 'functional', so that instead of overriding, you give them a closure to invoke.  The other advantage there is that you would be able to provide real booleans, rather than Booleans, etc.

There was only one more gotcha.  Not only do I not subclass, I also don't use null very often, so I'd forgotten how to deal with boolean operators and null (I had to use Boolean, not boolean, thanks to generics).  See if you can reduce my bestOf method into a chain of ternary operator calls.  I couldn't do that without getting a NPE.  I might investigate later.

There were no more gotchas.  It was actually really easy, 30 minutes end to end.  I included an 'empty' if statement in the source itself for testing.  I think my next task should be to find places where @Override should be..

After all that fanfare, the code wouldn't fit in my blog's layout particularly well, so I'm afraid you'll have to click here to see it.



Really Own Your Code
2007-04-09T21:45:00.000+01:00
Walking to the bus stop, with a coffee burning one hand and my breakfast the other, I noticed how poor my area looks; an urban area where many students and young professionals rent houses.  The tenants aren't poor, the owners neither, but the area itself is impoverished, because the owners don't care and the tenants don't care.  I got on the bus, and dropped half my coffee on the bus floor as it went over a speed bump, because I was holding the coffee so as not to burn my hand too much.  Ah, well, glad it wasn't my bus.

It struck me right there how little tenants look after the property they rent.  Some local teenagers smashed a bottle of beer outside my house, and I didn't clear it up.  I didn't ask the local authorities to do so either.  I just looked at it with disdain whenever I passed it, and resented paying my local taxes even more.

In my job as a coder, on a 1.5-man project, I am a tenant.  I'm actually a good tenant, in that I look after the codebase pretty well, but I'm still a tenant in the most important, and crippling, way.  I don't own my code.

This isn't the same as responsibility for my code; I have that.  It's my code, but it's not mine.  I can't walk away from my employer and reuse the code elsewhere.  That's not completely true; for a couple of libraries I wrote for the project my boss has given me licence to release them for free.  For those libraries, I actually find myself reserving my best coding practices, because I know that other people will need to read the code.  Not just in some abstract "sometime after I leave my job" way, but in a "they already do read it" way.

I don't bother to create separate libraries for code that I can't release separately.  In other words, I don't decouple as much as I could, because the practical benefits are too indirect.

Realistically, I know that my project will die when I leave.  That's not to say that I won't be very gracious in documenting it.  My boss doesn't have the skills to maintain it well, and he's uncertain as to how to make more money from it, so he probably won't want to release the whole thing for free until it suffers from bitrot (which will probably be whenever IPv6 becomes dominant).

The fact that I don't own the code means that my goals and the project's goals are slightly at odds with each other.

There's an IP stack in the project, a serialisation library, clever handling of scroll bars when stuff is dragged around the display, but only our 200 students (per year) are benefitting from those.  The funding comes from a nationwide education body, but they don't have rights over the code either.

In that respect, it's a waste of money, because other establishments can't easily customise it for their requirements, and largely they don't know about it.  In fact, even within our department, another member of staff wanted to use the software, and she couldn't, because it didn't have a particular feature.  I offered to implement it, but my boss declined because it wasn't part of what the funding was paying for.  If it was my code, I'd happily implement it in a separate branch, in my spare time, and I'd feel enthusiastic about it.

Now let's look at the most recent entry in Bugzilla for this project:

"A long term thing for a quiet Friday afternoon (if such a thing exists!)

In the same way you have files in the jar for customisation of "installation
number", "installed to" etc, I think there may be a long-term need to use a
similar mechanism to allow some customisation of the conformance rules (this
will be important if other people use IPSim, as there may be some things that I
want to give feedback on that other people may not care about"

We lost 150 users already by not doing this for the other member of staff.  Plus, I could only code in advance customisations that I can anticipate.  If I'm on holiday or busy coding a new feature that we already have funding for, we'll miss out on the opportunity.

For a small project like this one, developer motivation and the quality of the project would increase if the developer could reuse the work elsewhere.  That could mean open-sourcing it, as has been done for two small parts (a functional programming library and a Java layout manager), or it could mean an agreement not to use the code for commercial purposes for a fixed term after employment.

How about for a larger team?  I'd suggest to assign each source file to a single developer, who has responsibility over that file, and has actual ownership, so that he can reuse the file at some point, even now, in another company.  Then any changes made by any other developer to that file are automatically licenced so that the owner can use them.


Is Josh Bloch the biggest problem for closures?
2007-04-04T14:52:00.000+01:00
CORRECTION: Neal has since informed me that he, since the presentation that this post references, has learned that there is no such JCP rule.  I'll still keep this post, because it makes some useful points (and so do the comments).

Josh Bloch, with "crazy" Bob Lee and Doug Lea, proposed their Concise Instance Creation Expression solution for closures for Java, last year.

They didn't address returning from the enclosing scope from within the closure.  Theirs is a trivial syntax change, probably fairly straightforward to implement but it wouldn't address simple cases such as finding the first line in a file that matches an expression.  If you're already familiar with why CICE is insufficient, feel free to skip past the code here.

Here's the psuedocode:

forEachLine ln in "/etc/passwd" do
   if ln contains "root"
      return ln


Here's the usual Java 5:


BufferedReader reader=null;
try
{
    reader=...;
    String line;
    while ((line=reader.readLine())!=null)
    {
        if (line.matches("root"))
            return line;
    }
}
finally
{
    if (reader!=null)
        try
        {
            reader.close();
        }
        catch (IOException exception)
        {
            log(exception); //but don't rethrow it
        }
}


So, of course, as the good programmers we are, we write an abstraction for that:


forEachLine("/etc/passwd",new LineProcessor()
{
    public void processLine(String line)
    {
        if (line.matches("root"))
            return line; //won't compile
    }
});


There's a problem there, the 'return line;' is returning from the run method, not the enclosing method.  Ok, we can solve that, let's make LineProcessor return a String, which is normally null unless we want to end the loop.  Then forEachLine can also return a String, which will be the return value of the method.


return forEachLine("/etc/passwd",new LineProcessor()
{
    public String processLine(String line)
    {
        return line.matches("root") ? line : null;
    }
});


This works, albeit with some strange semantics, which could be made more obvious by adding a class to hold the result, but that's not important for this post.

So forEachLine now has to know that the LineProcessor it runs might want to return early.  When callers of forEachLine don't want to return early, they still have to return null all the time.  It's inconvenient; to not punish callers that don't want to return early, we now need two versions of forEachLine, one that supports early return, and one that doesn't.  That's confusing.  The bad smell wafts up, making you either overload, or use names like "forEachLineWithAPossibleEarlyReturn".

What does CICE do for this?


return forEachLine("/etc/passwd",LineProcessor()
{
    return line.matches("root") ? line : null;
});


That's right.  It removes a bit of punctuation and the 'new' keyword.  Without underestimating that contribution, it doesn't actually help to make the end result resemble the psuedocode.  It would be hard to get back from this end result into the psuedocode, there's a loss in translation.  It isn't expressive.

The BGGA proposal allows this:


forEachLine(Line line: "/etc/passwd")
    if (line.matches("root"))
        return line;


Now just to remind you of the psuedocode again:


forEachLine ln in "/etc/passwd" do
   if ln contains "root"
      return ln


It doesn't need a special version of forEachLine for each use case.  It's generally better, more expressive, with one wart that I'll mention at the end.

So why does this matter?  Neal Gafter, the proposal lead, used to work for Sun, he's well respected, so he can easily make this into a JSR.  Or not.

Neal Gafter now works for Google.  All the work he's put into BGGA has been on his own time, it's not allowed to be his "20% project" (Google employees spend 20% of their paid time on ideas of their own).  He is not the JCP's contact for Google.  Surprise, surprise, Josh Bloch is, and Josh has his own more limited, less expressive, proposal, as I've demonstrated just now.

The JCP has a rule that disallows individuals from being on the JCP if their employer is on it.  I felt sorry for Neal at the end of his presentation on closures at Google.  Here's a quote from the last few minutes (emphasis from original):

From the audience: "Is someone planning on opening up a JSR on this?"

Neal: "There's a long answer and, I'm afraid, a longer answer.  I was planning on writing a JSR for this.  Sun Microsystems actually asked me to write a JSR proposal for this.  But I can't do that, because I am not Google's JCP representative.  And I cannot do it as an individual contributor to the Java Community Process, because as a Google employee I cannot be an individual contributor to the Java Community Process.  

Google's JCP representative is Joshua Bloch, and he has other ideas about what should be done in the Java language in this space.  As far as I know he is not currently planning on submitting a JSR on this.  My hope is that creating a prototype, which by the way, I'm doing on my own time, will be something that Sun can use as a justification for Sun creating a JSR to do this into the language, because I think that's the only way this will happen in Java at this point."

Here's a little of Josh Bloch on closures:

"I like closures. I think they work very well in languages that are designed around them, like Smalltalk and Scheme and so forth. I think closures in java are a good thing. We've basically had them since 1.1 in the form of anonymous class instances and those are a bit clunky so I definitely see a place for improving support for closures; on the other hand some of the proposals that I've seen floating around are overly complex; they involve massive changes to the type system, things like function types and I'm severely worried about pushing the complexity of the language beyond the point where Joe Java can't use the language anymore.

If we add more support for closures I think it has to be in the spirit of the current support, which means that closures should take the form of instances, of types that have a single abstract method, whether they are interface types such as Runnable , or class types such as TimerTask. I think it would be nice if a better syntax in the current anonymous class syntax were adopted and if requirement with regards to Final variables were somehow made more tangible, which doesn't mean necessarily relaxed; I think it's actually good that you can not close over non Final variables, but it's a pain to actually mark them final, so if they automatically marked themselves final which would be nice."

His points are all good, except that function types aren't really a big overhaul of the type system - they will be resolved into interfaces.  They're no bigger a conceptual problem than array types are.

Also, I think that if I could write a forEachLine in the BGGA style, I'd be more than happy for novices to use it.

The wart I mentioned is that Tennents' Correspondence Principle is violated by the BGGA when a closure throws a checked exception, and the interface that the closure conversion converts it to doesn't include any checked exceptions.

Neal proposes an extension to the generic type system to allow writing interfaces whose single method throws 0, 1 or many checked exceptions, but for those cases where the interface is already fixed, such as SwingUtilities.invokeAndWait(Runnable), the exceptions cannot be passed back to the caller.  The code fails to compile.  This is an inconsistency with the rest of the proposal.  You can return from within a Runnable closure, but you can't throw a checked exception from within it.

I think that it would be possible to achieve exception transparency without needing anything special on the interface, as checked exceptions are purely a compile-time concept.



Refactoring or Decoupling - which one do you want?  Both.
2007-03-23T10:17:00.000Z
A lot of time is spent on developing and using good refactoring tools, so that if we do decide to change something, we can, with minimal effort.  However, refactoring doesn't generally work between codebases, encouraging large, monolithic codebases.

The old wisdom about always programming to interfaces, in case you want to change which implementation to use later, seems a bit flawed in the presence of refactoring IDEs.  When I'm teaching, to keep things simple I let students use ArrayList references, rather than List ones.  Of course, that breaks down with subList, because subList doesn't return an ArrayList.

They'll never ever want to change their ArrayList to be a LinkedList, or a Vector, and even if they did, it would be a simple change with a refactoring IDE.  Ok, so the difference is that my students aren't generally writing APIs..

Well, Josh Bloch suggests that writing internal code as if you're writing APIs is a good idea.  After playing with a dynamic language again, I can see the benefits of that straight away, if you don't have a refactoring IDE.  I can also see that the refactoring IDE encourages you to use one codebase.  My main project, a network simulator, relies on a set of functional programming utilities (functionalpeas) that I keep as a separate project.  If I want to change the API of the FP utilities, I open the FP project, edit, build a jar, then go back to the simulator, copy the jar across into lib/, rebuild and fix the errors.

This experience makes me less likely to split my work into several separate modules.  I can, at any point in the networking code, get access to any part of the GUI.  Obviously I don't do this very often, maybe just during debugging, but it would be nice to be able to actually prevent myself from doing that by accident.

I see two solutions to this: either I can get it right first time, every time, or IDEs can start to support refactoring between projects, possibly even pushing the refactor out to remote developers.  Imagine this: "A library that this project uses has been updated.  Would you like to review the refactoring script that comes with the update?".


Programming Language: Servant or Master?
2007-03-23T09:13:00.000Z
In discussions about language features for Java 7, I've seen many suggestions that mythical 'other programmers', presumably ones who don't read/write blogs, will write terrible code that uses the language feature.

Programmers will write terrible code, regardless of the restrictions you place on them.  If you make it harder for them to write their terrible code, they will learn more slowly.  Many even stop learning, but that's a subject for another day.

Personally, I've had some ridiculous ideas over the years, and those ideas only lived so long because I couldn't try them out very easily.  Some good ideas died for the same reason.

In C and Lisp, you are the master over the language.  That seems to be true of Python, Ruby and Smalltalk too.  Of course, C's #define is so coarse as to make communication between master and servant a bit stilted, but the theory's there.

In Java, C#, Pascal and other academic languages, you are the servant.  Of course, any servant can learn to express himself, but it takes longer.  It's like being forced to speak another language, such as the Welsh were.  I expect that the literary production of the Welsh people suffered for a time when they were forced to speak English.

If I want a switch statement that works on Strings, I can write it myself, as a developer, not a compiler-writer, in a less restrictive language.  Then I can come across problems with it (such as fragility) and find a better solution.

If I instead have to boilerplate out a load of if..else statements, I'll take longer to realise that my idea is faulty, rather than the switch statement being inflexible, because I'll be blaming the wrong thing for my boilerplate.

If you're worried that operator overloading would let your fellow programmers write awful code, ask them what they would do with it.  If you worry about what open classes would result in, write a program in Ruby.  If you think that the program was too small to determine it, write a bigger one.  If you worry about Lisp macros, write some, or define the following macro:

(defmacro defmacro () ())

Now you can't define any more macros.  You can remove defun in the same way.  I imagine that wouldn't be so good for your career.


Why I should learn Spanish
2007-03-11T04:24:00.000Z
I work as a lecturer, a programmer, a DJ in a latin club, and a salsa teacher, in descending order of pay per hour.

"¿Habla español?"
"Como un idiota."

On Friday night, a Cuban lady came to my DJ box, during a Los Van Van song, asking for Cuban music.  I asked her which country she thought Los Van Van was from (they're Cuban; as far as I know, the second most popular band in Cuba).  She didn't understand me.  I got the message through via her friend, but she didn't seem to recognise the band as Cuban, and pulled a face like she'd just found out that some meat she was eating belonged to the rattus genus.  I asked her which band she'd like to hear.  Sometimes this is interesting - I hear a name I've never heard of before, can't work out how to spell it, pull out a pad and a pen, and the requester writes it down.  Usually, at this point, I recognise the name, but couldn't understand the requester's flavour of spoken Spanish, partly because I speak Spanish like an idiot, and partly because there are just too many decibels around.  At times it's tempting to forget about the other 200 people in the room and turn the music down just so that I can hear someone speak.

Anyway, this time; Blank expression, prolonged, then "Celia Cruz".  Celia died a few years back, but will forever be known as the Queen of salsa, with a huge number of platinum albums.  She is also the only Cuban female artist that many Westerners have actually heard of, apart from Gloria Estefan, who, according to my sources, doesn't count.  My suspicion was aroused - was she really Cuban?  I'd only seen her speak bad English, and look confused, which doesn't necessarily make one Cuban, though it probably helps.  It probably didn't help that I mistook her for a Colombian lady who had made repeated attempts to steal my beer a year prior.  Anyway, I'd already played 2 tracks in the last hour by Celia Cruz, so I wasn't too keen on playing another so soon.  I did oblige, and played another, about half an hour later, during which she shrugged her shoulders as if I'd ignored her request.  I shrugged mine in response, ironically.  I'm not sure how well irony works without speech.

One reason to learn Spanish is so that I can give sarcastic answers, like "Cuban?  Where do you think this track's from, Alaska?".  I can speak enough Spanish to say "Where's this song from?", but I don't know how to make it sound sarcastic.  I've checked, and latinos do use sarcasm.

On the same night, during the third reggaetón track in a row, someone came and asked if I was going to play any reggaetón tonight.  It's hardly as if the tracks weren't clearly reggaetón - the dancers nearby made it very, ahem, clear.

I'd only heard about this kind of idiocy from other DJs, I'd never seen it actually happen.  But two in one night?  On the plus side I got a lot of compliments about the music that night, and the night after.

Another reason to learn Spanish is the small fact that my girlfriend is Argentinian.  I wish I had heard her speak before I'd learned any Spanish though.  In theory, the different accent from Argentina should be beautiful, but to me, it sounds almost as strange as Texan English.  She sometimes laughs at my Spanish, but in a friendly way, and I do the same for her English, though I do correct her if she repeats a mistake, because she has exams in English soon.


A funny compiler bug in Java 5
2007-03-07T19:23:00.000Z
I made a couple of versioning mistakes recently when moving my development environment from one computer to another (and to another).

This basically resulted in me sending a 1.6-only binary to a QA team (read: my boss) with a 1.5 JVM.  I decided that the best way to solve it was to move my development environment to 1.5, and just use 1.6 to run applets and ordinary programs (such as IDEA).  I was already using -source 1.5 in my ant file, but I had forgotten to tell IDEA about that..

When I downgraded, I recompiled, and found a very amusing compiler bug in the latest release of Java 5.

A bit of background:

I've never liked the usual workaround for the 'final' restriction on local variables accessed from within a local class, i.e., the single-element array.  Hopefully it will go away with Java 7.

So, I made a Var class.  It used to be all fake-OO, it would have a set and a get, a bit like Shai Almog's amusing properties proposal.  Since upgrading to 1.6, I decided that as I did no validation with a Var, I may as well expose its value directly.

public final class Var<T>
{
    public T value;
    public Var(T value)
    {
        this.value=value;
    }
}

The Java 5 compiler was crashing on var.value++ within local classes.  I had to change those to var.value=var.value+1.

I wonder how that manages to be fixed in Java 6, but fails consistently with Java 5 versions (I've seen it once before, but didn't bother to debug it at the time, as I was mid-refactor).  I'm not sure it's even worth reporting as a bug, because somebody must have fixed it for Java 6, and made a conscious decision not to bother for Java 5.  Or so I assume.

Perhaps the Java compiler is trying to force me to use encapsulation.  Personally, I prefer one of Lisp's styles of encapsulation - don't even make the attribute that you're concerned about be a part of the object (e.g., (let ((x 5)) (defun inc () (incf x 5))), assuming I've got that right), but we can't do that, so I just make it public until there's a reason not to.


Google without the Search - Google Reader
2007-02-23T16:18:00.000Z
I use quite a few Google services, but I noticed for the first time that Google Reader has no search facility - I sometimes don't remember to bookmark something I've replied to, and then have to search for it again if I want to check for updates.  It seems fairly odd for a search company to omit search.

For a few hours on Google Reader yesterday, I noticed that in the List View the titles of the articles weren't shown unless I clicked on them.  I just switched to the Expanded View for that period, so I'm glad I only had a couple of hundred posts to scroll through (and a mousewheel!).

I also find Google Docs a bit weird, it's started telling me occasionally that another collaborator has made some edits, and the edits from the last few seconds will be removed.  Not a huge problem, but I'm the only person with the rights to edit the document in question.  Marking things as bold, or changing the fonts, etc., sometimes happens to other text besides the bit I've selected, too.

Perhaps Google's code quality is suffering as the organisation grows, or perhaps there's too little (none?) testing before features are pushed out to beta projects.  After all, Google is using beta users as testers, but we're also using them.  The reason everyone was happy to use Gmail in beta was because it was so damn good.  It's a shame that other Google services aren't treated as killer apps like Gmail was.


Sanitising some code
2007-02-20T01:25:00.000Z
A simple but valuable refactor - converting an interface plus two similar implementations into a final class. This is taken from real code; I wrote this document while refactoring.

Here I have an interface that represents an entry in an ARP table.

public interface ArpEntry extends Stringable
{
 MacAddress getMacAddress() throws CheckedIllegalStateException;
 void age();
 boolean dead();
}
Stringable has one method, asString() - I call that instead of toString(), so that I never get java.lang.SomeType@89345aef3 in logs, etc.

getMacAddress() may fail, because there are two kinds of ARP entry - some have a MAC address, and some do not (in real implementations, the second kind have a zero MAC address).  The ones that have no MAC address are entries to say that a request has been sent.  That's why it throws an exception.

age() will age the entry by a unit - approximately one second, but as we use this simulation for teaching purposes, we may allow slower students to slow this down.

dead() tests to see whether the entry is dead (aged beyond a certain limit).

Currently, I have two implementations, both anonymous classes:

public class ArpEntryUtility
{
 public static final int START_TTL=20;

 public static ArpEntry arpEntry(final MacAddress macAddress)
 {
  return new ArpEntry()
  {
   int timeToLive=START_TTL;

   public MacAddress getMacAddress()
   {
    return macAddress;
   }

   public void age()
   {
    timeToLive--;
   }

   public boolean dead()
   {
    return timeToLive<=0;
   }

   public String asString()
   {
    return macAddress.getRawValue()+"; expires in "+timeToLive+" seconds";
   }
  };
 }

 public static ArpEntry incompleteArpEntry()
 {
  return new ArpEntry()
  {
   private int timeToLive=START_TTL;

   public void age()
   {
    timeToLive--;
   }

   public boolean dead()
   {
    return timeToLive<=0;
   }

   public MacAddress getMacAddress() throws CheckedIllegalStateException
   {
    throw new CheckedIllegalStateException();
   }

   public String asString()
   {
    return "incomplete ARP entry - expires in "+timeToLive+" seconds";
   }
  };
 }
}
It's trivial to see duplication here.  One approach would be to make the interface a superclass, or to add a superclass that implements the interface, as a base for common code.  However, that isn't the only choice.  Let's look at a more classic form of code reuse - calling a method.

I'll add two methods, getTimeToLive, setTimeToLive, to the interface.  These will do no validation, just pass things through.  Don't panic, these methods won't live long.

public interface ArpEntry extends Stringable
{
 MacAddress getMacAddress() throws CheckedIllegalStateException;

 void age();
 boolean dead();

 int getTimeToLive();
 void setTimeToLive(int ttl);
}

Now we can implement age() and dead() as static methods in ArpEntryUtility, and call them from the two implementations.  Of course, those method calls are still duplication - we can call the static methods directly and remove the methods from the interface.  IDEA has a refactor for this whole paragraph - Make Static.  It will sort out the callers for you too, changing entry.age() to ArpEntryUtility.age(entry).

Now almost the only case-specific code is getMacAddress().  If I change getMacAddress() so that it returns Maybe<MacAddress>, then there's no need for the exception, and hence no need for getMacAddress() to be implemented differently between each implementation.

public interface ArpEntry extends Stringable
{
 Maybe<MacAddress> getMacAddress();
 int getTimeToLive();
 void setTimeToLive(int ttl);
}
Looking at all the use sites, I see that getMacAddress() is only used once, and in that case the exception is caught and converted to a Maybe anyway, so I've just made the use site simpler too, by chance.

It looks almost like a struct now, the only case-specific code left is the asString() implementations.  I can make that a static method that does different things based on the Maybe<MacAddress>.

The two implementations now only differ in how they are constructed.  One is passed a MacAddress, the other isn't.  Easy to solve, pass a Maybe and now we only have one implementation.

One interface, with one implementation.  Needless indirection.  Let's change the interface to be a final class, and the implementation to just be a constructor call.

Finally, we can get rid of the getters and setter, making macAddress a public final field, and timeToLive a public field.  That gets rid of some extra needless indirection.

public final class ArpEntry
{
 public final Maybe<MacAddress> macAddress;
 public int timeToLive=20;

 public ArpEntry(Maybe<MacAddress> macAddress)
 {
  this.macAddress=macAddress;
 }
}

public class ArpEntryUtility
{
 public static void age(final ArpEntry arpEntry)
 {
  arpEntry.timeToLive--;
 }

 public static String asString(ArpEntry arpEntry)
 {
  return arpEntry.macAddress.apply(new Lazy<String>()
  {
   public String invoke()
   {
    return "incomplete ARP entry - expires in "+arpEntry.timeToLive+" seconds";
   }
  },new Function<MacAddress,String>()
  {
   public String run(MacAddress macAddress)
   {
    return macAddress.getRawValue()+"; expires in "+arpEntry.timeToLive+" seconds";
   }
  });
 }

 public static boolean dead(ArpEntry entry)
 {
  return entry.timeToLive<=0;
 }
}

You might now decide to restrict the users of ArpEntry so that they have to access everything via the provided static methods.  That's fairly simple, just merge the classes and make the fields private.  However, I like to keep my 'bags of functions' separate from my instantiable classes.

While I was refactoring, one of my automated tests started to fail.  It actually took me about two hours to fix the problem.  Inadvertently, I had made some of ArpEntry's client code more logical, the code that decides what to do with an outgoing ARP packet from a computer.  However, some parent code to that, the code that decides whether to send an outgoing ARP packet) had a logic error, which I'd never noticed before.  This refactoring wasn't as straightforward as it could have been, mainly because IDEA doesn't know much about the Maybe type.  But overall, I'm pleased I spotted the logic error now, rather than when I'm closer to a release!


Ricky's Properties for Java
2007-02-18T22:48:00.000Z

While I think there should be some language change to make properties really useful, it's worth looking at how close we can get to properties without changing the language, to work out what needs changing.

Suppose you were to write a method called, say, setField, or setf for short, that takes a property and a value, and sets it.  This is fairly reasonable as something you might like to do with arbitrary properties, for, say, a GUI.
  
  <T> T setf(Property<T> property,T value)
  {
      property.value=value;
  }
This approach relies on having one object per property, so it's easy to see it as a memory leak.  It's not actually a memory leak, it's a memory overhead.  Suppose that you create 1000 objects, each of which take 100 bytes normally.  Now you change them to expose Property objects as public final fields, and each object now takes 1000 bytes.  It's only a (potential) leak if you need to create objects every time you use a Property.  What we do have now though, is n more objects, where n is the number of properties. 

This isn't a hopeless situation; there is a possible VM-based solution, holding Property objects directly, i.e., without a pointer, as part of the object they're in.  This requires the size of a Property to be known by the verifier, so the actual Property implementation would need to be known at load time.

While this might seem like a hopelessly early optimisation, it's worth thinking about now, because if properties do get implemented in the language, and they are completely flexible (so that you can replace a Property object at runtime), then we'll no longer be left with the possibility of this optimisation.  A halfway house would be to make it possible to prevent a Property object from being replaced, or for the VM to be smart enough to tell which Properties aren't going to be replaced. 

  Obtaining a Property object is tricky

  
If we want setf to work with properties defined by existing code, we should be able to recognise the getX/setX convention, and make those into Properties.  Let's look at how we can create a legacy Property using current (Java 5/6) code:
    Property<String> nameProperty=new LegacyProperty<String>(new GetterAndSetter<String>()
  {
    public String get()
    {
      return object.getName();
    }
  
    public void set(String s)
    {
      object.setName(s);
    }
  });
This gets a bit shorter with the BGGA closures syntax, or even method references, but it doesn't get any sweeter.  It's still pointless duplication.

Another implementation would use reflection.  Property<String> nameProperty=new ReflectiveProperty<String>("name",object); 
Obviously there are the usual problems with this, such as type safety not being guaranteed at compile time, performance, that it requires tool support if the programmer is to be certain that it is refactor-proof.  There is an extra problem caused by erasure of generic types; there's no way of knowing that nameProperty really is a Property<String>.  setName could take an instance of some 'Name' class.  This is not simply a case of choosing dynamic typing over static typing, because erasure doesn't give us a choice.  The reflective solution is not typesafe at all unless we either implement reification or give ReflectiveProperty a 'type token', in this case String.class.

    Property<String> nameProperty=new
  ReflectiveProperty<String>(object,String.class,"name");
  It doesn't work with legacy bean-manipulating code
  

Suppose I write a new class and don't write getters or setters, but instead expose my fields via Properties.

    class Person
  {
    public final Property<String> name=new DirectProperty<String>("unknown");
  }

Now any code that reflects on Person looking for getX/setX methods won't find any.  It's arguable that the code should use Introspector to introspect, rather than direct manipulation, and hence that I could provide a BeanInfo class for Person, but not all the code that manipulates beans uses Introspector.

  Erasure could make a List of Properties useless.
  

  Suppose you asked a bean for all its properties, either directly or via some introspector.  You'd get a List<Property<What?>>.  It cannot be Object.  It can be ?, though this would prevent set from being called.  It can be a raw type.  In any case, erasure will stop us from seeing the actual type of the property, unless we add a type token, as mentioned earlier.

  What needs to change? 

  Now let's take the above and make it convenient to use by changing the
  language a little.

  1.  All getX/setX pairs are properties.  This includes isX, read-only properties and write-only properties.  This allows new code to work with existing beans.

  2.  All explicit property declarations generate getX/setX or isX/setX pairs at compile time.  This allows new beans to work with existing code.

  3.  The generated code simply calls the Property's get/set methods.  There is no generated field in the declaring class, other than for the Property itself.

  4.  A syntax is provided for getting at a named Property given the name of a bean.  This is statically checked for correctness.

  5.  A syntax is provided for getting the value or setting the value of a property.  The '.' operator will suffice.  A field and a property cannot exist with the same name, which avoids compatibility issues with existing code.

The easiest argument against this is also the easiest to refute, namely that it calls non-obvious code.  The same argument could be used to reject polymorphism.  Plus, there are already precedents in Java.  arrayElement[index]=value is an assignment that does more than it appears to - it checks bounds.  String concatenation calls .toString() on objects.  + promotes low primitive types to int.  These are all good things; there's nothing fundamentally wrong with calling non-obvious code, as long as it is possible to discover what code is actually called. 


The strongest argument in favour is readability - there should be no readability price for using properties.  Currently there is a price.



Closures without instances, and safer non-local returns
2007-01-12T19:12:00.000Z
It's possible to implement synchronous closures without instances, and to make non-local returns in closures not subject to unchecked exceptions.  This article explains why.

I like using Google Docs as a word processor, but the last time I published to blogger automatically from it, I lost some formatting, and even a random backslash, so this time I'm not going to bother.  The actual article is here.  Comment on this blog.


Why Closures in Dolphin is a Good Idea
2006-12-18T14:14:00.000Z
 Why Closures in Dolphin is a Good Idea

On Javalobby, Mikael Grev argues that, while he personally likes closures, and would use them, he would not want them to exist in Java.

He makes quite some deal out of keypresses - namely, the excessive amount of keypresses required for an anonymous class in Java.  However, this misses the point somewhat - if something takes many keypresses to type, it will take many 'brain cycles' to parse.  In fact, in even the simplest code, an anonymous class is so distracting from the intent of the code that it almost prohibits some excellent coding styles.  That is demonstrated in this article.

Apologies for the code formatting - posting from Google Docs seems a bit dodgy.  Here's the original.  If you're not familiar with use cases for closures, I strongly suggest you take a look at Neal Gafter's blog  , and the links he has on there, which, even if you disagree with the authors' points, the articles are at least very entertaining.
Closures Make Code Easier to Understand

Consider the following Haskell code:

map (+2) [1..10]

This is the usual Java code that's roughly equivalent:

int[] list=seq(1,10);
for (int a=0;a<list.length;a++)
    list[a]+=2;


Fairly readable, but it's not as expressive.  It doesn't say 'add 2 to each element of a list from 1 to 10'.  Instead, it says 'make a list.  for each element of that list, add 2'.

That's two sentences, and the second has a sub-clause.

If I try to write the more expressive form (the Haskell version) in Java, I get something like:

result=map(new Function<Integer,Integer>()
{
    public Integer run(Integer input)
    {
        return input+2;
    }
},seq(1,10));

As you can see, the original less expressive form actually maps better onto Java than this better style.  And not just in number of keypresses, but in readability.  After all, I wrote it, you're just reading it, and you're probably cringing.

The (+2) syntax from Haskell is a way of specifying the + operator, but with one of its values pre-filled.  There is a more verbose, probably more familiar, syntax, that resembles Java's impending closures more:

map (x -> x+2) [1..10]

This is an anonymous function that takes a value, x, and returns x+2.  The word anonymous is important.  The moment we 'simplify' things by giving names to such code snippets, e.g., a named class, or a field that holds the function, or even a local variable that holds the function, we're not actually simplifying, we're increasing the number of sentences needed to express ourselves.

Function<Integer,Integer> addTwo=new Function()
{
    public Integer run(Integer input)
    {
        return input+2;
    }
};

result=map(addTwo,seq(1,10));

Yes, this starts to look more attractive, but it's not really.  It just means that we need to understand what addTwo is, as well as what map is and what seq is.  We're adding to the number of things we either need to commit to the subconscious, or hold in primary mental space.

For this extremely trivial example, you might wonder what the big deal is.  If this was all the benefit one could get from closures, I'd agree with Mikael.  However, by making trivial code exceedingly trivial, you can make less trivial code trivial, and complex code, well, readable.  Being able to understand more code at once means that you can spot mistakes in it better.

Closures help to keep your code DRY, and encourage excellence.

DRY is Don't Repeat Yourself.  By making the above more expressive code also more attractive, you open yourself up to all sorts of optimisations (removal of repetition - I'm not talking about performance, though that does come into it somewhat).

Most operations on lists or Strings can be expressed in terms of mapping or folding (also called reducing).  For example, joining a list of Strings to add colons in between is a fold:

result=fold(new String[]{"root","0","/bin/bash"},new Function<Pair<String,String>,String>()
{
    public String run(Pair<String,String> pair)
    {
        return pair.first()+":"+pair.second();
    }
});

Now, at first glance, that code is garbage.  Let's add closures:

result=fold(new String[]{"root","0","/bin/bash"},{first,second => first+":"+second});

Now, if you understand that to 'fold' is to run a function on the first and second elements of a list, then run the function on the result of that and the third element, etc., then you'll probably quickly understand the code above - but the excess notation in the anonymous inner class version makes it harder to grasp.  This makes using fold unattractive.  fold and map are some of the best techniques available for working with lists of data.  They are immensely flexible and scalable.  Google's famous MapReduce algorithm is entirely based on them.

So, without closures, we are not likely to come up with algorithms like MapReduce - that is, we are actively discouraged from writing the best code.  Of course, we are able to think outside of the programming language that we use, but it tends to be slightly harder to do.  I doubt that many Java programmers think in terms of folds and then convert that into a suitable Java version.  Instead, we think in terms of the Java version, and maybe realise later that it was another hand-coded fold implementation.

Further, by keeping code DRY, you keep maintenance costs low, e.g., if you have a bug in your withLock() implementation, your using() implementation, your withResource() implementation, etc., you can fix it in one place.  If you didn't use those, but hand-coded (or IDE-generated) it every time, then you're fixing it in many many places.

I once looked through some of the JDK source, and found that most of the resource allocations don't follow the suggested best practice - the try..finally{try{close}catch{log}} idiom.  I wager that this would not have been the case had closures existed from the start.  Reusable solutions would have been more attractive - more convenient.
And Now to Refute Some Points in General
These are from Mikael:

"the benefits must be proven to be measurably greater than the costs".  It's impossible to prove that, as the benefits and the costs both have humans as part of their variables.

"I would guess that the more advanced coders, the ones that is usually on the closure side, does this".  In that statement, 'this' meant auto-generating code using an IDE for anonymous classes.  That is probably true, but auto-generating code is a workaround for a missing language feature (not necessarily a feature that should be there though - it's only with this clause that I can make the generalisation).  More advanced coders probably get a slight pang of 'this sucks' whenever they auto-generate an anonymous class, or getters and setters.

"That is unless you have to use one of the proposed syntaxes for handling exceptions thrown in the closure or have some funky return structure".  Clearly, anonymous classes aren't going to be removed from the language, so if you find the syntax hard to understand, you can always revert to anonymous classes.  I expect IDEs will provide automated routes to and from closures and anonymous classes.

"The solution to this aesthetics problem isn't closures though, it can be solved without adding complexity by just allowing a little syntactic sugar for the AICs."  Even the syntactic sugar for AICs (anonymous [inner] classes) detracts from the expressiveness, and still discourages DRY and excellent code in the same way that AICs do now.  Consider:

result=fold(new String[]{"root","0","/bin/bash"},new Function<Pair<String,String>,String>()
{
        return pair.first()+":"+pair.second();
});

It's not a lot better, it's still got a lot of verbosity that could be inferred (the type parameters to Function, the word Function itself).  It's still distracting.

"Closures can do many things that AICs can't. Change the variables outside their scope for instance".  Like with autoboxing, you could conceivably configure your IDE to prevent yourself from doing this.  For most cases it won't matter.  Neal Gafter explained the reasoning behind making code that's inside a closure behave the same as code that's outside it.  It doesn't break the WYSIWYG nature of Java, because it's damn obvious that, say invokeAndWait{frame=new JFrame();} will assign to the nearest variable called frame.

"I still think that the AIC should only be working on a copy of the value."  This could only promote out-of-sync bugs.

"The primary cost here is that Java developers need to learn new constructs. Constructs that are not very Java-ish and therefore will take some time to getting used to."  That's not a cost, it's a benefit.  Learning how to use generics benefits those who have.  Generics didn't look very Java-ish, but they worked well.  It actually helps programmers to learn new concepts.

"Remember that not all are as bright as you and you gain nothing from alienating the Java-Joes however good that feels for your ego."  Actually, I do teach some new Java programmers, and I'd be much more comfortable introducing them to:

invokeLater{frame.setVisible(true);}

than:

invokeLater(new Runnable()
{
    public void run()
    {
        frame.setVisible(true);  //and, er, you'll have to make frame final.
    }
});

Closures are simpler, for all levels of programmers.

"Take the much loved Collection framework. If it'd been closure-enabled from day one it would've been even better. Now you need to squeeze in closures".  Or make sure that closures are implemented in such a way that they are useful with the framework.  For example, we can implement a Comparator as a closure, and don't even have to say the word 'Comparator'.  It's inferred.  Type inference is good.

Collections.sort(list,(x,y => y.intValue()-x.intValue());

If the JDK had to include another version of sort that was closure compatible, which it doesn't, then I would agree with you.

"You could argue the same way [against] for anything that gives more power to the developer. #DEFINE is such a thing.".  The use cases for #DEFINE in C and C++, such as inclusion of header files, portability (hoho), definitions of function-like macros, are largely eliminated by simpler features in Java.  What features combined are simpler than closures, for all (or most) use cases that closures have?

"With closures you can code Java that doesn't look like Java and that isn't something I'd like for the Java community.".  You could replace 'closures' with 'generics' in that statement, rewind a few years, rinse and repeat.

And just a humourous note:  This is from Mikael's top ten tips on how to become a Rock Star Programmer: "Write smart cool compressed code constructs".  He's joking, but surely that's, well, closures.  From the same place: "Less code, in a smart way, means less to maintain.".  Agreed.  Smart doesn't mean 'hard to understand'.

"Frankly (sort of) defining new keywords on a developer level scares the bejesus out of me."  I wonder whether there was a time that allowing developers to write their own functions (rather than having them hard-wired into the machine) was scary.

Shai Almog said:

"Generally I tend to be wary from features that are designed for "experts"".  Closures in Java appear not to be designed for experts, but for programmers.  It looks to me like every effort is being spent to make programming in Java better.  The usage syntax is very compelling.

"VM changes are that much worse even worse than half baked implementations (e.g. generics)."  I've found that generics were cooked for just long enough.  They could use some extra features, some garnish, but they taste nice.  The worst thing about them is that they're awkward to talk about in comments on other peoples' blogs, with the old < etc.

This one's hard to quote, but, Shai conjectured that a closure-accepting method would be hard to maintain, because average programmers wouldn't understand the method.

1.  Don't let code into your codebase that is above ALL your staff.
2.  IDEs could probably refactor it into an equivalent interface-accepting method anyway with no change to the use site.
3.  It's already possible to write code that is above the level of other programmers, e.g., with generics, enums, finally (yes, there are programmers who don't get finally), etc.




Preventing NullPointerExceptions, Maybe
2006-12-15T12:43:00.000Z
Null has always bothered me.  I can write code without causing NullPointerExceptions, fairly easily, but without the techniques documented here, some still slip through.  Of course, my automated tests are entirely comprehensive (joke), so there's no problem, right?

Wrong.  Writing tests doesn't solve the problem that null exists in the first place.  If we place a bollard in the middle of a street, and test all the cars to make sure that they can get around it without hitting the houses, that doesn't make the bollard acceptable.

One rule absolutely solves this.  Assign a value to each field as soon as it's declared.  A non-null value.  To be picky, you'd have to also ban the new Object[x] form of array creation, and never give a local variable a null value.  Let's not be picky.

The instinctive reaction to this is to say that you don't always have a value to put in the field, and therefore that null is the best value.

Partly true.  However, null is not the best value.  The likely first thought is the NullObject pattern.  For example, if we have a java.sql.Connection field, we might set it up with java.lang.reflect.Proxy, so that we can call methods on the Connection, though they do nothing.  This only hides the problem, in obscure runtime behaviour.  Usually, we'd rather see clear runtime behaviour (a NullPointerException) than obscure runtime behaviour ("I thought I'd saved to the DB, but it was the NullConnection").  NullObject isn't going to help.

It's better to have a real distinction between a useful value and a useless value - one that forces you to 'check', or even checks for you.  There are a couple of ways of doing this.  The @NotNull and @Nullable annotations introduced by IntelliJ IDEA is one - though I haven't used those myself.  Another way is possible, using only the Java language.  Though it has to be said, the Java 7 language will make this more comfortable.

And Now To The Meat

The following concept was shamelessly stolen from Haskell.

Given a field that may have a Connection, or null, I'll change it to 'maybe a Connection', or Maybe<Connection>.  There are two implementations of Maybe - one of them does have a Connection (well, T), and one of them has Nothing.

Then, rather than testing it to see whether it really has a Connection, I tell it what I want it to do if it has a Connection, and what I want it to do if it doesn't have a Connection.  Oh, and for maximum flexibility, return me the result.

Let's go with a less flexible version for a moment, as an explanation.

interface Maybe<T>
{
    void apply(SideEffect<T> runThisIfTheresAnObject,Runnable runThisIfThereIsnt);
}

interface SideEffect<T>
{
    void run(T input);
}

So, when I call maybeConnection.apply(saveStuffToDB,initialiseConnectionAndSaveStuff), if maybeConnection is 'just' a Connection, it will call saveStuffToDB.run(connection), and if it is Nothing, it will call initialiseConnectionAndSaveStuff.run().

However, there are two problems with this approach.  One is that I tend towards functional programming, and this stuff relies on side effects, so it irritates me.  The other is that programming side effects with anonymous classes can really be a pain in Java, thanks to the 'final' requirement on enclosing local variables.

So what I really want to do is to change apply so that it returns something.  I could make it return Object, but then I'm just reintroducing the old ClassCastException possibility.  I could make Maybe take two type parameters, T and R, R being the return type of apply.  However, that would mean that each Maybe would only be able to run 'functions' that return one type - impractical.

Generics allows you to declare type parameters on methods, not just whole classes/interfaces, so let's do that.  If you don't like the look of this, skip to the bottom and eye up the alternative implementation (visitor).

interface Maybe<T>
{
    <R> R apply(Function<T,R> ifT,R ifNothing);
}

Let's just walk through that syntax.  <R> just declares a type parameter.  If you don't like that, simply ignore it.  apply takes in a Function, which has one method, R run(T), and it takes an R.  If there is a 'real' object, a T, the Function's run method will be invoked, and the R that it returns will be returned from apply.  If there isn't a real object, then ifNothing is returned.

It's rather like encapsulating an if statement.  By taking the responsibility for checking null away from the user of Maybe, we're taking the possible bug away too.  Note that we're only taking it as far as Maybe - of course, if the two implementations of Maybe are broken, then the bug will be everywhere.

And now for example usage:

Maybe<Connection> maybeConnection=MaybeUtility.nothing();
... some code, might set maybeConnection to something else, might not.
String outputToUser=maybeConnection.apply(new Function<Connection,String>()
{
    public String run(Connection connection)
    {
        some code that uses a PreparedStatement etc. and returns a String.
    }
},"Er, some fool forgot to connect to the database.  Fire Fred");

What we're doing here is implementing dynamic dispatch.  It's  another way of implementing the visitor pattern.  In fact, Maybe can be implemented easily via the standard idiom for the visitor pattern - the only reason I don't is that I like single-method interfaces.  I find that they fit my thinking better.  They also fit the closure proposal better, which is probably worth bearing in mind now.

Here's Maybe implemented with a more obvious visitor approach:

interface Maybe<T>
{
    <R> R accept(MaybeVisitor<T,R> visitor);
}

interface MaybeVisitor<T,R>
{
    R ifJust(T t); //in Haskell, the opposite of Nothing is Just, in terms of the Maybe type.
    R ifNothing();
}


Maybe and friends can all be found in Functional Peas, which is currently a placeholder for some useful bits and pieces of functional (or nearly-functional) code.

Yeah, but..

If you think that this is wasteful in terms of programmer time, I might agree with you - until we have good syntax for closures, using Maybe isn't syntactically that attractive.  This can be dealt with to an extent - such as by reducing the need for null from the original code, or choosing the visitor approach.  I will blog about techniques for doing that, probably under the heading 'Reducing Mutability'.  Another way is to prefer function composition over always writing 'closures', which personally I'm not very good at that yet.

If you think this is useless, because people don't make mistakes if they test enough, I refer you back to the bollard analogy at the beginning.

If you think this is useless, because I am not on a large team, I'm young, I work in a University, and therefore don't know what I'm talking about, then please don't bother commenting, and have a nice life.

If you think that this is useful, but that your colleagues won't understand or agree, just discuss it with them.  They might have a better idea.



Making equals(Object) type-safe
2006-12-02T17:21:00.000Z
Update - Jean-Francoise Briere came up with a better solution - I've included it at the bottom.

It's easy to end up comparing two obviously incomparable objects using the equals(Object) method.  This is because equals is not generic.  By incomparable, I mean that it is possible to tell from the source that they are incomparable, such as new StringBuffer().equals("").

The contract of equals does not permit throwing an exception when incomparable objects are compared, because comparing two incomparable objects is not a problem; comparing two incomparable references is.

That is, it makes sense if you have an List<Object> and add an Integer to it, then look in it for a String, for the equals method to be called, so it should behave normally (of course, it's quite likely that the hashCode method will be called instead).

So we can't really ask the objects themselves to help us out, unless we create lots of overloaded equals methods.  E.g., an Integer would need equalTo(Integer), equalTo(Number) and equalTo(Object).  Clearly a pain in the proverbial.

We can instead ask the static type system to help us out.
First Bad Solution; Type-Specific Statics
Wrap equals calls up, say, in static methods.

public class Equaliser
{
   public static boolean equals(String first,String second)
   {
       return first.equals(second);
   }
}
There are two problems with this:

1. Repetition - you'd have to do this for every type.
2. It doesn't work if you put a supertype in, e.g., equals(Object,Object) would match any calls and you wouldn't notice.
Second Bad Solution - Naive Application of Generics
public static <T> boolean equals(T first,T second)
{
   return first.equals(second);
}
This will actually allow a comparison between an Integer and a String, because T is resolved to Object, unless you use the clunky qualifying syntax - ClassName.<Integer>equals(1,"blah"), which is so bad it's worth avoiding in most cases.  Your favourite IDE will confirm that the unqualified version resolves T to Object when you hover over the call.
First Not-So-Bad Solution - Less Naive Application of Generics

interface Equalator<T>
{
    boolean isEqualTo(T other);
}

public static <T> Equalator<T> equalator(final T first)
{
   return new Equalator<T>()
   {
       public boolean isEqualTo(T second)
       {
           return first.equals(second);
       }
   };
}
This is called as: equalator(someReference).isEqualTo(someOtherReference), and will catch more bad comparisons.

Of course, if your references (not your objects) are actually of type Object, then this won't be useful at all.
And Finally, The Same Thing But More Reusable
In my own code, I implement this as a (badly named) method, equalT:

    public static <T> Function<T,Boolean> equalT(final T first)
    {
        return new Function<T,Boolean>()
        {
            public Boolean run(final T second)
            {
                return first==second || first.equals(second);
            }
        };
    }


Now it's called as: equalT(one).run(two).  I should probably get rid of the first==second part of that.

Function is a type I defined in the publically-available functionalpeas package - it represents a function with one argument and one return value.

Now I can pass equalT(someString) to a method that expects a Function<String,Boolean>, which can be handy, and explains the 'reusable' part of this section's subtitle.  

One really cool thing about it is that if you replace all your calls to equals AND all your code to == with this, including primitive==primitive, then you won't get any of those pesky problems caused by comparing references instead of objects.

Have fun.  Next time I'll look at eliminating NullPointerExceptions.  No, really.

Update

Jean-Francoise Briere's solution is based on my 'naive implementation using generics', but is less naive:

public static <T,U extends T> boolean equalT(T t,U u)
{
    return t.equals(u);
}


This is better than my final solution, because it doesn't rely on creating a new object for each comparison, and there are less keypresses.  Thanks, Jean-Francoise!


A fairer brevity comparison between Ruby and Java
2006-10-20T17:32:00.000+01:00
As a Java programmer, I'm not completely convinced that brevity is always good.  I know that I can write some pretty unreadable brief code.  ~(~0>>>prefixLength) is a nice little example.  It converts a prefix length, e.g., /24, in a network number, into a netmask, e.g., 255.255.255.0 (as an unsigned int).  However, for this article, I'll put readability on the backburner, somewhat.

When I stumbled across Sometimes Less is More by Peter Szinek, who appears to like being photographed with camels, I found that some of the Java code posted seemed to be written by, well, someone who didn't like Java.  In this post I'll try to suggest better Java examples.  I will omit imports and method declarations unless they are relevant.

1.  Ruby:

10.times { print "ho" }
or
print "ho" * 10

Perl possibly has a more sane syntax, print "ho" x 10; - this way '*' doesn't mean both multiplication and repetition.  I'm not too bothered either way on this.

The article actually gave no Java equivalent, here's one:
out.println(format("%1$s%1$s%1$s%1$s%1$s%1$s%1$s%1$s%1$s%1$s","ho"));

out is System.out, and format is String.format, imported statically.  Obviously if you make a method to do this, the calling code becomes very very short: out.println(repeat("ho",10));

repeat comes from cirrus.hibernate.helpers.StringHelper (found via Google Code Search).

2. Ruby: if 11.odd? print "Odd!"
The article showed this as the Java equivalent:
if ( 1 % 2 == 1 ) System.err.println("Odd!");
However, this only works for positives and zero.  I'd prefer:
if (1%2!=0) out.println("Odd!");
(-3%2==-1)

Again, with prewritten methods, this can become clearer - there is a prewritten 'odd' method in the JDK 1.4 demos, should this prove hard to write yourself.

if (odd(11)) print("Odd!");  //let's assume print is a method that does System.out.println.
3.  Ruby: 102.megabytes + 24.kbytes + 10.bytes

Java: 102 * 1024 * 1024 + 24 * 1024 + 10

I'd prefer this a little: 102<<20 + 24<<10 + 10;

or with a little predefinition: 102*MB+24*KB+10.  I really wonder what the bytes method does in Ruby!

4.  Ruby: print "Currently in the #{2.ordinalize} trimester"

Java: System.err.println("Currently in the" + Util.ordinalize(2) + "trimester");

My suggested Java: out.printf("Currently in the %s trimester",ordinalize(2));

5.  Ruby: puts "Running time: #{1.hour + 15.minutes + 10.seconds} seconds"

Java: System.out.println("Running time: " + (3600 + 15 * 60 + 10) + "seconds");

Suggested Java: out.printf("Running time: %d seconds", 1*HOURS+15*MINUTES+10);

6. Ruby: 20.minutes.ago

Java: new Date(new Date().getTime() - 20 * 60 * 1000)

Suggested Java: new DateTime().minusMinutes(20) - DateTime is from Joda Time.

7. Ruby: 20.minutes.until("2006-10-9 11:00:00".to_time)

Java: Date d1 = new GregorianCalendar(2006,9,6,11,00).getTime();

Date d2 = new Date(d1.getTime() - (20 * 60 * 1000));

Suggested Java: new DateTime(2006,10,9,11,0,0).minusMinutes(20)

8. Ruby:

class Circle
  attr_accessor :center, :radius
end

Java: Too long, see the original article!

Suggested Java:
class Circle
{
    public Coordinate center;
    public float radius;
}

"a simple class definition having 10 fields in Java will have 80+ lines of code compared to 1 lines of the same code in Ruby."

For a lot of classes, many of those fields will be immutable anyway, or at least not exposed via getters/setters.  In the case of an anonymous class, there are zero extra lines of code for them.

9. Ruby: stuff = []
stuff << "Java", "Ruby", "Python" #add some elements

Suggested Java: List<String> stuff=arrayList();
stuff.addAll(asList("Java","Ruby","Python"));

10. Ruby: stuff = [”Java”, “Ruby”, “Python”]

Suggested Java: List<String> stuff=asList("Java","Ruby","Python");

11. The author complains that you have to sort arrays using Arrays.sort(array) instead of array.sort() - however, this can become sort(array) via a static import.

12. I think the stuff about stacks and arrays would be solved by using java.util.Stack, which has pop/push/subList etc.  I don't see a great need for the static implementation to appear as part of the object though.  I prefer thin objects.

13. The author seems to think that adding 'nil' values to an array/list when you try to add to an index beyond the end of the array/list is a good thing.  I rather like the little protection that Java gives in that if you want null values you have to add them yourself (except with the new String[10] syntax).

14. Ruby: File.read('test.txt').scan(/.*?\. /).each { |s| puts s if s =~ /Ruby/ }

Suggested Java:

import static java.lang.System.out;
import java.io.*;

class Test {
        public static void main(String[] args) throws IOException
        {
                File file=new File("filename");
                byte[] bytes=new byte[(int)file.length()];
                DataInputStream input=new DataInputStream(new FileInputStream(file));
                input.readFully(bytes);
                String[] sentences=new String(bytes,"ASCII").split("\n");

                for (String sentence: sentences)
                        if (sentence.indexOf("Ruby")!=-1)
                                out.println(sentence);
                input.close();
        }
}

I expect that I could mimic the Ruby way, given time and inclination.  The end result would be this, with supporting methods:

File.read("test.txt").scan("\n").each(ifMatches("Ruby",print));

15. Ruby:


          tree = a {
            b { d e }
            c { f g h }
          }


Suggested Java:


Tree tree=tree("a",
    tree("b",leaves("d","e")),
    tree("c",leaves("f","g","h"))
  );


And just as a general comment, I'd gravitate closer to Haskell, with its type inference and very powerful static type system, than towards Ruby, Python et al, because I like the freedom to be able to mess around with my code, knowing that there is an automatic eye-over-my-shoulder that's going to tell me when I do something stupid.

This post is not to invalidate the original article; it's clear that the Ruby way of doing things is obviously much simpler in many cases.  However, often the idiomatic Java way is not the best anyway.  Personally I've been experimenting with functional programming from within Java (is this like asking your wife to dress up as someone else?), and I think closures could really help in any future 'brevity wars'.


Where does static really fit?
2006-10-06T11:29:00.000+01:00
Graham Rocher suggests that closures cannot be added to Java because Collection cannot compatibly grow in number of methods (methods like forEach, any, all, etc.), and apparently adding static methods to the Collections class is not a solution in these halcyon days of dependency injection.

He does acknowledge that static methods are sometimes useful, but so far he hasn't said where or why.

He says that Math.abs is a poor example, because you never need an alternative implementation.  However, StrictMath.abs is an alternative implementation (though not used often).

Suppose that you wanted to be able to choose whether to use Math or StrictMath's abs implementation.

You could make or use an interface that contains the 'abs' method, and make two implementations of it.

I know that many blog readers find generics difficult to understand, and some even find anonymous classes tricky, so I'll be accommodating..


interface Abs
{
    double abs(double value);
}

final class LaxAbs implements Abs
{
    public double abs(double value)
    {
        return Math.abs(value);
    }
}

final class StrictAbs implements Abs
{
    public double abs(double value)
    {
        return StrictMath.abs(value);
    }
}


Graeme's argument is that, because substitutability is required, 'forEach' and friends should not be static, so that their implementation can be provided through dependency injection.

It can be seen from the above example that, given static methods, it is possible to make versions that are DI-compatible, so the static implementation can then be ignored (as it is wrapped).

Then, why can't the implementation of forEach be static?  For the users who just want to use the implementation given in the SDK, Collections.forEach is fine.  For those who want to be able to substitute it for others, wrapping it, or using a wrapped version provided by the SDK, is trivial.

I'd suggest that you can safely implement any algorithm as static, and, if there is a need, you can provide a non-static way of accessing it.


Autoboxed closures?
2006-09-11T10:31:00.000+01:00
In response to Graeme Rocher, whose blog doesn't seem to process comments particularly well..

Graeme said:

"Java has been around for a decade now. It has jumped several iterations to Java 5 (with Java 6 coming shortly) and the APIs have progressed hugely. The implications of adding closures would be huge, you would have to go back and revisit ALL the Java APIs. I mean, if closures had been around since the beginning the collections API would be entirely different."

I expect there could be some autoboxing to get around the API issue, e.g., a closure that takes no input and gives no return value could be automatically boxed to a Runnable.

Ditto for the other one-method interfaces, possibly.  It'd certainly be interesting if you could do that for your own interfaces too.

insert usual moans about autoboxing

Yes, autoboxing adds a bump to the learning curve, but if, when it is understood, it removes complexity when reading and writing code, then it is a good thing.

I think the autoboxing feature added in Java 1.5 works well, if you understand its mechanisms, though obviously if generics were able to work with primitive types directly (or appear to), autoboxing would be a lot less useful.


Duck Typing in Java, and no reflection
2006-07-12T21:19:00.000+01:00
interface CanQuack
{
 void quack();
}

interface CanWalk
{
 void walk();
}

<T extends CanQuack & CanWalk> void doDucklikeThings(T t)
{
 t.quack();
 t.walk();
}


You can pass anything to this method that implements CanWalk and CanQuack.

Of course, this is optimally flexible when you have one method per interface, but that's not a problem.

I can easily see the dependencies of doDucklikeThings, whereas if I had a larger Duck interface, e.g.:

interface Duck
{
 void quack();
 void walk();
 void eatAFish();
}


I wouldn't be able to tell by looking at the signature of a method that used Duck, whether it would call eatAFish.

This works well for reducing coupling, and would probably be better than this Proxy-based attempt.

There is no runtime cost, thanks to type erasure.  It's rare that someone thanks Sun for type erasure, it seems, but I have got to grips with generics fairly well, thanks to Angelika Langer's excellent generics FAQ, and I find it very useful.

I have applied this technique to a lot of the 20kloc program that I work on, IPSim (a network simulator), but I am still in this process.

I hope you find this useful, or give me some damn good reasons why it isn't!


Eclipse 3.1 User Experience
2005-07-14T15:04:00.000+01:00
I gave up using Eclipse in favour of vim and ant some time ago, mainly because I did some work from home, on a 233MHz machine (my faster personal machine was stolen last year), and Eclipse was too slow to be useful there.

But now I came across a bug in a unit test, and I can't figure out how to write a unit test to find that bug, so I have to do some debugging.

Not having Eclipse installed, the first thing I did was launch jdb and type help, which wasn't too 'help'ful.  I googled for how to use jdb, and only found a site that seemed inaccurate or out of date.  I had a play with ODB (Omniscient Debugger), but couldn't get it to instrument all my classes, just the particular one I was running.

So, I'm back with Eclipse, a fresh download.

My first thought is that it seems a bit easier to start up, I don't seem to have to tell it on the command line where to put its workspace, it has a nice dialog.

I go to Project->New Project and it has an entry for creating the project from an ant build.xml.  I'm staggered.  So I go about it and it seems to understand the build.xml properly.

Eclipse then decides that I've got 100 syntax errors, merely because I haven't told it that I'm using Java 1.5.

Let me check that build.xml.. hmm, I didn't have a source="1.5", but the ONLY compiler available on my system is the 1.5 compiler, so Eclipse could easily have detected that and just moved on.

The Quick Fix for the first error seems good though - change workspace compliance and JRE to 5.0, or change project compliance and JRE to 5.0.  I select workspace.

So Eclipse rebuilds and gives me the following warning:
taskdef class edu.umd.cs.findbugs.anttask.FindBugsTask cannot be found

Back to the build.xml..

 
  

  

  etc.

I can't see how Eclipse could fail to find that, seeing as ant can.  But hey, maybe if I go to Quick Fix I can tell Eclipse where the findbugs task is.  Hmm, Quick Fix is greyed out.

Thankfully, that's just a warning, so I'll try to ignore this annoyance and continue with my work.

All the other warnings seem to be pretty innocent, unused imports, etc.  It does seem that Eclipse is hiding some warnings though, as the errors before and the warnings now list exactly 100.  Maybe 100 is the limit.  I hope it isn't hiding any important warnings.

Eclipse seems to italicise my static import of netsim.java.lang.Assertion.assertTrue, which seems like a nice little feature.  On further use of the debugger, I see that it is italicising all static methods, nice.

Well, I found my bug, mainly that two parameters were the wrong way around, and I've exited Eclipse.

I'll still be using vim and ant, but I will keep Eclipse around for its debugger.  The build.xml parsing is very handy, but not perfect.  I hope Eclipse can improve this.  I'll still try other debuggers, but ODB left a sour taste in the mouth - poor documentation.  I should have blogged that user experience.  Maybe the next one.


A new design pattern (Farm) and a nifty use of annotations.
2005-04-09T16:29:00.000+01:00
It struck me that a difficulty with using interfaces and factories in Java is that you need to know which factory goes with which interface.  A naming convention can help here, but it might not be consistent.

Instead, I suggest that you annotate your interface, to know about a default implementation.

This immediately brings up "Interfaces shouldn't be bound to their implementations!!!  This is unpure, evil code!!!".  When you are navigating code, trying to find implementations for a particular interface can be annoying and time-consuming.

I have set out here the most cohesive way I can think of for finding out which class to use as a default implementation when all you have is an interface.  It doesn't 'bind' the implementation to the interface.  It neither 'binds' nor mentions the implementation.  The interface 'knows about' a factory.

I experimented with non-annotation ways of doing this, such as static blocks in interfaces, xdoclet-based solutions (well, I thought about xdoclet), but I decided they were all ugly.  I think what I have is the most elegant but pragmatic way.

Time for some code.

@DefaultFactory(BlobbyFactory.class)
interface Blobby
{
        void doSomething();
}

This says that BlobbyFactory is the default factory for the Blobby interface.

So when you want a Blobby, you instantiate BlobbyFactory and call its newInstance method.

As a convenience, I wrote a Farm class that you can use as follows:

Blobby blobby=(Blobby)new Farm().newInstance(Blobby.class);

The reason newInstance is not a static method is that you may wish to override the default factory, in some part of your application.

So the unit testing part of your application might want a mock object instead of whatever BlobbyFactory gives you (actually a BlobbyImplementation in this case).

Farm farm=new Farm();
farm.setFactory(Blobby.class,new MockBlobbyFactory());

then elsewhere:
Blobby blobby=(Blobby)farm.newInstance(Blobby.class);

In effect, a Farm is a Map with some nice methods for abstracting the rubbish away, and for using annotations when there is no map entry.

The code is all available at http://lavender.cime.net/~ricky/farm.zip
To build just what you need for using it, run 'ant farm.jar' and farm.jar will be produced.

It's released under the BSD licence, because I want you to be able to use it.  If there is some problem with my licence choice, let me know.

To build an executable example jar, run 'ant farm-example.jar' and farm-example.jar' will be produced.

The ant build is not very advanced, so you can work out how to do it using just javac if you don't want to use ant.

javac com/rickyclarkson/farm/*.java com/rickyclarkson/farm/example/*.java
java com.rickyclarkson.farm.example.Main

should work, but I haven't tested that.  (Use ant!)

This isn't just a Java design pattern, I've also implemented it in C just for fun.  Yes, I have a strange idea of fun.


Requirement Orientated Development
2005-04-06T18:27:00.000+01:00
For a long time I have thought that most software solutions are too far removed from the problem they attempt to solve.

The simple answer seems to be to identify the use cases and base the system around those.  However, normal software development seems to move away from the use cases, so I have created a new development model, called Requirement Orientated Development, or ROD, which aims to make the use case part of the code.

A system consists of a number of UseCases.  Each UseCase has a number of Actions which can take inputs and give outputs, and can do some processing.

A UserInterface (e.g., ServletUserInterface, SwingUserInterface, AtmUserInterface, UnitTestUserInterface) works out the possible Actions at any particular time and gives the user those to choose from.

I have not yet dealt with use cases which can loop, and branching is done with duplication, so it is still quite naive, but I aim to make this progress.

Am I reinventing old concepts, simplifying matters too much, or have I come across something worth following up?  Let me know your thoughts.

I'll post some code in another blog post sometime.


Lame Java Benchmarks - Freeing Memory
2004-10-11T21:54:00.000+01:00
Someone said that Java doesn't free up memory to the OS during a program's execution, so this is a simple test of that assertion.

import java.util.ArrayList;
import java.util.List;

public final class Main
{
        public static void main(final String[] args) throws InterruptedException
        {
                allocate();
                System.out.println(Runtime.getRuntime().freeMemory());
                System.gc();
                System.out.println(Runtime.getRuntime().freeMemory());
                System.out.println("Here");

                Thread.sleep(60000);
        }

        private static void allocate() throws InterruptedException
        {
                List list=new ArrayList();

                for (int a=0;a<5000000;a++)
                        list.add(new Object());
        }
}           


The output of running ps aux repeatedly on Linux while this is running shows that the garbage collection never has an external effect.  The output of the program is simply :

6207736
66317488
Here

This demonstrates that the garbage collection is happening.