According to the combined wisdom of the online pundits, Rich Hickey, the creator of Clojure, is set to become one of the most influential figures in technology within the next few years.

“I think programmers have become
inured to incidental complexity…
when they encounter complexity,
they consider it a challenge to overcome,
rather than an obstacle to remove.
Overcoming complexity isn’t work, it’s waste.”

Could he really become as well known as John McCarthy who created the Stanford Artificial Intelligence Project in 1964? Even before then, John had already made several towering contributions to the world of computing, including the LISP language, which became the standard for the AI researcher forty years ago.

The connection of course is that Clojure, (a pun on the closure programming construct) is an off-shoot or more correctly a dialect, of McCarthy’s creation. In Hickey’s case, it was a language designed for anyone wanting to write robust programs that are simple and fast.

So what was he trying to solve?

‘We suffer from so much incidental complexity in traditional OO languages, both syntactic and semantic, that I don’t think we even realise it anymore. I wanted to make ‘doing the right thing’ not a matter of convention and discipline, but the default. I wanted a solid concurrency story and great interoperability with existing Java libraries,’ says Hickey.

He originally targeted both the Java Virtual Machine and the Common Language  Runtime (CLR) but eventually decided  he wanted to do twice as much, rather than everything  twice, so  chose the JVM because of the much larger open source ecosystem surrounding it. It has proved to be a good choice.

One of the biggest challenges in developing Clojure Rich adds was getting the persistent data structures right, with sufficient performance such that Clojure could be a viable alternative to Java and C#. Without that, the project  would have stalled.

Hickey remains an independent software designer, consultant and application architect with over two decades  of experience in all areas of software development and has worked on scheduling systems, broadcast automation, audio analysis and fingerprinting, database design, yield management, exit poll systems, and machine listening, in a variety of languages. He lives in New York City.

RM:

Rich, plenty of people must have been tired of programming in C++/Java/C# as you did but simply put up with it. What was the trigger to develop Clojure? Was it a matter of waking up in the middle of the night and realising what the problem was?

RH:

One watershed moment for me was when I designed and implemented a new system from scratch in Common Lisp, which was a new language for me at the time, only to have the client require it be rewritten in C++, with which I was quite proficient. The port to C++ took three times as long as the original work, was much more code, and wasn’t significantly faster. I knew then that I wanted to work in a Lisp, but realized that it had to be client-compatible. I worked on bridging Common Lisp and Java, while doing functional-style programming for clients in C#, for a couple of years, then bit the bullet and wrote Clojure.

RM:

Why do you think those C++/Java and C# in particular don’t have enough built in technical or social wherewithal to grow easily and these somewhat painful growth processes?

RH:

There’s always a trade-off between change and stability, and I’m not sure that mainstream languages should evolve too rapidly. C# has been handling evolution pretty well. Java has been very slow. On the other hand, the stability and conservatism of Java is precisely what has engendered the huge ecosystem around it, so you really can’t fault it.

RM:

Lisps are traditionally extremely extensible so what prompted you to start with a clean slate?

RH:

Even Lisps are built on some primitive things, and those primitive data structures in traditional Lisps are mutable. I wanted the core data structures of Clojure to be immutable. In addition, I wanted the core algorithms to be based upon abstractions rather than concrete data structures as in traditional Lisps. These are things that can’t be retrofitted in a compatible way.

RM:

With multi-core CPUs becoming more common and a resurgence of hyper threading, dealing with concurrent tasks is now more important. How does Clojure deal with this?

RH:

First and foremost, Clojure makes it easy and idiomatic to program with immutable values, and the core library functions are pure. Immutable values and pure functions are relatively free of concurrency issues. Second, when state is required, Clojure supports a consistent model that separates state, identity and value, while supplying a variety of concurrency semantics.

RM:

What can you tell me about the support for parallelism and the upcoming Java Fork Join framework?

RH:

While concurrency focuses on coordinating multiple tasks, parallelism focuses on dividing up a single task to leverage these multi-cores to get the result faster. I didn’t build any low-level infrastructure for parallelism into Clojure since the Java concurrency experts were already doing that in the form of the Fork Join framework, a sophisticated thread pool and work-stealing system for parallel computation. As that framework is stabilising and moving towards inclusion in Java 7 (and usable with Java 6), I’ve started implementing parallel algorithms, like mapping a function across a vector by breaking it into subtasks, using Fork Join. Clojure’s data structures are well suited for this decomposition, so I expect to see a rich set of parallel functions on the existing data structures – for example , you won’t have to use special ‘parallel’ data structures.

RM:

How much does a choice of language really matter? Are there good reasons to choose one language over another or does it all come down to taste?

RH:

I think it matters quite a bit. A good language is opinionated, and strives to make a particular style of programming easy and idiomatic. It only seems a matter of taste when you are comparing languages that are more similar than they are different, like Java/C# or Python/Ruby. Try something really different like Clojure, Haskell, or Erlang and the fact that it matters becomes readily apparent.

RM:

What languages have you used seriously?

RH:

C, C++, Java, C#, Common Lisp, Clojure

RM:

Do you think languages are getting better? Is it easier to write software now because of the advances we’ve made?

RH:

As you’ve noted, existing languages change slowly, if at all. Are we getting better languages? Many of the hot ‘new’ languages like Haskell and Erlang are in fact pretty old. Clojure builds on Lisp, which is really old. What is new is the willingness to utilize these languages in mainstream (or, at least, internet) production environments. And that is a great thing, as we have largely maxed out the expressive capabilities of the C++/Java/C# languages, as evidenced by the prominence of XML and IDEs. The procedural/mutable execution model of those languages may also be on a collision course with multicore.

RM:

And what are the problems that are making it more difficult?

RH:

There will always be social and organizational resistance to change, and there should be, as that mitigates risk. More insidious are attempts to commoditize programming or programmers, which will leave us with mediocre technology, in a world where technical agility becomes more business-critical every day. Another hurdle for truly new or different languages is their (in)ability to deal with legacy systems and code. Without a good story there, it is quite difficult to get any adoption.

RM:

There’s a school of thought that says that the only way to manage complexity is to keep things, including programming languages simple. Does Clojure manage that?

RH:

Complexity certainly is a prime concern of mine, and I think Clojure does pretty well. At its core, Clojure is quite simple. And Clojure focuses on reducing both apparent and incidental complexity, the latter being complexity arising not from the problem domain but the solution domain – our languages and tools. I think programmers have become inured to incidental complexity, in particular by confusing familiar or concise with simple. And when they encounter complexity, they consider it a challenge to overcome, rather than an obstacle to remove. Overcoming complexity isn’t work, it’s waste.

RM:

Are there features that can make programmers more productive? You’ve designed a language so you obviously have an opinion about this.

RH:

Yes! Productivity comes from being able to focus. Interactive development environments (REPLs etc) help you focus, by letting you  try things immediately. At the language level, expressivity helps you focus, by moving your code as close as possible to the problem statement. At the language and library level, minimizing incidental complexity helps you focus on domain complexities only. And finally, being able to reuse existing code in the non-application-specific parts of your program facilitates focusing your efforts on the problem at hand.

RM:

Do you think that given the constraints Bjarne Stroustroup set himself with C++, which was to make an object oriented language that would be fully backwards compatible with C was a fatal flaw because it’s a set of difficulties that can’t be overcome? After all C has a corrupt type system.

RH:

I think C++ largely succeeded in its objectives, and still has appropriate application domains today, albeit fewer than in the past. The one expressed objective in which it has had less success, in my opinion, is as a library language. The lack of standard automatic memory management is a significant obstacle to independently developed, interoperable libraries.