A sampler of ClojureScript application designs

2013 June 24 Kevin J. Lynagh

I’ve found Clojure on the JVM to be a great language for expressing data transformations. However, I’ve had trouble expressing application and view-manipulation logic on the client with ClojureScript as naturally as I can express data transformations in Clojure on the server. One could argue that Clojure—a functional language with immutable data structures—is inherently unsuited to building interactive applications. After all, what is a rich clientside application if not a giant pile of side effects: drawing to the screen, handling user inputs, sending and receiving messages, and mutating state on remote servers via POST requests?

However, I’ve come to appreciate is that Clojure is more than just a functional programming language, and that ClojureScript is more than just a bunch of Lispers moving their magical sand box onto another runtime so they don’t have to deal with a distasteful reality. Clojure is, above all else, a pragmatic language. While Clojure has opinions—functional programming, immutable data structures, dynamic typing, macros, runtime polymorphism, platform interoperability—which of those opinions you leverage is ultimately your choice.

This post outlines several approaches I’ve used over the past 18 months to build rich, clientside applications with ClojureScript. I’m not claiming that one approach is more idiomatic than another or that one is, god help us, a best practice. These are just some benefits and tradeoffs as I see ‘em.

The problem

The problem of building a single page, clientside web application factors into several orthogonal concerns. Two of the largest, which I will focus on in this article, are rendering views and managing state. When users click buttons or new information is received from a server, the state of the world changes. The application must recalculate derived state and redraw components on the screen. Ideally, the state and views sync after a minimum number of calculations and redraws. (Extra emphasis on “minimum” for mobile devices, since the hardware is limited and DOM access is particularly slow.)

Typical JavaScript approaches

A great many JavaScript frameworks have been written to solve these problems (see the excellent TodoMVC Rosetta stone). The most popular frameworks address these concerns by factoring an application into some variant of the Model-View-Controller architecture. Typically, the view component relies on some flavor of string-based templating (e.g., Handlebars). The template is a function of arbitrary properties that returns a string of HTML which is is injected into the DOM. For example:

weather_station_template({name: "Portland", region: "Oregon", tempC: 11})
//=> '<li class="station">
//      <span class="city">Portland</span>
//      <span class="region">Oregon</span>
//      <span class="temperature">11</span>
//    </li>'

To update, the view component either explicitly manipulates the live DOM nodes to reflect new underlying data or the DOM subtree is replaced wholesale by setting innerHTML to a string returned from a new template invocation.

Application state changes are usually propagated via callbacks or an event system. For instance, after receiving forecast data from the server a weather app might need to do three things:

The easiest solution is to just perform these three operations directly in the ajax callback. This tight coupling quickly leads to trouble in all but the simplest application, so typically you’d decouple via an event system. For instance, our three concerns could be split across three functions, all subscribed to the forecast-changed event. When new data is received from the server, the ajax callback fires a forecast-changed event and the three functions execute independently.

New dog, same tricks

That’s the gist of many current JavaScript solutions; before delving into alternative designs, I want to note that these JavaScript patterns can be used in ClojureScript. You can instantiate plain JavaScript objects and arrays, use set! to mutate their members in place, and invoke whatever native or library JavaScript functions you so desire. ClojureScript is built on top of the Google Closure JavaScript library and compiler, which includes (among many other things) a classical object system, a pubsub bus, and a string-based templating library. If you use them, they’ll be concatenated automatically with your application code; if you don’t then the Closure compiler will leave 'em out. Alternatively, you can ignore the Closure library, throw jQuery, Handlebars, and D3 into your <head> and go crazy.

That said, Clojure’s power comes from its opinions; rather than ignoring Clojure’s semantics entirely, it’s worth thinking about how to reconcile them with the world of JavaScript.

Vanilla Atoms

When I first started using Clojure, the most jarring change coming from other languages was immutability. All of Clojure’s data structures are immutable. You don’t add a key to a map, you create a new map that consists of the old key/value pairs and the new key/value pair:

(let [m {:k "value"}]
  (assoc m :k2 "val2") ;;=> {:k "value", :k2 "val2"}
  m ;;=> {:k "value"}
  )

If it seems impossible to get anything done this way, you’ll just have to trust me that it’s quite nice once you get the hang of it. (If you’re haven’t seen Clojure before, you may want to play around in an online REPL to familiarize yourself.)

Of course, some problems inherently require mutable state. E.g., the current weather conditions or the current screen at the foreground of an app. Clojure provides several reference types for dealing with mutable state and coordinated change. Since JavaScript runtimes don’t have shared memory threads, ClojureScript implements only one reference type, the atom. An atom is a reference to an immutable value; the value is retrieved by dereferencing the atom (the @ operator) and the reference can be updated with the reset! or swap! functions:

(def a (atom {:k "value"}))
@a ;;=> {:k "value"}

(reset! a {:k2 "val2"})
@a ;;=> {:k2 "val2"}

(swap! a assoc :k3 "val3")
@a ;;=> {:k2 "val2", :k3 "val3"}

Note that all of these values are immutable—we can hold onto the {:k2 "val2"} from the second line and do whatever we want with it. Calling swap! on the third line changes the reference a to point to the map {:k2 "val2", :k3 "val3"}; the map from the second line is untouched. There is never any need to “deep copy” or defensively clone a value, because Clojure values are values—they are immutable.

Watcher functions can be attached to atoms, which are called with the old and new values of their atoms whenever a reset! or swap! against the atom is called. Thus, you can write very similar code in ClojureScript with atoms as you can with, e.g., a Backbone.js model’s .set() and associated change listeners. And just like event listeners in JavaScript, ClojureScript watchers are just callbacks, so they must side-effect somehow to be of any use.

As soon as you move beyond a trivial application, you’ll need to make a decision: retain all state within a single atom (i.e., storing a map) or juggle multiple atoms. If you store a complex value within a single atom, then it’s much easier to avoid glitches (inconsistent state). Indeed, all of the mutable state for your application must be stored in this atom, which means it’s simple to print out for debugging or stub out during testing. The downside is that all of the watchers and manipulators of the atom inherit the complexity of manual namespacing and granularity. Instead of using reset! you’ll tend to update the state like

(swap! app-atom update-in [:my :subsection :of :app] update-subsection-fn)

and side-effect (i.e., draw to the screen) like

(add-watch app-atom :redraw-subsection
           (fn [_ _ old new]
             (when (not= (get-in old [:my :subsection :of :app])
                         (get-in new [:my :subsection :of :app]))

               (do-stuff! new)
               (do-more-stuff! new))))

Furthermore, because there is only one atom, you must balance the performance hit of running extra (potentially unneeded) watcher functions on every state change against the code-complexity hit of checking subsections of state to determine if non-indempotent code needs to be run (as in the example above’s (when (not= ...)) line).

The alternative to keeping all state within a single atom is to logically partition across multiple atoms within multiple ClojureScript namespaces; having one atom per screen or per logical grouping of controls/settings. This feels more natural to those coming from languages with permiscuous state (i.e., most other languages), but opens its own can of worms. Since the state is now spread out across the application, it’s more difficult to print or stub out. If you want to enforce some invarant that depends on multiple atoms, then you must carve out functions that can be called from multiple watchers. For instance, say you’re building a weather app; you’ve got two atoms:

(def current-weather
  (atom {"Portland" {:temp 32 :condition "rain"}
         "Seattle" {:temp 29 :condition "clear"}}))

(def selected-station (atom "Portland"))

and the information drawn to the screen with an update-dom! function depends on both (it’s idomatic clojure to add a bang suffix to side-effecting functions, but the bang character has no special language semantics). You could add two watchers explicitly,

(add-watch current-weather :redraw
           (fn [_ _ _ _]
             (update-dom! (get-in @current-weather @selected-station))))

(add-watch selected-station :redraw
           (fn [_ _ _ _]
             (update-dom! (get-in @current-weather @selected-station))))

factor out a single, no-arg watcher function,

(defn update! []
  (update-dom! (get-in @current-weather @selected-station)))

(add-watch selected-station :redraw update!)
(add-watch current-weather :redraw update!)

or (since the bodies are identical) loop using doseq:

(doseq [a [current-weather selected-station]]
  (add-watch a :redraw
             (fn [_ _ _ _]
               (update-dom! (get-in @current-weather @selected-station)))))

You can probably imagine how quickly this becomes nasty with invariants of more than two atoms.

Cleaning it up with macros

ClojureScript has compile-time macros, which we can use to clean up multi-atom code. See my Reflex library for the deets, but the short of it is that we can use a macro to automatically detect atom dependencies and setup the appropriate callbacks. One such macro that Reflex provides is constrain!, which runs a body of code for its side effects whenever an atom dereferenced within the body later updates. Our example would be rewritten as:

(constrain!
 (update-dom! (get-in @current-weather @selected-station)))

Props to the Knockout.js library and its computed observables, where I drew inspiration for Reflex.

Higher-level semantics for the DOM

With these primitives for tracking state, we can build a nice story around manipulating the DOM. At its simplest, the update-dom! function above would be implemented as

(defn update-dom! [new-model]
  (set! (.-innerHTML $element)
        (string-template-fn new-value)))

However, a wholesale innerHTML replacement can be problematic because it means:

Ideally we’d be able to update live DOM nodes in place, adding or removing nodes only if the underlying model calls for it (i.e., if we are rendering a collection and the new state has more or fewer items).

The D3.js data visualization library very nicely solves this problem for a subset of DOM-manipulations by enforcing an invariant for you: “children matching this selector should always correspond to these data under this mapping”. That is, D3 will create and remove DOM nodes automatically when new data are added and removed from the underlying data array, as well as update the attributes and styles of individual nodes as their corresponding data are updated.

However, it’s awkward to use D3 to manipulate complex DOM subtrees. The expressiveness of the library is limited by its jQuery-like chained manipulation syntax; there is always an implicit node or group of nodes under the side-effecting knife, but no syntatically clean way to express “different shapes” with varying styles and attributes on their nodes.

The C2 Clojure library extends D3’s declarative ideas beyond collections of single nodes to arbitrary DOM subtrees. Rather than a chained-syntax with an implicit target, C2 builds representations of DOM subtrees as explicit values—data structure literals and functions returning the same. In essence, you use standard Clojure idioms to build up a data structure representing the DOM you want, then hand it off to a low-level library and say “go make the live DOM look like this”. That low-level library will then walk the live DOM and add/remove/update nodes as appropriate. Essentially what you’re doing is decoupling the specification of what you want from the actual rendering. There are a number of benefits to this approach:

One of the nicest benefits, though, is the clear, declarative nature of the code. Here’s a snippet from a todo list implementation built in this style:

(bind! "#main"
  [:section#main {:style {:display (when (zero? (count @core/!todos)) "none")}}
   [:input#toggle-all {:type "checkbox"
                       :properties {:checked (every? :completed? @core/!todos)}}]
   [:label {:for "toggle-all"}
    "Mark all as complete"]
   [:ul#todo-list
    (unify (case @core/!filter
             :active    (remove :completed? @core/!todos)
             :completed (filter :completed? @core/!todos)
             ;;default to showing all events
             @core/!todos)
           todo*)]])

The bind! declares, “I want the #main DOM subtree to always correspond to the following body”. The body is a function of two pieces of state: !todos, an atom refering to a collection of todo list items (ClojureScript maps), and !filter, an atom referring to a keyword specifying whether to show completed, uncompleted, or all todo list items. Whenever those atoms change, the body is re-evaluated, resulting in a data structure corresponding the desired state of the DOM. A low-level library takes this data structure and manipulates the live DOM until the latter matches the former. (The todo* is a function that takes a todo-item map and returns a data structure representing that todo’s DOM subtree.)

This approach is nice, but there are at least three major problems:

  1. markup is tightly coupled with code,
  2. DOM-walking is slow, and
  3. automatic callbacks are great until they’re hell.

The first point may be a non-issue for you, depending on how happy your designers are to give up their usual tooling, learn some Clojure or JavaScript, and represent markup with data structures and function invocations rather than interpolated strings.

The second point may also be a non-issue; desktop browsers are much faster than human brains, so if your updates render in under 50–100ms you’re good to go. However, if you’re rendering more than a handful of nodes on a mobile browser, you’re going to blow past the perceptual limit and your app will feel sluggish.

I’ve found the third point to be the most damning. JavaScript is single threaded, so whenever an event is fired or a change is made to a watched atom the thread of execution is effectively hijacked by the listeners. This leads to inefficiences and glitches. For instance, consider

(ns glitch-demo
  (:use-macros [reflex.macros :only [computed-observable]]
               [c2.util :only [bind!]])
  (:require reflex.core))

(def a (atom 1))
(def b (computed-observable (+ 1 @a)))

(bind! "body"
       [:body
        [:span
         @b " = " @a "+ 1"]])

Here b depends on a and the bind! depends on both b and a. After the initial execution, the page will display “2 = 1 + 1”. If we execute (reset! a 5), the body of bind! will be executed twice—once by the callback watching a and again by the callback watching b. Depending on the order in which the callbacks execute, the DOM may show an invalid state; if the callback watching a reruns the body of bind! before b recomputes itself, then the page will say “2 = 2 + 1” immediately before “3 = 2 + 1”. Not only did we unnecessarily update the DOM, we were in an inconsistent state!

It gets worse, actually; there is nothing to prevent callbacks from making further changes to managed state and immediately executing more callbacks. This means it’s possible to get into hard-to-debug infinite loops.

Ultimately the problem with spreading mutable state across multiple atoms and keeping derived values (including the DOM) synchronized with callbacks is that vanilla callbacks cannot be coordinated. Sometimes they’re too fine-grained, leading to inefficency and glitches Other times the inability to selectively invoke watchers can lead to to infinite loops or unnecessary computations (e.g., calling (swap! an-atom identity) will invoke all of an-atom’s callbacks, despite the fact that its value hasn’t changed).

Dirty checking

I’m aware of two ways to deal with the granularity problems of callbacks: a fancy way and a simple way. The fancy way is to explicitly track the dependency graph and manage callback order via topological sorting. The papers and frameworks I’ve seen on this topic also extend the discrete-time model of vanilla callbacks with a continuous-time model of “behaviors”. I haven’t built any serious applications with these approaches, so I can’t comment on them. See Deprecating the observer pattern, FRTime, Flapjax, and Javelin; the latter two have todo list implementations.

The simple solution is to separate callbacks from single, mutable values and instead treat them as invariants against a coarser scope. Each callback is associated with a sentinel value (e.g., a single reference within the scope or a function of multiple references within the scope). Changes within the scope are explicitly signaled, which starts a “digest loop”. Callbacks are run each time their sentinel value changes; once all sentinel values have stabilized the digest loop is terminated. This approach is known as “dirty checking”, and is used by Angular.js (see Misko Hevery’s StackOverflow answer for more relative merits of dirty checking vs. callbacks).

The approach is conceptually simple, and a dirty checking mechanism could be easily written in ClojureScript and used in favor of atoms+watchers. However, Angular.js has several other nice features beyond dirty checking, like DOM data-binding, custom directives, and a dependency injection and testing system. Angular.js is also built on top of a coherent, explicit model (which is more than most clientside web frameworks can say). If we use Angular, we avoid the three problems of the previous DOM-walking-after-callbacks approach:

Rather than reinvent the wheel, we decided to see how far can we get using Angular.js from ClojureScript. But that’s another blog post.

Recap

To recap, we’ve examined two problems in clientside application development and several solutions for each:

State management

  1. event dispatch / callback graphs
  2. dirty checking

View rendering

  1. innerHTML clobberin’ string templates
  2. data structure + DOM-walking
  3. scoped references to live DOM nodes

In my JavaScript and CoffeeScript days, most of the applications I worked on used a combination of A. with 1. & 3. Although this is largely idiomatic, I struggled to avoid glitches and maintain the model <-> DOM mapping manually. In ClojureScript I’ve used A. and 2., which I find much easier to reason about. However, on complex, multi-screen applications and mobile devices the callback graphs became tangled and DOM-walking became too slow. This (and business factors around designer accessibility and developer availability) led me to my current path of using Angular.js from ClojureScript.

Finally, note that these solutions are largely orthogonal. For some applications the combination of, say, dirty checking and references to live DOM nodes may be a sweet spot. ClojureScript is very flexible, and the community’s preference for libraries over frameworks means that you can (and must) think about your specific application and choose solutions for each component sub-problem. I hope this article has shed a bit of light on some of those issues.

Moar info

Thanks

Thanks to Darrick Wiebe, Scott Becker, Adam Wulf, and Tom White for reviewing this blog post.