Deraen's blog

Hello World – Building a blog using Boot

2015-07-04T00:00:00Z

I've been planning on starting a blog for some time already. But as I love hacking with build tools and such I usually spend some days trying a blog generator before deciding it's not good enough and start writing a new one, without ever finishing anything. Thus I was happy when I saw Perun which does pretty much everything I want:

Modular design, easy to extend
No forced metadata schema
~~RSS~~ Atom feed
Reads markdown
Templating using Hiccup

Perun

While working on a project I like to constantly see what is the result of the code I'm working on. On ClojureScript I use Figwheel. Thus I wanted the same experience for working on blog, though this is not that useful when writing it's useful while setting up the blog. To make this workflow possible with Perun a two changes were implemented:

Fast rebuilds

Parsing the markdown files and their metadata into Clojure data takes relatively long. If the file has not been changed that work is unnecessary. Boot's watch task and fileset provide an easy way to see which files changed since the last build. Perun's markdown-task which parses the files uses this to read only changed files and merge the changed metadata to existing metadata from previous builds. This provides build times of around 100ms when post content has been changed.

Clojure changes

Templating in Perun is done from Clojure using Hiccup. To automatically render the changes whenever the clj files change the render tasks use Boot pods to run the code in fresh environment. This way the Clojure namespaces are reloaded after the changes and changes are instantly seen. Because all namespaces required by the render namespaces has to be loaded when files change, it usually takes around 10 seconds to build the site after clj file change.

Alternative approach would be to use tools.namespace to reload the changed namespaces. That should be quite a bit faster at the expense of simplicity. This approach is used by boot-garden.

Extendability

As mentioned Perun should be easy to extend. This is achieved by implementing Perun using multiple Boot tasks which can be composed. Below is a example from this blog's build.boot file which shows some examples of the extendability. The tasks can be categorized in three types:

Tasks which read metadata from files, currently only markdown (1)
Tasks which manipulate the metadata (2)
Tasks which render some output (3, 4)

(deftask split-keywords []
  (boot/with-pre-wrap fileset
    (->> fileset
         (perun/get-perun-meta)
         (perun/map-vals
           (fn [{:keys [keywords] :as post}]
             (if (string? keywords)
               (assoc post :keywords (->> (string/split keywords #",")
                                          (mapv string/trim)))
               post)))
         (perun/with-perun-meta fileset))))

(deftask build
  [p prod bool "Build rss, sitemap etc."]
  (comp (less :source-map true :compress prod)
        ;; 1
        (markdown)
        ;; 2
        (if prod (draft) identity)
        (slug)
        (permalink)
        (split-keywords)
        ;; 3
        (render :renderer 'blog.views.post/render)
        (collection :renderer 'blog.views.index/render :page "index.html")
        (collection :renderer 'blog.views.tags/render :page "tags/index.html")
        (collection :renderer 'blog.views.atom/render :page "atom.xml")
        ;; 4
        (if prod (sitemap :filename "sitemap.xml") identity)
        (if prod
          (rss :title "Blog"
               :description "Deraen's blog"
               :link "http://deraen.github.io")
          identity)))

The task split-keywords is a task which manipulates the metadata by splitting keyword strings. Though I now see that I should instead just define the keywords as arrays in YAML metadata of the posts.

In this example there are five tasks which output files. A task which renders all the posts, two tasks which render a collection view and tasks for RSS and sitemap. The second collection task collects a list of all of tags (keywords) in the posts and creates a tag cloud out of those, what is cool here is that to create a tag page no special task was needed as the render function can itself do the necessary work (one reduce).

Render tasks do not need to necessarily output HTML, they can produce string in any way which is then written to the file. collection-task can for example produce HTML using Hiccup or XML using data.xml.

HTML and CSS livereload

Figwheel style ClojureScript development is already possible with Boot using boot-reload but that is heavily built for ClojureScript use cases. The client is written in ClojureScript so a built step is required. ~~Also boot-reload doesn't handle HTML file reloads which are the essential for being useful for static page development.~~ Boot-reload does handle HTML reloads.

Previously I've been using LiveReload.js with Gulp.js so I thought that should be a good fit for Boot also. Luckily a Clojure implementation of LiveReload already existed. It required some API changes to make it generally usable library. After the changes it was a breeze to create a boot task.

Cool features and possibilities

As it is easy to manipulate the data between reading the files and rendering there is great opportunity to move some features which are usually implemented in the browser to build process. The value I see there is that RSS readers (AFAIK) do not execute JS so those readers are left without e.g. code highlighting.

Another possibility is to hyphenate the text. This is even more useful as the hyphenation is relatively expensive operation so it's great if it only needs to be executed once instead of each page load. For this reason I already started writing clj-hyphenate which implements Franklin M. Liang's hyphenation algorithm in Clojure. The algorithm is the same as used by TeX, LibreOffice and Hyphenator.js. To hyphenate HTML it is possible to insert soft-hyphens into the text which the browser only shows if the word needs to be split into multiple lines.

Boot ClojureScript tooling updates and what's up next

2015-10-04T00:00:00Z

One of the features missing from Boot ClojureScript tooling in comparison to Figwheel was heads-up display (HUD). Thanks to Martin Klepsch this is now implemented in Boot-cljs and Boot-reload. I've made a screen cast about the new feature so check that to see how it works. Read on for some details about the implementation and to see what's up next for Boot ClojureScript tooling.

Demonstration

Implementation

As Boot ClojureScript tooling consists of multiple separate tasks, implementation of HUD required changes to two tasks:

Boot-cljs

Implementation of HUD requires that information about ClojureScript warnings and exceptions is available for sending to the client.

To catch the information about Cljs warnings we'll set up custom warning-handler. The handler with both (1) print the warning to console and (2) store the warnings in an atom. The reason why we overwrite default warning handler which would also print the warnings to console, is that we want to process file-path of the warning before printing. Because ClojureScript sees to source files at Boot temporary-directories the file paths include the temporary directory path. To make the warnings cleaner we (3) retrieve path for the original file. Data about warnings is attached to .cljs.edn file metadata on the fileset.

(fn [warning-type env extra]
  (when (warning-enabled? warning-type)
    (when-let [s (ana/error-message warning-type extra)]
                 ;; 3
      (let [path (util/find-original-path source-paths dirs ana/*cljs-file*)]
        ;; 1
        (butil/warn "WARNING: %s %s\n" s (when (:line env)
                                           (str "at line " (:line env) " " path)))
        ;; 2
        (swap! warnings conj {:message s
                              :file path
                              :line (:line env)
                              :type warning-type})))))

Handling exceptions is a bit more trickier because ClojureScript compiler is running inside a pod. Usually the communication between pods happens using pr-str and read-string, very similarly to how Leiningen communicates between multiple JVMs. But this doesn't happen with exceptions, they are instead directly thrown. The problem here is that for some reason when exceptions are thrown from one classloader to another, they lose their ex-info metadata. For ClojureScript exceptions the metadata contain all the interesting data: file-path, line number and column number.

A workaround I found for this is to manually serialize and deserialize exceptions, including metadata, stack-trace and cause stack. This way it's possible to throw exception with correct metadata from Boot-cljs to Boot-reload.

As with warnings, file paths in exceptions are changed to contain path to the original file instead of to a file in temporary directory.

Boot-reload

Boot-reload will either read warnings from .cljs.edn file metadata on the fileset or catch the exceptions thrown by Boot-cljs. Because Boot tasks are implemented using middleware pattern it's simple as just calling next-task inside try-catch. Boot-cljs will tag the exceptions so that we can display only the (1) interesting exceptions on the browser. All exceptions are rethrown so that other tasks can access the exception data and to tell that the build failed.

(try
  (next-task fileset)
  (catch Exception e
        ;; 1
    (if (= :boot-cljs (:from (ex-data e)))
      (send-visual! @pod {:exception (merge {:message (.getMessage e)}
                                            (ex-data e))}))
    (throw e)))

The heads-up display user interface is implemented purely using Google Closure library. This keeps the build simpler as we don't need any additional dependencies. Even though we are not using sophisticated framework like React, the user interface implementation is only about one hundred lines, including CSS definitions. UI is even implemented using immediate mode rendering: Whenever new :visual message is received from the server, old DOM container is removed and a new one is created.

Next up

Boot-reload fixes

Current file-reloading implementation has some problems when one has multiple ClojureScript builds in one project. Boot-reload tries to load all changes JS files in browser but it's possible that the files don't belong to the open application and cause problems.

Google Closure library defines a (private) dependency graph of namespaces and it should be possible to use that to determine if changed file is required by any loaded namespace, if it's not, we don't need to load the changed file.

The same dependency graph can be used to sort the changed files in dependency order. Currently Boot-cljs is calculating this dependency order and passing it to Boot-reload, this is additional work as ClojureScript compiler has already done this and passed the data to Closure.

Figwheel is already using Closure dependency data so I'll be looking on it's implementation and copying relevant parts to Boot-reload.

Boot-cljs-repl fixes and improvements

I have now working ClojureScript REPL setup with Vim-fireplace so I'll be fixing problems as I encounter them.

Boot-cljs performance (fileset performance)

There have been multiple reports of Boot-cljs being slower than Leiningen Cljsbuild or Figwheel. On most cases the ClojureScript compiler works just as fast, but Boot filesets cause some overhead which shows especially with incremental recompiles. Performance profiling should help to find the bottlenecks. Improving this should help all Boot tasks.