Open a Portal Closer to Your Data

A key value proposition of ClojureScript has always been that it is Clojure. In this blog post we explore how true that statement is by trying to port the UI of Portal to run server-side to solve some of the more challenging aspects of Portal's RPC-based architecture.

Before we get into the how, let's start with the why. Since its introduction, Portal has had a typical modern web architecture, a rich JavaScript client (written in ClojureScript) connected to a server via WebSockets. Your data structures are serialized and shuttled through the socket to various reagent components to help visualize your data.

On the surface, it appears to be a reasonable approach, and at first I thought it was. However, a key challenge with this approach has been finding a good out-of-process representation of arbitrary in-process objects. There is a whole layer that exists on both the server and the client dedicated to this problem.

The challenge of a good representation is multi-faceted:

• It should work for any object a user may want to inspect.
• It should link back to the original runtime object.
• It should work for objects that may produce invalid edn via pprint.
• It should allow for writing simple viewers.
• It should be fast to produce and consume.
• It should support multiple runtimes.
• And most importantly, it can never fail!

These requirements led me down the rabbit hole of building CSON. It addresses many of these challenges, but that implementation is far from simple. The Portal runtime uses CSON in conjunction with a bunch of state on the server for tracking object references and watching atoms. Moreover, the server needs to inform the client when an atom has changed, and the client needs to inform the server when it no longer cares for a particular value. Don't even get me started on doing all of this stuff concurrently (which I have failed at several times now)!

With the resurgence of interest in thinner clients that do more work on the server via tools such as htmx and data-star, I got to thinking about what Portal would look like with a similar architecture. I could completely bypass the challenge of a good out-of-process representation if the UI lived in the same process! Sometimes the best solution is to solve a simpler problem. However, as we all know, simple is not always easy.

The Portal UI consists mostly of reagent views, which is a minimalistic ClojureScript interface to React.js, and React.js is a JavaScript library. Moreover, the UI took years of work and represents about half of the code in the Portal repository. I didn't want to rewrite the exact same UI in a new framework just so I could run it on the server. Also, I didn't want to sign up for maintaining two independent versions of the Portal UI. I just wanted to run the UI closer to the data it cared about. Since that didn't seem doable at the time, I didn't dig much deeper.

Then I watched the talk From Tomorrow Back to Yesterday: A Tale of Two Web Architectures by David Yang and got bit by the SSR bug again. The talk described how David's team went from a client-centric architecture to a server-focused one after facing their own challenges with rich clients. He described how HTML elements would receive UUIDs that would be used by server-side event handlers. And how events would be collected into a queue and processed by a hiccup-based render loop. The loop would be capped at 60Hz, similar to how a video game would implement vertical sync. After the talk, I was still uncertain about how to port my reagent components, but the idea of a SSR version of Portal seemed far more tangible.

A few months after the talk, I had a wild idea: what if I had a version of reagent that I could use on the server? Wouldn't that just solve all my problems? That was the biggest blocker in running the Portal UI code on the server, as most of it was just common Clojure and renaming the files from .cljs to .cljc is trivial.

It didn't need to be a full implementation, just enough to get the Portal viewers working on the server. I could start small, with basic reagent components that returned hiccup, and slowly build it up with a set of unit tests to keep my confidence up as I went. Then I could add hooks like use-state and use-effect, as the Portal UI code makes good use of React hooks. So I got started, and before I knew it, I had something that might actually run Portal's UI code.

Now I knew my toy implementation of reagent probably had plenty of issues, so I started with a small yet challenging subset of Portal's UI code. I slowly took bits out of portal.ui.inspector since I knew that if I could debug my way through it, getting other parts working would be easier, and that the project was worth pursuing. As I found issues, I would update my unit tests to include edge cases I hadn't considered, and little by little I found myself with more and more of portal.ui.inspector working.

Over the course of a few weeks, I was able to get the majority of Portal's UI code rendering. For the features that couldn't be moved to the server, such as visibility detection and window scrolling, I borrowed the approach from David Nolen's A ClojureScript Survival Kit Conj talk and used Web Components! If you want all the details, you can view them here.

When I was happy with rendering, I moved to implementing a minimal harness for the render loop. It looks very similar to how David described his architecture, but leverages Portal's existing infrastructure so that porting SSR to other runtimes is easier.

After this point, there were still a few minor problems. Portal needed a thin client to receive the output from the render loop and resolve JavaScript dependencies for the Web Components. And with that, the hard parts were proven out and things were starting to come together.

It was a pleasure seeing the Portal UI slowly come to life via this new SSR implementation and watching how seamlessly Portal's viewers ran on the JVM. This code was never intended to run on the JVM, and yet here it was, showcasing how portable Clojure is and how well ClojureScript implements Clojure.

Now, as a Portal user, you may wonder, how difficult it is to get the SSR version of Portal running? That's the best part! As of 0.64.0, it's as easy as the following:

(require '[portal.api :as p])
(p/open {:mode :ssr})
;; use portal.api as you normally would

Based on what features you use in the Portal UI, you may find it exceedingly challenging to tell the RPC and SSR versions apart. If you can't, I would call this port a success!

The RPC version of Portal isn't going anywhere, and will remain the default to preserve backward compatibility. Ideally, since they share much of the same code, improvements to either mode would improve the other. Eventually, perhaps the RPC version can be simplified since SSR can be used when a deeper integration with your runtime is required.

SSR also enables UI extensibility from the comfort of your REPL runtime, without the challenge of managing multiple runtimes. This should reduce the barrier to entry for implementing and sharing custom viewers, but only time will tell.

My priorities moving forward for Portal are to fix issues as they are encountered, but also to explore what's possible now that wasn't before. For example, getting infinite sequences would be substantially simpler for SSR, but wasn't part of the initial scope.

If you are feeling inspired or have any interesting thoughts you'd like to share, please leave a comment below, or join the conversation on the Clojurians Slack #portal.