Porting Portal to Basilisp

Last year I watched the Basilisp London Clojurians talk by Chris Rink:

Basilisp is a Clojure-compatible(-ish) Lisp dialect targeting Python 3.9+. I had heard of it before but hadn't realized how mature of a Clojure implementation it had become. After the talk, I began to wonder how difficult it would be to port my visual tool Portal to this python based Clojure runtime.

Portal already had support for other alternative Clojure runtimes: node, babashka, nbb, joyride and clojure-clr. So I already had experience with this type of port, but I was familiar with many of those runtimes, except the CLR but it felt very similar to the JVM. This post will document my journey learning about python, basilisp and porting Portal. I hope this post sheds some light on how to get your Clojure code working on other platforms and demonstrates how portable Clojure truly is. If you see anything that is incorrect or could be improved, please let me know.

As I began my port, I was pleasantly surprised how easy it was to get a development environment setup. I started by trying to install Basilisp via:

pip install basilisp

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Looking at the output, I deciphered that I needed to setup a virtual environment. I had encountered the concept in passing before and knew it was best practice, but didn't realize it was enforced. So as the error message asserted, I setup my virtual env via:

python -m venv py
./py/bin/pip install basilisp

After getting basilisp installed, my second objective was to get an editor connected repl which was trivial since it has a build-in nREPL server which can be started via:

PYTHONPATH='dev:src:test' ./py/bin/basilisp nrepl-server

You can even automate starting of the nREPL server and editor connection for Calva with connect-sequences via the following VS Code workspace settings file:

{
  "calva.replConnectSequences": [
    {
      "name": "basilisp",
      "projectType": "deps.edn",
      "projectRootPath": [
        "."
      ],
      "customJackInCommandLine": "./py/bin/basilisp nrepl-server",
      "jackInEnv": {
        "PYTHONPATH": "dev:src:test"
      },
      "nReplPortFile": [
        ".nrepl-port"
      ],
      "menuSelections": {
        "cljAliases": [
          ":dev",
          ":cider"
        ]
      }
    }
  ]
}

Typically, my approach to porting is to start with building a remote client for Portal. That allows me to still leverage Portal as a development tool, just in a remote capacity. For clients, I tend to prefer minimal dependencies or zero dependencies when possible. When researching how to make a simple http request in python, I stumbled upon an example via urllib, which is perfect since that's built into python. Next is the question of serialization, which for Clojure is as simple as pr-str. I also found that basilisp has a built-in json library and added that option as well.

The Portal client turned out to look like this:

(ns portal.client.py
  (:require [basilisp.json :as json])
  (:import [urllib.request :as request]))

(defn- serialize [encoding value]
  (try
    (.encode
     (case encoding
       :json    (json/write-str  value)
       :edn     (binding [*print-meta* true]
                  (pr-str value)))
     "utf-8")
    (catch Exception ex
      (serialize
       encoding
       {:cause "Error"
        :message (ex-message ex)
        :data (ex-data ex)}))))

(defn submit
  ([value] (submit nil value))
  ([{:keys [encoding port host]
     :or   {encoding :edn
            host     "localhost"
            port     53755}}
    value]
   (let [req (request/Request
              (str "http://" host ":" port "/submit")
              ** :data (serialize encoding value))]
     (.add_header
      req "content-type"
      (case encoding
        :json    "application/json"
        :transit "application/transit+json"
        :edn     "application/edn"))
     (request/urlopen req))))

Now I can host a Portal instance on any of the existing runtimes and start throwing data at it over the wire with tap> via:

;; in the basilisp repl
(require '[portal.client.py :as p])
(def submit (partial p/submit {:port 9999}))
(add-tap #'submit)

With the remote instance being opened via:

;; in the remote process (jvm/node/clr/...)
(require '[portal.api :as p])
(p/open {:port 9999 :launcher :auto})

However, while pr-str is great for bootstrapping, it doesn't always guarantee that the produced output is readable EDN. The final client will also support Portal's custom .cljc based serialization format, but that will have to wait until after it is also ported.

After getting a client working, I continued with getting some tests passing, typically runtime tests, and with Basilisp it was no different. I started with portal.runtime.fs specifically, as most runtimes share slightly different but comparable filesystem APIs. Since the surprises should be minimal, this also affords me the opportunity to start getting comfortable working in this new and unfamiliar environment.

Of the existing fs APIs I've used, the python one felt the most like node.js. To give you a sense of what this code looks like, here is the basename fn:

(defn basename [path]
  #?(:clj  (.getName (io/file path))
     :cljs (path/basename path)
     :cljr (Path/GetFileName path)
     :lpy  (os.path/basename path)))

One thing I noticed in Basilisp is the absence of clojure.test/run-test, but since a deftest will emit a var, I've been invoking the test vars directly at the REPL. I think for now, testing in Basilisp is focused around pytest. By the end of this port, I ended up implementing portal.test-runner, which has closer semantics and output to run-test. Perhaps in the future, something like this will exist natively in Basilisp.

After googling around a bit on how to do various filesystem things in python, the tests were passing and I was ready to tackle something more challenging.

As mentioned above, Portal implements a custom serialization protocol CSON to communicate values to the UI client. Portal initially used transit-clj / transit-cljs which worked very well, but that was later replaced. Since Portal is primarily used as a library, transitive dependency versions are resolved by the user. This can sometimes be older versions which contain bugs that were fixed or missing features that Portal needs to work. So in order to provide stability for users, Portal opted to implement its own serialization. The upside of this decision has been the ability to easily port the protocol to other Clojure runtimes.

CSON is implemented via the following internal protocol where buffer is flat list that will eventually be converted to a JSON array so any value added to the buffer must to be a valid JSON type:

(defprotocol ToJson (to-json* [value buffer]))

;; CSON implementation for keywords
(extend-type #?(:clj  clojure.lang.Keyword
                :cljr clojure.lang.Keyword
                :cljs Keyword
                :lpy lang.keyword/Keyword)
  ToJson
  (to-json* [value buffer]
    (if-let [ns (namespace value)]
      (-> buffer
          (json/push-string ";")
          (json/push-string ns)
          (json/push-string (name value)))
      (-> buffer
          (json/push-string ":")
          (json/push-string (name value))))))

The complete implementation for the various Basilisp Clojure types looks very similar to the above code. And as you can imagine, the parsing is simply the inverse. However, while the implementation is compete, I haven't spent time digging into the performance, and the Basilisp version of CSON appears to be the slowest:

After serialization is complete, things will typically ramp up in difficulty, as the next step is getting an http server with websockets up and running. For Basilisp specifically, it made me directly confront the single threaded nature of python and learn the abstractions / APIs / libraries in the python ecosystem. I settled on aiohttp as the http server library and subsequently asyncio for dealing with asynchronous io. This code can be found in portal.runtime.python.launcher and portal.runtime.python.server.

An interesting implementation detail of the Portal http server, is how requests are routed. Since Portal aims to be a multi-runtime application, early on I chose to go with the most portable solution I could and settled on defmethod. This may seem like an odd solution, but with Portal's goals in mind, I think it makes sense. The code looks like:

(defmulti route (juxt :request-method :uri))

(defmethod route :default [_request]
  {:status 400})

...

(defmethod route [:get "/main.js"] [request]
  {:status  200
   :headers {"Content-Type" "text/javascript"}
   :body
   (fs/slurp
    (case (-> request :session :options :mode)
      :dev (resource "portal-dev/main.js")
      (resource "portal/main.js")))})

...

One outstanding issue I haven't been able to solve with the server is graceful shutdown of the http server on portal.api/stop. It typically works, but when it doesn't the python process will not exit. This tends to happen consistently on OSX and sometimes on CI.

The good news is that once I got comfortable with async/await in Basilisp, I was able to get a mostly working server implemented for the UI.

Once the server was up and running, I could load the UI and start viewing data from my Basilisp runtime. One namespace I enjoy using from Portal is portal.console. It provides a few logging functions that send values to Portal, just like tap>, but also includes extra data such as source location. Additionally, it will tag the specific runtime the where data came from, in this case it would be python, so I updated :portal.viewer/log to support that type of runtime. Here is a preview of that viewer:

Once I ironed out all the bugs in the runtime code, and made sure it worked well with the Portal UI, I was ready to build and deploy a package. For this, I noticed that Basilisp itself uses poetry so I decided to do the same. I setup the following pyproject.toml:

[project]
name = "djblue.portal"
version = "0.60.1"
license = { file = "LICENSE" }
readme = "README.md"
description = "A clojure tool to navigate through your data."
authors = [
    { name = "djblue", email = "djblue@users.noreply.github.com" },
]
keywords = [ "clojure", "basilisp", "inspector", "portal", "datafy", "nav" ]
requires-python = ">= 3.8"
dependencies = [
    "aiohttp>=3.11.11",
    "asyncio>=3.4.3",
    "basilisp>=0.4.0",
]

[tool.poetry]
packages = [
    { include = "portal", from = "src" },
]
include = [
    { path = "resources/portal/*", format = [ "sdist", "wheel" ] }
]

[project.urls]
Homepage  = "https://github.com/djblue/portal"
Standalone = "https://djblue.github.io/portal/"
Documentation  = "https://cljdoc.org/d/djblue/portal/"
Issues = "https://github.com/djblue/portal/issues"
Changelog = "https://github.com/djblue/portal/blob/master/CHANGELOG.md"

[build-system]
requires = ["poetry-core==2.1.3"]
build-backend = "poetry.core.masonry.api"

An interesting bit here is the usage of `include = [ ... ]` to bundle the Portal UI client in the distribution artifact. Moreover, I also discovered importlib.resources which is super useful for resolving static resources on your path, the same way you would do for java and the classpath.

While this initial port represents the main pieces that need to be there to use Portal, there are still some features that are missing but can be completed in future releases. Things such as a nREPL middleware, support for launching editor integrations and improved CSON performance.

Now with all of those details sorted out, I was able to deploy a version of Portal to pypi and now users can follow this simple guide to get started with Basilip and Portal!