January 2023 Project Updates

By Kathy Davis

Our first project update for 2023 includes reports for Clj-Kondo, Clojure Data Cookbook, ClojureDart, Mathbox, and Tablecloth. Thanks to all. We’re off to a great start!

Project Clj-Kondo: Michiel Borkent

In this post, I’ll give updates about my open source during November and December 2022. But first off, I’d like to thank all the sponsors and contributors that make this work possible! Top sponsors:
Clojurists Together
Roam Research
Nextjournal
Toyokumo
Cognitect
Kepler16
Adgoji

If you want to ensure that the projects I work on are sustainably maintained, you can sponsor this work via the following organizations. Thank you!
Clojurists Together
Github Sponsors
OpenCollective
(also see the clj-kondo one)

Here are my updates for the Clj-Kondo projects:

http-client

The new babashka http-client aims to become default HTTP client solution in babashka. Babashka has several built-in options for making HTTP requests, including: babashka.curl; http-kit; java.net.http

In addition, it allows to use several libraries to be used as a dependency: java-http-clj; hato; clj-http-lite;

The built-in clients come with their own trade-offs. E.g. babashka.curl shells out to curl which on Windows requires your local curl to be updated. Http-kit buffers the entire response in memory. Using java.net.http directly can be a bit verbose. Babashka’s http-client aims to be a good default for most scripting use cases and is built on top of java.net.http and can be used as a dependency-free JVM library as well. The API is mostly compatible with babashka.curl so it can be used as a drop-in replacement. The other built-in solutions will not be removed any time soon.

Babashka

Native, fast starting Clojure interpreter for scripting.

I had the honor to write a guest blog post for the GraalVM blog about babashka. You can read it here. Daniel Higginbotham from Brave Clojure wrote Babashka Babooka which I helped reviewing. I also wrote a blog on how to test babashka scripts. Versions 1.0.165 - 1.0.169 were released. Visit the changelog for details. Highlights:

Squint and Cherry

Squint and cherry are two flavors of the same CLJS compiler. Squint is a CLJS syntax to JS compiler for use case where you want to write JS, but do it using CLJS syntax and tooling instead. Cherry comes with the CLJS standard library and is as such much closer to the normal ClojureScript, but the minimal amount of JS is a little bigger.

In the past two months, I’ve been restructuring code between squint and cherry: a bit boring but necessary to keep going forward. Along with some minor bugfixes and features, one new JSX feature landed: you can pass a props map using a new notation inspired by helix. You can read details in the changelog.

The video of the talk I did on ClojureDays 2022 came online!

SCI

Configurable Clojure interpreter suitable for scripting and Clojure DSLs. This is the workhorse that powers babashka, nbb, Joyride, and many other projects. Many improvements have happened over the last two months, both in Clojure compatibility and performance. JS and JVM interop has become up to 5x faster. All of these changes benefit babashka, nbb, joyride, etc. See changelog for more details.

Clj-kondo

Static analyzer and linter for Clojure code that sparks joy… Two new releases with many fixes and improvements. Check the changelog for details. Some highlights:

Scittle

Execute Clojure(Script) directly from browser script tags via SCI. See it in action. Version 0.4.11 introduced the re-frame plugin. You can play around with it in the playground here. Several other releases were made. Details in the changelog.

Process

Clojure library for shelling out / spawning subprocesses. This library traditionally had the syntax: (process [ & cmd-args ] ?opts) but in practice, it turned out that having the syntax (process ?opts & cmd-args) is more convenient, since you can use it with apply and command-line-args. All functions in babashka.process have been rewritten to support this syntax. See changelog for details.

Quickdoc

Quickdoc is a tool to generate documentation from namespace/var analysis done by clj-kondo. It’s fast and spits out an API.md file in the root of your project, so you can immediately view it on Github. Minor fixes and improvements were made.

Fs

File system utility library for Clojure. Fs has gotten one new function: update-file, that alters the contents of a (text) file using a function. The function is reminiscent of swap!. See changelog for more details.

Neil

A CLI to add common aliases and features to deps.edn-based projects. A NEIL_GITHUB_TOKEN can now be configured to avoid hitting the rate limit of the Github API, thanks to Russ Matney.

Quickblog

Light-weight static blog engine for Clojure and babashka. The blog you’re currently reading is made with quickblog. Version 0.1.0 was finally released with much thanks to Josh Glover. See changelog for more details.

Rewrite-edn

Utility lib on top of rewrite-clj with common operations to update EDN while preserving whitespace and comments. Minor fixes and enhancements. Several functions have been added like fnil and conj. See changelog.

Sci.configs

A collection of ready to be used SCI configs for e.g. Reagent, Promesa, Re-frame and other projects that are used in nbb, joyride, scittle, etc. See recent commits for what’s been improved.

Nbb

Scripting in Clojure on Node.js using SCI. Because it’s so easy to deploy to npm, I usually publish a new version for each issue that is resolved. No big changes, but many small bugfixes and improvements in the last two months. See changelog.

Edamame

Configurable EDN/Clojure parser with location metadata. It has been stable for a while and reached version 1.0.0. The API is exposed now in babashka and nbb as well. See Changelog

lein2deps

Lein to deps.edn converter. This tool can convert a project.edn file to a deps.edn file. It even supports Java compilation and evaluation of code within project.clj. Several minor enhancements were made. See changelog.

Joyride

Modify VSCode by executing ClojureScript (SCI) code in your REPL and/or run scripts via keyboard shortcuts. I’m working on this project together with Peter Strömberg (known for his work on Calva) and I’m mostly reviewing Peter’s PR instead of writing code. Read the changelog here.

Deps.clj

Regular maintainance, keeping up with the official Clojure CLI and tools jar!

Clj-kondo configs

Library configurations as dependencies for clj-kondo. The claypoole configuration was improved.

Babashka CLI

Turn Clojure functions into CLIs! Minor fixes. See changelog.

Babashka pods

The pods library contains the code that supports using pods in babashka and the JVM. A critical error was fixed that would hang babashka and a new JVM release was pushed to Clojars (v0.1.0).

Babashka compatibility in external libs

I contributed to RCF, deep-diff2 and clj-diff to make these libraries babashka compatible. Discuss this post here.

Published: 2023-01-06. Tagged: clojure oss updates

Project Clojure Data Cookbook: Kira McCLean

Last updated December 30, 2022

The work that’s been done in this period for the Clojure Data Cookbook includes:

The next phase will be spent filling out the code examples for the rest of chapters 2 & 3, as well as finalizing the introductory text for the first chapter.

Project ClojureDart: Christophe Grand

(Disclaimer: because of various holidays, our months have been longer than calendar months…)

First reporting period: The biggest effort the during first period was the rework of our ill-named analyzer.

Dart doesn’t provide the level of introspection the JVM provides — there are “mirrors” but they provide incomplete information and are only available to the Dart VM (not to natively compiled code or code compiled to javascript). Being VM-only isn’t a too big requirement since dev workflow is centered on the VM but not having precise enough information is a deal-breaker. All the information we need is available through generated dart docs or through LSP. Both leverages a lib of the Dart SDK which is called the analyzer.

Based on the same lib we had our own “analyzer” which was dumping in a single EDN file every piece of information about classes, fields and functions found in all libs of all deps (pubspec.yaml). Effective but slow as the dump may take minutes with large dependencies.

So this period we reworked our analyzer to be a subprocess of the compiler and not a batch. This leads to faster startups without adverse consequence on compilation time (the compiler caches analyzer answers). Overall making the developer experience more pleasing.

Second reporting Period: During the second period we focused on making hot-reload less brittle. This is significant for two reasons:

The Dart VM can hot reload code. In practical terms it means functions and methods bodies can be updated while preserving existing state (at some point you’ll end up with a mismatch with the new code and the old state and you’ll have to “hot restart” which is a plain restart of the application – but not of the VM). The goal of this game is to minimize how often the application has to be restarted.

One crucial point are Dart top-levels (to which Clojure vars are mapped) which are initialized on demand. Once initialized you can’t update the initialization code — or rather you can but it won’t be run again. Which proves to be a problem for all vars which don’t compile to a simple Dart function. That’s why it was frequently required to hot restart (and thus lose state) to see the effect of redefs.

With the changes we made, when a var is redefined it’s compiled to a new top-level whose name contains a reload count. And all dependents are updated to point to the new name. Thus we get a fresh init and everything points to the new version (closing over a stale instance is still possible though but just like in Clojure). This hasn’t been merged yet into main.

While we are on unmerged branches: we have also sorted collections in the works, maps are done, sets are almost done, some helper functions are missing. Sorted collections are not a hot topic and are rarely used, however we put a lot of work in them: they use a novel implementation influenced by B+trees, treaps and zip-trees. An important feature of all ClojureDart maps (sorted or not) is that they are history-independent: their layout is determined by their content only, not by the insertion order nor the removal of some keys etc.

This is going to allow us to offer accelerated operations on maps: merge, merge-with, join, diff, merge3. Hopefully this will enable some interesting in-memory databases.

Project Mathbox-cljs: Sam Ritchie

Final Update (January 25, 2023)

Overview

First of all, apologies that this last phase stretched out over an extra month. I’m really happy with the amount of code I was able to ship; I’m especially pleased that I was able to keep my standards for writing and documentation high, though I did have to relax a touch to get these last two very complex libraries out the door.

Libraries Shipped

I’ve shipped final versions of the following libraries, along with interactive docs notebooks for each:

and was able to get started with a set of Clerk templates that will make it easy for folks to use all of these libraries: https://github.com/nextjournal/clerk-cljs-demo/pull/2 Notice that MathBox.cljs, the original goal of all of this work, is out!

Mafs

I noticed https://mafs.dev/ while trawling around for good 2d interactive visualization systems. JSXGraph is more complete, but Mafs is simple and beautiful; when I found it it wasn’t in active development, but the maintainer has since done a ton of work and the project is moving quickly. I’ve been working on the API with Steven Petryk in his Discord, and submitted a PR to speed up all linear algebra in the library. The API, specifically the “movable points”, relied on React hooks; the biggest change required by this wrapper involved making these points work well with Reagent’s atoms with the ease that ClojureScript developers should expect.

MathBox

I updated the MathBox README with instructions on how to use all of our new work from JS land: https://github.com/unconed/mathbox

The interactive docs page contains a full port of the “getting started” intro from MathBox: https://mathbox.mentat.org/

There are a bunch of things that this wrapper can do much better than the original library, thanks to React. I’ve continued to work with Chris Chudzicki to push improvements back to his mathbox-react library. I was able to port a bunch of the examples over to MathBox.cljs, but my index page is currently broken due to a Clerk bug… I’m working on a fix here that should be out and announced soon.

Leva.cljs

This worked out REALLY WELL… I spent a lot of time making every input element in Leva compatible with my “sync via atom” approach, and I think it feels even better than the original. The docs at https://leva.mentat.org/ are certainly better than anything in JS land.

Last Month’s Stretch Goals

I wasn’t able to complete the Clerk project templates, but those are the next logical thing to do, so I’ll move on to that next. I’ve also changed the name of my SICMUtils project to “Emmy”, to prepare for integration with all of these libraries. I’ve started work on my “ultra-fast ODE solvers” at https://github.com/sicmutils/sicmutils/pull/533 (yet-to-be-migrated to Emmy).

Thanks to Clojurists Together, I now have a stable foundation of every 2d and 3d visualization that I need to represent all of Emmy’s mathematical objects and algorithms… and the community has all of these tools as well, along with the guarantee that they all work with Clerk and Reagent, and coming-soon guides on how to integrate all of this easily. (Of course each docs notebook is sort of a guide on how to integrate all three.)

Project Tablecloth: Ethan Miller

Final Update (December 31, 2022)