May & June 2024 Long-Term Project Updates

By Kathy Davis

A huge thank you to our 2024 long-term developers for their amazing work in May and June. Check out their latest project updates!

Bozhidar Batsov: CIDER
Michiel Borkent: squint, babashka, neil, cherry, clj-kondo, and more
Toby Crawley: clojars-web
Thomas Heller: shadow-cljs, shadow-grove
Kira McLean: Scicloj Libraries. tcutils, Clojure Data Cookbook, and more
Nikita Prokopov: Humble UI, Datascript, AlleKinos, Clj-reload, and more
Tommi Reiman: Reitit 7.0. Malli, jsonista, and more
Peter Taoussanis: Carmine, Nippy, Telemere, and more

Bozhidar Batsov

This period was quite busy and productive for CIDER and friends. The highlights are 3 (!!!) CIDER releases and a couple of nREPL releases:

CIDER 1.14 is our most ambitious release since CIDER 1.8 (“Geneva”), which was released last autumn.

The single most notable user-visible change of this release is that CIDER is now more robust when evaluating and displaying large values. CIDER will no longer hang when C-x C-eing a big value in a source buffer or stepping over such a value with CIDER debugger.

I’m guessing that many people will also appreciate the improvements we’ve made to flex completion (which is finally fully compliant with the Emacs completion API), the inspector and to the cider-cheatsheet functionality which was mostly redesigned.

nREPL 1.2 restores the ability to interrupt evaluation on JDK 20+ (see https://github.com/nrepl/nrepl/pull/318 for details) and CIDER 1.15 implements support for nREPL 1.2.

More interesting work is in progress, so I hope I’ll have another exciting report for all of you in a couple of months!

Michiel Borkent

Updates In this post I’ll give updates about open source I worked on during May and June 2024. To see previous OSS updates, go here.

Sponsors

I’d like to thank all the sponsors and contributors that make this work possible. Without you, the below projects would not be as mature or wouldn’t exist or be maintained at all.

Current top tier sponsors:

If you want to ensure that the projects I work on are sustainably maintained, you can sponsor this work in the following ways. Thank you!

If you’re used to sponsoring through some other means which aren’t listed above, please get in touch.

On to the projects that I’ve been working on!

Updates

Here are updates about the projects/libraries I’ve worked on in May and June.

Other projects

There are many other projects I’m involved with but that had little to no activity in the past month. Check out the Other Projects section (more details) of my blog here to see a full list.

Published 30-06-2024.

Toby Crawley

Commit Logs: clojars-web, infrastructure

June 2024

Commit Logs: clojars-web, infrastructure

Thomas Heller

Time was mostly spent on doing maintenance work and some bugfixes. As well as helping people out via the typical channels (e.g. Clojurians Slack).

Current shadow-cljs version: 2.28.10 Changelog

Notable Updates

Kira McLean

This is a summary of the open source work I’ve spent my time on throughout May and June, 2024. There were lots of small bug fixes and reports, driven by work on the Clojure Data Cookbook. This work was also the impetus for my initial release of tcutils, a library of utility functions for working with tablecloth datasets. I also had the wonderful opportunity to attend PyData London in June and found it really insightful and inspiring. Read on for more details.

Sponsors

This work is made possible by the generous ongoing support of my sponsors. I appreciate all of the support the community has given to my work and would like to give a special thanks to Clojurists Together and Nubank for providing me with lucrative enough grants that I can reduce my client work significantly and afford to spend more time on these projects.

If you find my work valuable, please share it with others and consider supporting it financially. There are details about how to do that on my GitHub sponsors page. On to the updates!

Ecosystem issue reports and bug fixes

Working on the cookbook these last couple of months turned up a few small issues in ecosystem libraries. The other developers of Clojure’s data science tools are such a pleasure to work with, it’s so rare and nice to have a distributed team of people capable of getting cool things built asynchronously. Here are some details of a few particular issues that came up:

Initial release of tcutils

In my explorations of other languages' tools for working data I often come across nice utility functions that are super simple but have a big impact on the ergonomics of using the tools. I wanted to start bringing some of these convenience utilities to Clojure, so for now I’m putting them in tcutils. So far only a handful of helpers are implemented (lag, lead, cumsum, and clean-column-names). The goal is to eventually fill out more utilities that save people from having to dig into the documentation of half a dozen different libraries to figure out how to implement things like these. The goal is not to achieve feature parity or to exactly copy similar libraries, like pandas or dplyr, but rather to take inspiration from them and make our tools easier to use for people who are used to these conveniences.

Progress on Clojure Data Cookbook

I spent a lot of time on the Clojure Data Cookbook over these last two months. Notable progress includes:

It’s still very much in progress, but in the interest of transparency the work-in-progress version is available online now. It will continue to evolve and change as I fill out more and more of the chapters, but there’s enough of it available now to hopefully give a sense of the style and tone I’m going for. I also finally have the publishing workflow set up and it’s generating a nice-looking Quarto book, thanks to all of Daniel Slutsky’s amazing work on Clay and Quarto integration recently.

Progress on high-level goals

The high-level goal of my work in general remains to steward Clojure’s data science ecosystem to a state of maturity and flourishing so that data practitioners can use it to get real work done. Toward this end, I set up a project board to track progress toward what I see as the main components of this project.

Over the last couple of months, beginning with a prototype demoed at my London Clojurians talk in April, Daniel Slutsky has made tremendous progress on our goal of implementing a grammar of graphics in Clojure in the new hanamicloth library. The near-term goal is to stabilize the API of this library enough that it can be used to provide a user-friendly way to accomplish all of the simple data visualization tasks that are currently possible with our other tools. The long term goal is to take the lessons we learn from this library and build a JVM-only grammar of graphics library for doing data visualization “right” in Clojure.

The development and surrounding discussions of hanamicloth have also made me realize it would be useful to write an overview of the current state of dataviz options for Clojure and why we’re working on building something new. That’s on my list for the coming months, but lower priority than actual development work.

Impressions from PyData London

I got to attend PyData London this year thanks to a client of mine who was sponsoring the conference. I learned a lot and found the talks very interesting. My overall impression is that data science is maturing as a discipline, with more polished methods and robust theory backing up different approaches to data-related problems. With this maturation, though, comes higher expectations for production-ready, professional quality results. Most of the talks focused on high-level concerns like observability, scalability, and long-term stewardship of large open-source projects.

There are a lot of reasons why Python is just not ideal for building highly available, high-performance systems, and I really believe this is a good time to be building alternative tools for data science. Python is obviously entrenched as the current default language for working with data, but it is difficult and slow to write code that can take full advantage of modern hardware (because of the infamous global interpreter lock, reference counting, slow I/O, among other reasons). And to be fair, the Python community knows this. It’s why virtually all of the libraries that do the heavy lifting for data science in Python are actually implemented in C (numpy, pandas) or Rust (Polars, Pydantic), or are wrappers around C++ (PyTorch, TensorFlow, matplotlib) or Java (PySpark, Pydoop, confluent-kafka) libraries.

I think this provides a lot of insights into what data practitioners want. It’s clear that users want approachable, simple, human-readable interfaces for all of these tools, and that any new tool needs to interoperate with the rest of the ones currently in use. People are also tired of churn and are craving stability. I think Clojure has a lot to offer in all of these areas and is well placed to become more widely adopted for data science.

Ongoing work

My focus over the next two months will remain on the cookbook. My main goal is to finish the introductory chapter with the housing price analysis and to continue putting together the data import section with instructions and examples for all file formats that can reasonably be supported easily at this time.

I’ll continue to support and contribute to all of the ecosystem libraries I come across in my writings and analysis work in hopes of smoothing out all the rough edges I find.

Thanks for reading. I always love hearing from people who are interested in any of the things I’m working on. If that’s you, don’t hesitate to be in touch :)

Nikita Prokopov

Hi, I’m Nikitonsky and this is my open-source update for the past two months. Some good work was done on Humble UI (finally!), DataScript and new project — AlleKinos.de.

New project: AlleKinos.de, a no-nonsense movie showtimes site for the entire Germany:

HumbleUI, Clojure Desktop UI framework:

DataScript, immutable database and Datalog query engine for Clojure, ClojureScript, and JS:

clj-reload, a smarter way to reload Clojure code:

Clojure Sublimed, Clojure support for Sublime Text 4:

Sublime Executor, Sublime Text plugin to run any executable from your working dir:

Blogging:

Talks:

Love,
Niki

Tommi Reiman

Started my 3 month sabbatical in June with a road-trip with the kids, a welcome reset! Now back to home, learning and doing.
Refreshed my knowledge of the latest TypeScript, Zod and XState with a goal to pull some of the good things to Clojure (into Malli + a fully Xstate-compatible FSM-library). Also working on a template-project with monorepo + malli + reitit, using Java21 and Virtual Threads.

Library Releases

reitit 0.7.1 (active)

Fixing regression bugs from 0.7.0 + latest features via dependent libraries. Changelog here.

malli 0.16.2 (active)

Welcome Experimental Simplified Function Schemas!

[:-> :any] ; [:=> :cat :any]
[:-> :int :any] ; [:=> [:cat :int] :any]
[:-> [:cat :int] :any]  ; [:=> [:cat [:cat :int]] :any]
[:-> a b c d :any] ; [:=> [:cat a b c d] :any]

;; guard property
[:-> {:guard (fn [[[arg] ret]] ...)} :string :boolean]
; [:=> [:cat :string] :boolean [:fn (fn [[[arg] ret]] ...)]]

Also, small fixes and additions. Changelog here.

There is a big bunch of WIP work from myself and contributors waiting to be finished.

jsonista 0.3.9 (stable)

:do-not-fail-on-empty-beans option + updated dependencies, Changelog here

ring-http-response 0.9.4 (stable)

Teapots welcome! Changelog here.

spec-tools 0.10.7 (inactive)

Small fixes and improvements, Changelog here. If you are a user of spec-tools and want to help, feel free to ping me on Clojurians Slack, happy to take a new contributor here.

Something Else

Old abandoned Soviet-era sanatorium in Latvia.

baltics

Peter Taoussanis

A big thanks to Clojurists Together, Nubank, lambdaschmiede, and other sponsors of my open source work! I realise that it’s a tough time for a lot of folks and businesses lately, and that sponsorships aren’t always easy 🙏

2024 May - Jun

Hi folks! 👋

The last couple months have been light on big-ticket releases. Have been focused on maintenance, support, and groundwork for future releases. Output included:

Nippy and Carmine security releases

If you haven’t yet, please do try update to the latest versions of Nippy and/or Carmine when possible:

These include a fix to address a security vulnerability described in more detail in Nippy’s release notes.

In short: Carmine uses Nippy for its serialization, and Nippy uses a Java compression library for its compression. Earlier releases of that Java library may be vulnerable when decompressing malicious data directly crafted by an attacker. The attack is believed to require arbitrary control of the data provided to Nippy for thawing.

Relevant posts were made to the Clojure subreddit, Clojurians Slack, and my X account.

Telemere

Work has continued on Telemere, my new structured logging and telemetry library for Clojure/Script.

There were numerous minor beta releases to address various issues that came up, and to polish sharp edges and documentation, etc.

Instead of detailing all that here, I’ll just point to the current release - v1.0.0-beta14. The latest beta release will always include a summary of all major recent changes.

I’m aiming to try out RC1 around the end of August, but won’t needlessly rush. I’d like the API to be completely stable after v1 final is out, so I’d rather go a bit slower now to get things right.

Big thanks to early adopters and testers for all the valuable feedback so far! 🙏

Carmine

Work has continued on Carmine v4. It’s quite an undertaking, but I’ve recently updated and merged the first parts of the new v4 core into mainline.

The current plan is for all the new stuff to live in a parallel taoensso.carmine-v4 namespace. This’ll make it easier for me to roll out the new work in stages, and get feedback from early adopters without negatively impacting existing users.

There’ll be a lot to say on Carmine v4, but that’ll come later.

Upcoming work

My current roadmap can always be found here, and it’s now also possible to vote to help guide my priorities.

Current objectives for July-August include:

Cheers!
- Peter Taoussanis