April 2022 Monthly Update

By Alyssa Parado

Overtone Playground

My goal for this project is to provide a clear instruction on how to set-up and use Overtone without the hassle of going through bunch of errors and wrong steps which can amplify the frustration and prevent users from exploring programming and music through live coding. Since Sonic-Pi Tutorial is a good example of how starting guide for live coding should look like, my goal is to make similar one for experimenting with Overtone.

So far I have covered:

This is the advantage of continuous guide publishing through blog form; it keeps people interested and they engage more often, providing valuable feedback which helps me to write more useful guides in the future, covering areas which maybe I won’t think of.

I have updated the Overtone Playground GitHub project directory which can be cloned and used right away. Some features from the first few chapters of Sonic-Pi tutorial are implemented and can be used in similar manner right away. Guides in my following blog post will cover that process in a similar fashion to Sonic-Pi tutorials, where possible. All my posts that are part of this guide are listed on My Website under the category Clojurists Together 2022 Q1, so you can bookmark the link if you want to stay tuned.

Datahike Server

The first iteration focused on planning and updating the server API to the latest library API as well as design and implementation of relevant metrics in Datahike. Additionally we started on the first history integration into the server API.

Server REST API

The basis for any updates was a comparison table between available functions on datahike.api versus the current REST API in datahike-server and datahike-client.api. Datahike has grown since our last update on the server, so several additional functions needed to be added and some existing functions needed to be updated with additional functions. This resulted in the current PR 39 (https://github.com/replikativ/datahike-server/pull/39) While updating the API we also improved the testing behavior but we’re not fully satisfied with the result. So we plan to extend the tests in the next iteration with better coverage.

Server History

With history as one of the core functionality in Datahike, we needed to integrate this to the REST API as well. After some experiments and approaches, we decided on simple header arguments for history database selection together with selected REST API endpoints that can support history like q or datoms. The first basic implementation was finished with the branch historical-db(https://github.com/replikativ/datahike-server/tree/historical-db) together with first tests. Together with the better test coverage plan from the Server API we want to extensively test these functions as well in the next iteration.

Database Metrics

We began implementation of database index count metrics with per-transaction updates, but found that the opacity of whether a net addition of a datom has occurred during upsert, plus the unavailbility of a cheap count field or operation on the hitchhiker tree, make a performant and precise synchronous counter for arbitrary index structures infeasible. For now, we have settled on an on-demand count operation with caching as applicable (see in-progress pull request), with additional or improved implementation options in the future. Database-level metrics reporting backend and lib versions are also in progress.

Outlook

The next phase will see three milestones. First we will finish up the tasks from the first iteration, that means first the testing framework with refactoring of the current testing namespaces into separate ones, and in paralllel the integration of metrics into library and server. Second the work on the definition of the JSON specification will start with a proposition on how to handle different data models accordingly and consequently. And third we will collect different platforms, where we want to create a simple client as a wrapper around the REST API. Currently we’re thinking about either C#, Go or Rust there.

Beyond Clojurists Together Tasks

This month saw some fixes in Datahike. The fix of the migration with attribute references, improved docs, major dependencies update with breaking changes for the storage layout and insertion behavior fixes. With the latest release we solved a problem with retractions within our index.

Biff

Since writing the last update, I have:

The Biff newsletter also has 54 subscribers as of writing this which is dandy.

Thanks again for sponsoring Biff! I’m excited for the future of the project.

Orchard

Cljs compiler investigation

I’ve checked the cljs compiler state for any possibility of getting the executed function calls. The good news is that cljs.analyzer.api/analyze returns a data structure that contains most of the data that is needed. The structure is a little awkward to handle since it’s used to create the js code.

Here’s an abbreviated version that shows the parts that are useful to get the references:

(ana/analyze test-env '(defn abc [] (map #(inc %) [1])))
;; =>
{
:init {
    :methods [{
        :body {
            :ret {
                :args [{
                    :methods: [{
                        :body {
                            :ret {
                                :js-op cljs.core/+
                            }
                        }}]
                    :op :fn-method
                }
                {
                    :op :vector 
                }]
            }
            :fn {
                :name /map
            }
        }
        }]
}

}

The actual structure is a lot larger, but can mostly be ignored. The advantage here, compared to the Clojure version, is that it’s possible to get anonymous function calls directly and that inlined calls aren’t optimized away yet.

Sadly since the output of the analyzer is meant to be used to create js there’s already a transformation happening before the output is returned. For example inc is turned into a js + call, but this will mostly be a problem for very low-level calls and can probably be reversed.

The bad news

You can instruct the cljs compiler to dump an analysis cache during compilation. Unless I’m mistaken, this cache does not hold the needed information though. It seems to be mostly a cache of metadata. So to be able to use the information that the analyzer can provide I will have to call it on the actual source files.

At that point, I’d be using the cljs compiler like a static analyzer. This does have advantages over stand-alone tools like clj-kondo (output would always be “correct”), but would not be nearly as performant as the Clojure version.

Next steps

  1. try to extend the caching, so that functions don’t have to be reanalyzed
  2. create a proof of concept version to test performance, if step one fails
  3. integrate the results into orchard

Typed Clojure

The goal of this project funded by Clojurists Together is to (resurrect) support for type checking ClojureScript files in Typed Clojure.

Since the last update, support for type-checking ClojureScript has improved enough to consider the CLJS type checker “resurrected” (but still young).

To stress test the ability for Typed Clojure to check .cljc files, I have been type checking malli’s implementation.

Checking in CLJS mode has revealed many bugs and shortcomings in the checker. Clojure mode has been outrageously successful–it found a bug in malli!

Some other feature work has been landing in Typed Clojure that has support for both Clojure and ClojureScript (enabled by this Clojurists Together project).

See the CHANGELOG.md for full details on everything else accomplished in the time period since the last update (versions 1.0.21-1.0.28).

Reveal

After completing the test runner I still had some time left, so I spent it updating Reveal’s website. It is now much easier to navigate, and documentation was significantly improved:

Deep Diamond

My goal with this round is to implement Recurrent Neural Networks (RNN) support in Deep Diamond. The first month was dedicated to literature review (and other hammock-located work), exploration of OneDNN implementation of RNN layers, and implementation of RNN prototype in Clojure in Deep Diamond.

Now, in the second month, I managed to make the first iteration of fully functional RNN layer based on the DNNL backend (currently only Vanilla RNN implementation) that fits into the existing high-level network building code. (more details follow)

Deep Diamond currently supports general tensor operations, fully connected NN layers, and convolutional (CNN) layers, on CPU and GPU. Based on this, relatively stable, infrastructure, in the first month I added Vanilla RNN operation implementation backed by OneDNN (Intel, CPU).

Now, in the second month, I only managed to make a couple of commits, but the last one is pretty big. I couldn’t break it into many smaller ones, because I had to make a couple of subtle changes that affected the internal implementation and broke existing code (_does not affect users, just me!). I didn’t want to commit a completely broken code into the github repository.

Now, what does this commit do? A. It adds the first complete implementation of RNN layer that 1. technically works and pushes the numbers to the right places connected to it 2. is tested in isolation 3. is supported in high-level Deep Diamond code together with other layers type B. I implemented a supporting “Ending” layer for feeding the time-based RNN output to other layer types (Dense, CNN, etc.) C. I improved the existing infrastructure to support this, and also made it more flexible D. I fixed bugs and wrote tests…

That is fine, but I am not completely satisfied with this milestone. I hoped to make a fully working example of a network that learns something useful, but that proved a bit harder than I expected. I couldn’t make the RNN I have so far actually learn. It appears to be working, but even simple examples for RNN applications are not that simple, so I wasn’t be able to make it learn the artificial one I tried it with. I am not sure whether it’s due to bugs, or the example is simply pathological, and the Vanilla RNN (which is currently the only type I implemented) can’t do anything with it anyway. This is something that I’ll be doing the following days/weeks, and I expect to discover more when I implement more advanced LSTM and/or GRU RNN types.

Beside that, the CUDA backend is broken in the current commit since I haven’t updated it to the last changes, but it’s not a problem; once I nail DNNL, making CUDA compatible will be more or less routine.

Firefox

Custom Formatters for Firefox

Current status

After some back and forth between the reviewer and me, my patch for introducing the display of JsonML will land soon.

In order to handle custom formatters correctly and provide the user with some useful information in case of errors in the custom formatters themselves, there needs to be some exception handling. Therefore I’ve created bug 1764439.

There were also several discussions between the Firefox DevTools team and me regarding different implementation details. For example, there needs to be some handling for configuration objects. So I’ve created bug 1764443 to add that.

What’s next?

Bug 1734840 is still missing, which is the main thing to actually get the custom formatters to work. Once that’s done, the other bugs mentioned above need to be tackled.

Clojure-LSP

This month we had incredible important performance improvements, especially for big projects, along with support for ClojureDart, and new features!

Release 2022.04.18-00.59.32

We started to check how to improve some points of performance and CPU spikes on clojure-lsp, since this release we started to fix logs of cpu spikes in multiple places like: completion, startup, file changes, test tree and others. Now we have a cache layer for every external file analyzed, which means if user change anything on project that requires a full scan, it should be way faster than before. We moved the core of LSP to a separate project called lsp4clj making possible to create other LSPs for any languages in Clojure. Multiple improvements on java analysis for faster analysis and better UX settings. Multiple improvements on the Drag feature.

Here is the changelog of this release:

Release 2022.05.03-12.35.40

This release added the feature of when renaming files via editor, clojure-lsp will automatically rename the namespace and all references. A huge improvement on uncache startup with changes on clj-kondo which resulted on faster analysis. It’s now possible to sort package imports inside :import forms during clean-ns. Also, the progress report during startup is more precise resulting in a better UX.

Here is the changelog of this release:

Bozhidar Batsov

My main focus in the past couple of months was the preparation of CIDER 1.4. The release is almost ready and I expect to cut it any day now. It includes:

The whole changelog is here. We’ve also celebrated 10 years since the first CIDER commit on the 17th of April. You can read a bit more about CIDER’s history here

Note: Technically speaking CIDER 1.3 and clojure-mode 5.14 were released in March, but I already mentioned them in my previous update.

Also as usual - I’ve spent a lot of time discussing/investigating tickets on GitHub and providing support to various CIDER users.

Outside of CIDER:

Michiel Borkent

I’d like to thank all the sponsors and contributors that make this work possible.

Clj-kondo

A linter for Clojure (code) that sparks joy.

Published versions: 2022.03.04, 2022.03.08, 2022.03.09, 2022.04.08, 2022.04.23, 2022.04.25

Read the changelog here.

Highlights:

Babashka

Native, fast starting Clojure interpreter for scripting.

Published versions: 0.7.7, 0.7.8, 0.8.0, 0.8.1

Read the changelog here.

Highlights:

SCI

Configurable Clojure interpreter suitable for scripting and Clojure DSLs.

Published versions: 0.3.2, 0.3.3, 0.3.4

Read the changelog here.

Summary: many small incremental improvements.

Nbb

Ad-hoc CLJS scripting on Node.js using SCI.

Published versions: v0.2.1 - v0.3.10.

Read the changelog here.

Highlights:

Joyride

Modify VSCode by executing ClojureScript (SCI) code in your REPL and/or run scripts via keyboard shortcuts.

Read the changelog here.

This is a new project!

Grasp

Grep Clojure code using clojure.spec regexes.

Highlights:

Obb

Ad-hoc ClojureScript scripting of Mac applications via Apple's Open Scripting Architecture.

Published versions: 0.0.3

Read the changelog here.

Neil

A CLI to add common aliases and features to deps.edn-based projects.

Highlights:

Rewrite-edn

Utility lib on top of rewrite-clj with common operations to update EDN while preserving whitespace and comments.

Read the changelog here.

Tools.misc

Tools library similar to clojure.tools.logging but for misc. tools.

So far this is just an experiment, but join the discussion if you want.

Fs

File system utility library for Clojure.

Published versions: 0.1.4.

Read the changelog here.

Discuss this post here.

Dragan Djuric

In this reporting period, I continued work on Clojure Sound, and finally made it ready for the first release into Clojars.

In the previous period (January and February) the main objective was to learn the fundamentals of sound processing and communicating with control devices on the JVM, and start the work towards an initial version of full blown Clojure library that is capable of serious work in this area. Now I’ve wrapped up the first milestone with the initial release, that includes:

From the previous period:

Added and improved in this reporting period

Of course, I continued with lots of dedication to learning the fundamentals of this, including music theory, music instruments and devices, and whatnot. That is not something I can immediately convert into useful music related Clojure software (beyond Clojure Sound). However, month by month I feel the foundations are building so in a several months, or a year, or a couple of years (depends on how optimistic I am) I might be having some pleasant surprises for Clojurians!

As for the motivation, let me repeat (copy) what I wrote in the Jan-Feb report:

Although the motivation might be unclear to many Clojurists - who needs sound stuff in Clojure - I predict that this will be a very interesting and attractive project to many Clojurists, beginners and veterans alike. First of all, sound-related stuff is interesting, so it’s a good hobby coding and experimenting platform. As such, it’s a good choice for demonstrating various code techniques while avoiding boring topics. Next, it’s great for beginners to give them a topic that they know and like, music, as a playing ground for learning the topics that they don’t (yet) understand (Clojure and programming in general). Further, it covers things only incidentally related to music: communication with external hardware controller devices, which are often created for musicians in mind, but can be utilized in ingenious ways in general computing, with their abundance of knobs, slider, pads, and other convenient control methods. Last (but there will be more), digital signal processing is one of fundamental domains where high performance computing is applied. This will be an excellent demonstration platform for showing off Clojure’s HPC computing capabilities. Who knows, maybe we’ll even be able to build something that kicks-ass. Even if we don’t it will be an excellent platform for many Clojurists to make the first steps in DSP and HPC, and may even attract some people to look into Clojure.

Thomas Heller

Time was mostly spent on doing maintenance work and some bugfixes. As well as helping people out via the typical channels (eg. Clojurians Slack).

Current shadow-cljs version: 2.18.0 Changelog

Notable Updates

David Nolen

Nikita Prokopov

Hello everybody, this is Niki Tonsky and we are still building Clojure Desktop UI framework in public.

Humble UI:

First few apps built with Humble UI:

I am beyond excited!

JWM:

Skija:

DataScript, Rum and Tongue saw some maintenance releases.

Talks: