November and December 2023 Project Updates

By Kathy Davis


There is a lot of work to report on folks! We're closing 2023 with two groups of updates:

Thanks to all our 2023 developers for their incredible work throughout the year. We’re looking forward to an awesome 2024!

2023 Long-Term Developers:
Bozhidar Batsov: CIDER/REPL
Eric Dallo: Clojure-lsp, clojure-repl-intellij, clojure-lsp-intellij
Michiel Borkent: clj-kondo,babashka, squint, neil, CLI,clojure-mode, and more..
Nikita Prokopov: Humble UI, Clojure Sublimed, DataScript, Sublime Executor
Peter Stromberg: Calva, JavaScript REPL, Polylith
Peter Taoussanis: Carmine, Nippy, http-kit, Tempel, Telemere
Sean Corfield: clojure-doc.org, deps-new, honeySQL, expectations, next.jdbc, clj-Watson, org.clojure/java.data
Thomas Heller: Shadow-cljs
Toby Crawley: Clojars
Tommi Reiman: Malli, Reitit, Jsonista

Q3 2023 Updates:
Jeaye Wilkerson:Jank
Charles Comstock and Jack Rusher: Quil

Final Reports 2023 Long-Term Developers

Bozhidar Batsov

Happy New Year, everyone!

November and December were a bit slower than the previous couple of months, but we still made some nice progress. During this period we’ve continued the trend of refining the big changes introduced in CIDER 1.8. We’ve shipped 2 new releases - CIDER 1.11 and CIDER 1.12 - both of them feature mostly refinements to the Inspector functionality.

There’s currently a bit of work-in-progress to refine cider-log-mode that will likely land in the next feature release. The work to add clojure-ts-mode support in CIDER in in progress as well.

I also wrote an article on using clojure-lsp alongside CIDER and I plan to do a bit more work in that direction (e.g. expand CIDER’s documentation). And finally write some of the backlogged CIDER articles I have in my personal TODO list. :-)

Eric Dallo

During these 2 months I spent a considerable time doing mostly 2 things:

Even so, I can say I’m proud of the ongoing work on IntelliJ side related to clojure-lsp support, and I hope in a few weeks we should announce the new clojure-repl-intellij plugin and how to use it with clojure-lsp.

clojure-repl-intellij

It’s still pretty alpha and missing lots of REPL features, but this is how it’s looking for now:
clojure-repl-intellij-demo

clojure-lsp-intellij

clojure-lsp-intellij is way more stable, compatible with multiple intellij versions and other plugins like vim-intellij, also the main feature highlight is the new wizard to create new Clojure projects from Intellij itself:
clojure-lsp-intellij-wizard

0.13.1 - 0.14.1

clojure-lsp

The main highlights are performance improvements and fixes, there is a huge improvements in clojure-lsp API for the format task.

2023.12.29-12.09.27

General

Editor

API/CLI

Happy new year, and thank you all for this amazing journey!

Michiel Borkent

Happy new year!

First all, as this is the last day of 2023, I wish you all a happy new year. Hopefully many goods things may happen in the Clojure ecosystem. I’m grateful many of you have sponsored my work in 2023!

Sponsors

I’d like to thank all the sponsors and contributors that make this work possible. Without you, the below projects would not be as mature or wouldn’t exist or be maintained at all. Top sponsors:

If you want to ensure that the projects I work on are sustainably maintained, you can sponsor this work in the following ways. Thank you!

If you’re used to sponsoring through some other means which isn’t listed above, please get in touch. Thank you! On to the projects that I’ve been working on!

Advent of Code

It is Advent of Code time of year again. You can solve puzzles in an online squint or cherry playground here.

Change the /squint/ part of the url to /cherry/ to switch ClojureScript dialect versions. You can read more about the playground here.

November 2023 Updates

December 2023 Updates

Other projects

These are (some of the) other projects I’m involved with but little to no activity happened in November and December. See final section of each report (“Other Projects”) for details.
[https://blog.michielborkent.nl/oss-updates-nov-2023.html]
[https://blog.michielborkent.nl/oss-updates-dec-2023.html]

Discuss this post here
 Published: 1 and 31 December, 2023 \ Tagged: clojure oss updates


Nikita Prokopov

Last update of the passing year, so pour yourself a warm drink, cover yourself with a cozy wool blanket and let’s see what are we ending the year with.

Humble UI:

DataScript SQL Storage:

DataScript:

Clojure Sublimed:

Sublime Executor:

To sum the whole year up, these are the most notable things that happened:

Thanks Clojurists Together and its sponsors for funding this work. This is a dream job for me and I’m very thankful for a chance to be doing it. 2024 is going to be a open-source-as-a-full-time-job year, so even crazier!

Happy New Year!

Peter Stromberg

This summarizes my last two month of long term funding 2023. I can’t find words for how great this has been for me. Clojurists Together ROCKS! ❤️

Calva

The changes to Calva were mostly about fixing bugs. User support revealed quite a few places where Calva quality could be improved and was improved:

Joyride

Clojure language support updates:

Developer tooling:

Squint

I wanted to help test Squint and used it for solving some Advent of Code problems. Then I started instead to help fix some of the issues I reported as a result. Because Michiel Borkent is the way he is, helping with Squint development was more fun than solving AOC problems so I dropped out of AOC. My most significant contribution to Squint during this “sprint” was adding almost complete support for clojure.set.

Peter Taoussanis

A big thanks to Clojurists Together, Nubank, lambdaschmiede, and other sponsors of my open source work!

Recent work

In November I released the first public alpha of Tempel, a new data security framework for Clojure - and my first all-new Clojure library in 7 (!) years.

Its main objective - to make it practical for more Clojure applications to use encryption and cryptographic best practices to protect user data and other sensitive info.

It offers a particularly high-level API focused on common tasks including logins, symmetric & asymmetric encryption, end-to-end encryption, signing, and key management.

For lots more info (incl. beginner-oriented documentation), see the Tempel GitHub page.

Since then, I’ve been mostly concentrating on Telemere - another all-new library focused on providing an idiomatic and flexible Clojure/Script API for structured telemetry. Support is planned for OpenTelemetry and others. Will share more info on this at release.

Upcoming work

My current roadmap includes:

2023 recap

I’m happy to say that 2023 was a really productive year for my open source work. It was the first time I’ve had support from Clojurists Together - and their backing along with my usual contributors meant that I could put more time + effort into open source than I’ve been able to do in many years.

In particular, I took the opportunity to target some larger/hairier tasks that would have otherwise been infeasible. Some of that work has already borne fruit, some of it will bear fruit in the coming year.

Some notable results this year:

- Peter Taoussanis

Sean Corfield

In my previous Long-Term Funding update I said I would review and update of the “cookbooks” section and make another pass of “TBD” items in the “language” section.

clojure-doc.org

I reviewed and updated the cookbooks for Files and Directories, Mathematics, Middleware, Parsing XML in Clojure, and Strings, bringing them all up to Clojure 1.11 (and testing the examples – and fixing the broken ones).

For the Mathematics cookbook, that meant rewriting the content that previously used Java interop and/or math.numeric-tower to use the new-in-1.11 clojure.math namespace.

Several cookbooks got minor updates to take advantage of functions in clojure.core and clojure.string that have been added since Clojure 1.4, when most of the original material on clojure-doc was written.

I also went through all the Java documentation links and updated those to point to the Java 17 versions (they were mostly pointing at Java 7 previously!). These will get updated again once use of JDK 21 has become more widespread.

I’ve been slowly working my way through the “TBD” items in the various Language guides, including the Glossary, although some of them really need input from community members who have specialist knowledge in those areas. In particular, the Concurrency and Parallelism and Polymorphism guides still have a number of “TBD” items that I don’t feel qualified to write! Volunteers welcome!

Thank you to @adham-omran for a PR that added the Date and Time cookbook and to @samhedin for a PR that added a section about adding Java code to Clojure projects to the tools.build cookbook.

Finally, I made a logo and a favicon for the site with my very limited artistic “talents”!

To wrap up the year of work on clojure-doc.org, I consider the Clojurists Together funding to have been a massive success. The site has been completely overhauled at this point, bringing it up to date with Clojure 1.11 and removing all the outdated (and now-duplicated) material that was originally missing from the official Clojure documentation. In addition, by raising the profile of clojure-doc.org in the community, contributions have increased with two new cookbooks added via Pull Requests and several other sections of the site either getting PRs or being updated by me in response to extensive feedback from the community (mostly on Slack).

Keeping the site updated now feels like a tractable problem and I’m hoping to find time in 2024 and beyond to add more content to the site, especially when Clojure 1.12 is released and there are a lot of enhancements to Java interop!

deps-new

deps-new 0.6.0 was released with several documentation updates, a new :src-dirs option to make it easier to use deps-new as a library and use templates from the local file system, and a new :post-process-fn to make it possible to modify the generated project programmatically.

Expectations

No new release yet but several documentation updates for the clojure.test-compatible version of Expectations.

HoneySQL

HoneySQL 2.5.1103 was released with smarter quoting of entities, smarter handling of metadata in formatting, and new options to provide more control over both of those features.

next.jdbc

next.jdbc 1.3.909 brings improved compatibility with clojure.java.jdbc for insert-multi! and adds a :schema-opts option to provide more control over schema conventions for datafy/nav. There have also been several documentation updates, in particular around how to use next.jdbc/plan and next.jdbc.sql/find-by-keys. The build.clj has been updated to use the :pom-data option introduced in tools.build 0.9.6, as a better example for the community.

clj-watson

clj-watson is a great tool for checking your dependencies for known security vulnerabilities. It’s a wrapper around OWASP Dependency Check and NIST is requiring users of its NVD (National Vulnerability Database) to switch from using data feed downloads to a new API that requires a free key for access. The DependencyCheck library that clj-watson uses has been updated to use the new API, but it isn’t backward compatible so clj-watson needed changes to use the new version of the library – and to provide an easier way for users to specify their own NVD API key.

Although the clj-watson maintainer has moved on from Clojure, they’ve been receptive to my Pull Requests to update the documentation, update the library dependencies, add a new, optional properties file that users can provide to override defaults, as well as a new command line option to specify that file, if you don’t want it on the classpath, and to update the DependencyCheck library and provide documentation on how to obtain an NVD API key and how to use it with clj-watson.

A v5.0.0 release of clj-watson has been made, with all these changes, and is available as a git dependency. A Pull Request is pending with the README updates.

org.clojure/java.data

Finally, the java.data Contrib library has a new release, 1.1.103, which removes the dependency on org.clojure/tools.logging – which in turn means that next.jdbc no longer depends on tools.logging, reducing the chance of conflicts for users of either library.

Thomas Heller

Time was mostly spent on doing maintenance work and some bugfixes. As well as helping people out via the typical channels (e.g. Clojurians Slack).

Current shadow-cljs version: 2.26.2 Changelog

Toby Crawley

November 2023

Commit Logs: clojars-web, infrastructure

I rewrote the permissions system this month to support project-level scoping, allowing delegation of deploy rights to a user for a subset of projects under a group. This was released in December.

I also made improvements to the AMI release process.

December 2023

Commit Logs: clojars-web, infrastructure

This month I released the permissions system rewrite, updated the release logic to require a license in the POM for all releases, and made a few other minor improvements:

On the infrastructure front, I added disk space and SQS queue delay alarms so we can be better informed of when things go wrong.

Tommi Reiman

Many improvements are in the works, but no releases on libraries. My work spread over the following:

Working with Malli:

Drafting a Clojure specification for a 1:1 x-state compatible FSM implementation(xstate v5 released in december)

Malli Development Mode

Coercion

coercion

Schema creation

schema-creation

Something else

Looking forward to 2024! Lot’s of interesting things in the OS pipeline and thanks to new long-term funding, will continue to work on them. Also, New year starts with crispy -20 (in Celcius).

tampere

Q3 2023 Project Reports

Charles Comstock and Jack Rusher: Quil

We had hoped to have a new release by the time we submitted this update, but we – by which I mean Charles – are down in the weeds trying to get all of the test infrastructure to work for every major platform in Github CI. The good news:

Our plan is to continue the work a bit more in the new year to get everything clean and maintainable, then train some other devs in the community on the codebase.

Thanks for much for your help, Clojurists Together!
Published 27 December 2023

Jeaye Wilkerson: Jank

I’ve been quiet for the past couple of months, finishing up this work on jank’s module loading, class path handling, aliasing, and var referring. Along the way, I ran into some very interesting bugs and we’re in for a treat of technical detail in this holiday edition of jank development updates! A warm shout out to my Github sponsors and Clojurists Together for sponsoring this work.

Module loading progress

Ok, first and foremost, where is jank now with regard to module loading? I’m very pleased to say that everything I wanted to tackle this quarter has been finished and even more on top of that. There’s a PR up for the full changes here.

Let’s break this down by section.

Class paths

jank traverses a user-defined class path, which supports directories and JAR files, and can use that to find modules when you use require and friends. This is specifically designed to be compatible with the JVM, so once we hook in Leiningen or Clojure CLI, your existing dependency management should work just fine.

Necessary core functions

The following functions have all been implemented, which were required for module loading:

These take into account modules that are already loaded, flags for things like reloading, excluding, etc. For most use cases, they’re at functional parity with Clojure on the happy path. Error handling will improve once I have some better mechanisms for it.

Still, that’s not a very big list of functions, I know. How about this one?

All of these were needed by some of the above necessary functions, so I implemented them as much as possible. Most of them have complete functional parity with Clojure, but a few have interim implementations, especially since jank doesn’t yet have have an equivalent object type to Clojure JVM’s LazySeq. Still, jank feels, and looks, more and more like a proper Clojure every day.

(Bonus) Initial AOT compilation

You may have noticed, in that list, that compile has been implemented. This is an initial step toward AOT compilation and it compiles jank files into C++ files on the class path. Those can then be loaded in lieu of the jank files for a performance win. I also added a CMake job to jank’s build system to build the jank Clojure libs along with the compiler, so we can always have those pre-compiled and also always know they actually compile.

I’m currently working with the Cling developers to get support added to Cling for jank to pre-compile these C++ files into a closer equivalent to JVM class files. In my local testing, the startup time improvements by doing this were 10x. I’ll have more info on this once the work picks up.

(Bonus) CLI argument parsing

In order to support things like user-defined class paths, I’ve added a proper CLI arg parser to jank. You can see the current options in the help output here:

❯ ./build/jank -h
jank compiler
Usage: ./build/jank [OPTIONS] SUBCOMMAND

Options:
  -h,--help                   Print this help message and exit
  --class-path TEXT           A : separated list of directories, JAR files, and ZIP files to search for modules
  --output-dir TEXT           The base directory where compiled modules are written
  --profile                   Enable compiler and runtime profiling
  --profile-output TEXT       The file to write profile entries (will be overwritten)
  --gc-incremental            Enable incremental GC collection
  -O,--optimization INT:INT in [0 - 3]
                              The optimization level to use

Subcommands:
  run                         Load and run a file
  compile                     Compile a file and its dependencies
  repl                        Start up a terminal REPL and optional server

Each subcommand has its own help output, too. Speaking of subcommands, however, jank now has a repl subcommand which spins up a terminal REPL client with readline enabled for (single session) history and improved editing. This has been very handy for me as I’m testing out new things and was something that just came naturally after implementing the CLI argument parsing.

❯ ./build/jank repl
> (ns foo.bar)
nil
> *ns*
foo.bar
> (def wow "WOW!")
#'foo.bar/wow
> (def nice "NICE!")
#'foo.bar/nice
> (ns main)
nil
> *ns*
main
> (refer 'foo.bar :only '[wow])
nil
> wow
WOW!
> (alias 'fb 'foo.bar)
nil
> fb/nice
NICE!
> (ns omg.wow (:use [foo.bar :exclude [wow]]))
nil
> *ns*
omg.wow
> nice
NICE!
> (native/raw "*((char*)0) = 0;")
Segmentation fault (core dumped)

(Bonus) Maps, sets, keywords as functions

As part of implementing all of the new core functions this quarter, I also tackled these particular objects which behave as functions. Fortunately, because of the new object model design, these objects can have this behavior without the need for dynamic dispatch!

There will be bugs

jank is still pre-alpha software. I have an ever growing test suite, but no battle testing yet. As I develop more functionality, I find more issues and introduce more yet. That will remain the case until development can settle down and stable APIs can be decided. jank still isn’t ready to compile most Clojure programs, since it lacks support for some basic features like destructuring, lazy sequences, and even doc strings. While we’re talking about bugs, though, and since I’ve shown everything else I’ve built this quarter, let me tell you about such an interesting bug I found and how I fixed it.

Variadic argument matching bug

I fixed a few interesting bugs in the past couple of months, but this one was the most intriguing by far. So, the problem showed up in this case:

(defn ambiguous
  ([a]
   :fixed)
  ([a & args]
   :variadic))

(ambiguous :a) ; => should be :fixed

What jank was trying to do was call the variadic arity, with an empty seq for args, rather than to call the fixed arity. This is because both of them require one fixed argument first and the information I was storing for each function object was the required fixed args prior to variadic arg packing.

The equivalent function in Clojure JVM is RestFn.getRequiredArity, which returns the required fixed position arguments prior to the packed args. However, where Clojure JVM differs from jank is that Clojure uses dynamic dispatch to solve this ambiguity whereas jank does its own fixed vs variadic overload matching, for performance reasons.

To actually solve this problem, we need to know three things:

  1. Is the function variadic?
  2. Is there an ambiguous fixed overload?
  3. How many fixed arguments are required before the packed args?

We cannot perform the correct call without all of this information. Also, function calls in a functional programming language like Clojure are on the hottest of hot code paths, so I can’t exactly add two more virtual functions to jank’s callable interface to get this data. In truth, even keeping one function but putting all of this data in a struct proved too much of an impact on the performance. Thus, we need to encode the data more compactly.

jank now packs all of this into a single byte. Questions 1 and 2 each get a high bit and question 3 gets the 6 remaining bits (of which it uses 4) to store the fixed arg count. So, this byte for our ambiguous function above would look like this:

1  1  0  0  0  0  0  1
^  ^  ^---------------
|  |  |
|  |  /* How many fixed arguments are required before the packed args? */
|  /* Is there an ambiguous overload? */
/* Is the function variadic? */

From there, when we use it, we disable the bit for question 2 and we switch on the rest. This allows us to do a O(1) jump on the combination of whether it’s variadic and the required fixed args. Finally, we only need the question 2 bit to disambiguate one branch of each switch, which is the branch equal to however many arguments we received.

object_ptr dynamic_call(object_ptr const source, object_ptr const a1)
{
  return visit_object
  (
    [=](auto const typed_source) -> object_ptr
    {
      using T = typename decltype(typed_source)::value_type;

      if constexpr(function_like<T> || std::is_base_of_v<callable, T>)
      {
        /* This is the whole byte, answering all three questions. */
        auto const arity_flags(typed_source->get_arity_flags());
        /* We strip out the bit for ambiguous checking and switch on it. */
        auto const mask(callable::extract_variadic_arity_mask(arity_flags));

        /* We're matching on variadic + required arg position. */
        switch(mask)
        {
          case callable::mask_variadic_arity(0):
            return typed_source->call(make_box<obj::native_array_sequence>(a1));
          case callable::mask_variadic_arity(1):
            /* Only in the case where the arg count == the required arity do we
               check the extra bit in the flags. */
            if(!callable::is_variadic_ambiguous(arity_flags))
            { return typed_source->call(a1, obj::nil::nil_const()); }
            /* We're falling through! */
          default:
            /* The default case is not variadic. */
            return typed_source->call(a1);
        }
      }
      else
      { /* ... redacted error handling ... */ }
    },
    source
  );
}

The special case, which needs to check the ambiguous flag, incurs a performance cost, due to the if. Every other case is unaffected. This was a challenge to wrap my head around at first, but after I wrote out all the things I need to know, as well as a test suite for each of the cases, I could work toward a solution which addressed everything.

What’s next?

Firstly, dynamic vars. Once those are implemented, I’ll need to go through all of the different parts of the compiler and runtime to start filling in vars. This will allow everything from improved error messages by tracking file/line/function to cyclical dependency checks on module loading.

Also, in order for jank to operate alongside other Clojure dialects, we’ll need to support reader conditionals on the :jank key. Currently, jank doesn’t support any reader macros, so getting that system going will open up the door to things like #() and #{} being supported.

Finally, I’ll be improving the interop interpolation syntax to be consistent with ClojureScript, adding meta hint support, and more. Stay tuned!
Published 27 December 2023