November 2023 Short-Term Project Updates

By Kathy Davis

Check out the latest updates from Q2 and Q3 2023 Short Term Project developers. Thanks to members for your support - making this work possible!

Biff: Jacob O’Bryant
Bosquet: Žygis Medelis
Clj-kondo, babashka, cherry, SCI: Michiel Borkent
Deps-try: Gert Goet
GDL: Michael Sappler
Quil: Jack Rusher and Charles Comstock
Uncomplicate: Dragan Duric

Biff: Jacob O’Bryant

2023 Q3 Report 2. Published 30 November 2023.

Since the first update, I have completed:

On the last point: I was going to write up a quick accompanying how-to post to go along with that Biff-Docker example repo. However, I don’t really feel comfortable recommending Fly.io or DO App Platform at this stage. Fly.io had some weird, show-stopping networking errors when I tried to use it just recently. DO App Platform worked, but their UI around deployment/logs was clunky and didn’t inspire confidence (I did also encounter at least one small bug). I am extremely wary of unreliable deployments which have been a pain point for me in the past. If you’re small enough to consider using either of those platforms, I think you’re far better off going with plain VMs (e.g. DigitalOcean droplets), which is why that’s the default in Biff.

IMO Docker-based deployment is a much better fit for organizations that are to the point where using AWS/Azure/GCP/Kubernetes makes sense. As such, my plan is to get that example repo deployed on Kubernetes (DigitalOcean’s managed offering, specifically) and then write a how-to guide to go along with that.

I will say that I am excited about Fly.io. Of all the “next-gen Heroku” platforms I’m aware of (Fly, Render, Railway, DO App Platform), Fly is IMO the most promising. Making it easy to run any Docker-based app at the edge (instead of e.g. just Javascript apps) is a particularly good fit for apps that use htmx (like Biff apps, by default). I’m guessing that the issues I’ve experienced with Fly are due to them still having lots of growth and having trouble keeping up with it, which seems like not the worst problem you could have! After they’ve become as reliable as DO droplets, I think there’s a good chance I’ll have Biff use them as the default deployment option.

Anyway: thank you for the grant! I’m really happy with where Biff’s documentation is now–it solidly covers the four main areas of tutorials, reference, how-to guides and explanation (essays). I’ve also been receiving some really kind feedback from the community as I’ve been publishing these articles (for example.

Bosquet: Žygis Medelis

2023 Q3 Report 2. Published 30 October 2023.

  1. Settle on Clerk-generated static content, published on GH Pages as the home for Bosquet documentation.
  2. Documentation:
    1. Getting Started guide
    2. Configuration. Bosquet relies heavily on declaring how AI generation is executed. The parameter setting is getting quite rich.
    3. Use cases. Generating interpretable code for math calculations
    4. Papers. Implementation of various academic papers discussing LLM techniques.
    5. Guide on how to use Short-term memory
  3. Substantial time was spent on a dead end trying to work out local embedding vector generation. The rationale was to have a self-contained system offering embeddings. Deeplearning4j and StandfordNLP Java libs proved to be too opaque, with no good implementations for this task.
  4. Support for OpenAI embeddings added to Bosquet. Now library users can create and use OAI embeddings as part of memory-driven workflows.
  5. To store and use Embeddings we need Vector DB. I have settled on using Qdrant as the database of choice. The implementation is still to be tested and changed with more rigorous testing and possibly various LLM papers implementations (see ‘Papers’ in documentation)
  6. With that done, two types of memory are now introduced
    1. Short-term memory acts as a simple illustration of memory and retrieval concepts
    2. Long-term memory using Qdrant and OAI embeddings (as noted above likely to evolve and change). Follow :memory/long-term-embeddings in system.edn to see how it is built.
  7. To aid in embeddingless (simple memory) scenarios, I have added similarity metrics support to compare different texts. It is based on Apache Commons text package.
  8. The caching of generation results is reintroduced. Now LLM requests can be fulfilled from the cache if prompt and parameters do not change. Great to boost development speed development and save on some costs.
  9. Output coercion. To achieve more complex chained prompt executions (or in agent use cases) where outputs the output from one LLM step needs to feed into the other we need well-defined in and out formats. Bosquet introduced the coercing of LLM outputs into JSON and EDN formats. See Chain of Density paper implementation for details.

2023 Q3 Report 3. Published 30 November 2023

Clj-kondo, babashka, cherry, SCI: Michiel Borkent

2023 Q2 Report 3. Published 31 October and 30 November 2023.
To see previous OSS updates, go here.

Sponsors

I’d like to thank all the sponsors and contributors that make this work possible! Open the details section for more info.

Top sponsors:

If you want to ensure that the projects I work on are sustainably maintained, you can sponsor this work in the following ways. Thank you!

If you’re used to sponsoring through some other means which isn’t listed above, please get in touch.

October 2023 Update

November 2023 Update

Advent of Code

It is Advent of Code time of year again. You can solve puzzles in an online squint or cherry playground here.

Change the /squint/ part of the url to /cherry/ to switch ClojureScript dialect versions.

You can read more about the playground here.

November Updates

Open for Squint Details
  • Restore backward compatibility with code that is compiled with older versions of squint
  • Optimize various outputs for smaller size
  • Add js-in
  • Support into + xform
  • Support sort on strings
  • #386: allow expression in value position in map literal
  • Improvements with respect to laziness in mapcat and concat
  • Do not array mutate argument in reverse
  • Escape JSX attribute vector value (and more)
  • map + transduce support
  • Fix for in REPL mode
  • Throw when object is not iterable in for
  • Make next lazy when input is lazy
  • Fix playground shim (fixes issue in older versions of Safari)
  • Add js-mod and quot
  • #380: Don’t emit space in between #jsx tags
  • Add re-find
  • Add condp macro
  • Use compare as default compare function in sort (which fixes numerical sorting)
  • Allow assoc! to be called on arbitrary classes (regression)
  • Improve get to call get method when present.
  • Allow keywords and collections to be used as functions in HOFs
  • Make filter, etc aware of truthiness
  • Reduce code size for truthiness checks
  • Add str/split-lines
  • Add partition-by
  • Add parse-long
  • Add sort-by
  • Fix top level await
  • Support multiple dimensions in aset
  • Add coercive-= as alias for ==
  • Add js-delete
  • Fix min-key and max-key and improve tests
  • Add min-key and max-key
  • Fix defonce in REPL-mode
  • Fix doseq and for when binding name clashes with core var
  • Several REPL improvements
  • Improve https://squint-cljs.github.io/squint/
  • Allow alias name to be used as object in REPL mode
  • Copy resources when using squint compile or squint watch
  • Return map when select-keys is called with nil
  • nREPL server: print values through cljs.pprint (@PEZ)
  • Initial (incomplete!) nREPL server on Node.js: npx squint nrepl-server :port 1888
  • Update/refactor threejs example
  • #360: assoc-in! should not mutate objects in the middle if they already exist
  • Evaluate lazy-seq body just once
  • Avoid stackoverflow with min and max
  • #360: fix assoc-in! with immutable objects in the middle
  • Add mod, object?
  • Optimize get
  • Add threejs example
  • #357: fix version in help text
  • Fix iterating over objects
  • Add clojure.string’s triml, trimr, replace
  • Fix examples/vite-react by adding public/index.html
  • Add find, bounded-count, boolean?, merge-with, meta, with-meta, int?, ex-message, ex-cause, ex-info
  • Fix munging of reserved symbols in function arguments
Open for Babashka Details
  • Support self-contained binaries as uberjars!
  • Add java.security.KeyFactory, java.security.spec.PKCS8EncodedKeySpec, java.net.URISyntaxException, javax.crypto.spec.
  • IvParameterSpec
  • Fix babashka.process/exec wrt babashka.process/defaults
  • #1632: Partial fix for (.readPassword (System/console))
  • Enable producing self-contained binaries using uberjars
  • Bump httpkit to 2.8.0-beta3 (fixes GraalVM issue with virtual threads)
  • Bump deps.clj and fs
  • Expose taoensso.timbre.appenders.core
  • nREPL: implement ns-list op
  • SCI: optimize swap!, deref and reset! for normal atoms (rather than user-created IAtoms)
  • Add test for #1639
  • Upgrade to GraalVM 21.0.1
  • Still unreleased:
  • Add java.util.ScheduledFuture
  • Support Runnable to be used without import
  • Allow catch to be used as var name
Open for SCI Details
  • Bump edamame to 1.3.23
  • #889: allow (def foo/foo 1) when inside namespace foo
  • #891: reset file metadata on var when it’s re-evaluated from other file
  • #893: expose sci.async/eval-form and sci.async/eval-form+
  • Improve sci.async/eval-string, respect top-level do forms
  • Add experimental new :static-methods option to override how static methods get evaluated.
  • Expose destructure
  • Macroexpand (.foo bar) form
  • Optimize deref, swap!, reset! for host values
  • Add time macro to core namespace
  • #896: allow catch to be used as var name

Other projects

These are (some of the) other projects I’m involved with but little to no activity happened in the past month. Click for more details. Discuss this post here.
_Tagged: clojure

Deps-try: Gert Goet

2023 Q3 Report 2. Published 4 December 2023.

I’ve been working on recipe functionality for deps-try (a tool to quickly try out Clojure libraries on rebel-readline), and I am happy to announce this functionality got released as deps-try v0.10.0.

Recipes practically function as ‘walkthroughs for the REPL’ as steps get front-loaded in the REPL-history and a user can work their way through by submitting steps.

As recipe namespaces list required dependencies and requires, they can also simply be used as a way to jumpstart a REPL-session to work on a particular domain using the –recipe-ns flag (e.g. connect to a PostgreSQL database having all necessary libraries loaded).

I added some built-in recipes to get started (see below) and it was fun to explore writing tutorials in this format.

There’s more recipes in draft and a lot of ways I can see this functionality being extended. I’m also curious how others will use it and what recipes they come up with: if you have ideas for a recipe that others can benefit from, don’t hesistate to open a PR.

A big thanks to (everyone supporting) Clojurists Together 🙏!

Built-in recipes:

NAME TITLE
deps-try/recipes Introducing recipes
malli/malli-select Introduction to malli-select, a library for spec2-inspired selection of Malli-schemas
next-jdbc/intro-sqlite A next-jdbc introduction using SQLite
portal/intro Introduction to portal, a Clojure tool to navigate data
_________________________ ________________________________________________________________

GDL: Michael Sappler

2023 Q3 Report 2. Published 30 November 2023

I am taking an experimental approach to developing this game&engine by working bottom-up and fixing/evolving the code before focusing on new/more features.

Also I want to create a simple GUI for editing the sounds/animations/properties of the game entities. This could lead to an action RPG-maker which can easily be modded or even a general 2D game maker tool.

The main problems right now in the codebase (because it is basically a legacy codebase from a game I wrote more than 10 years ago during my studies) are lots of global state and hardcoded properties (sounds, images, animations, creature, projectile properties are all over the place and hard to change).

I have focused on simplifying the lower level functions and moving side effects and dependencies to higher level code, thus making them easier to understand and read and not hiding side-effects like ‘swap! entity …’ in them.

The engine GDL

In gdl I have investigated what changes would be required to remove the global states and dependencies between namespaces. For example in the gdl.audio/play function which receives a sound-file parameter string depends on asset-manager to get the sound asset.

If I want to remove this dependency I found out I have to fix all the hard coded sound values in CDQ and move them to the properties.edn file, which I can then edit and view with the property-editor GUI.

So it is quite interesting that refactoring the play-sound function would lead to such dramatic and interesting changes, also it lead me to find out that animations are in many cases played together with a sound and will be moved together into an audiovisual ‘effect’ property.

The game Cyber Dungeon Quest

Quil: Jack Rusher and Charles Comstock

2023 Q3 Report 2. Published 20 November 2023

In this cycle, we’ve made an official release of version 4.3.1323! Here are the release highlights:

Upcoming work:

Uncomplicate: Dragan Duric

2023 Q3 Report 2. Published 31 October 2023

My goal with this round is to polish Uncomplicate libraries (mainly Neanderthal, Deep Diamond, ClojureCUDA, ClojureCL, ClojureCPP), rather than develop new functionality.

In this month, I’ve continued with Deep Diamond coding related to the port to JavaCPP. I’ve updated DNNL to version 3.3, ported dense and RNN layers in the DNNL engine to (Clojure CPP) JavaCPP, updated tests and made them pass. Then I ported the cuDNN GPU engine to Clojure CPP, and made all tensor, directed and RNN tests pass. Along the way, I fixed many bugs caused by differences between how JavaCPP and JCuda work. I updated Fluokitten support.

While working on Deep Diamond’s port, I’ve discovered many opportunities to improve and polish Clojure CPP, so my major efforts went toward understanding and fixing everything in Clojure CPP, so it’s ready for the first release. This includes complete documentation and test suite.

I’ve also cleaned up and polished new version of ClojureCUDA that is based on Clojure CPP. I completed documentation and polished the test suite. I’ve cleaned up Neanderthal’s port to Clojure CPP, worked on some bugs/issues, and managed to polish the transition to Clojure CPP, and fixed the test suite..

Assorted bugfixes and improvements in all libraries. I’ve also identified places for further improvements and cleanups.

Although there’s places for countless improvements in all libraries, the current progress in the short term is:

In the third month, the majority of my efforts will go to Deep Diamond and Neanderthal.

The non-snapshot release will have to wait for the next release of JavaCPP, as now most Uncomplicate projects use its dependencies, which I expect in the following few months. Hopefully by the end of the year.

2023 Q3 Report 3. Published 30 November 2023
My goal with this round was to polish Uncomplicate libraries (mainly Neanderthal, Deep Diamond, ClojureCUDA, ClojureCL, ClojureCPP), rather than develop new functionality.

In the third month, the majority of my efforts went to Deep Diamond. I wrote documentation, fixed lots of bugs, updated deps to new versions, and wrote fluokitten implementation. I’ve resolved some outstanding issuses. However, it still needs more polishing, especially the RNN part. In Neanderthal I discovered a bug in JavaCPP implementation (didn’t find a culprit). I tidied up metadata and licenses.

Although there’s places for countless improvements in all libraries, the current progress in the short term is:

Since now ClojureCUDA, Neanderthal, and Deep Diamond depend on JavaCPP, and I used new JavaCPP features that are present only in snapshots, I have to wait for the next official release of JavaCPP 1.5.10 to be able to publish official jars to Clojars. Of course, everything is on GitHub, so whoever needs these new versions, can build them from the source. JavaCPP is usually released twice a year, so I expect it in December, or at least before the next spring.

In short, here’s the final state of what I’ve proposed to do:

In even shorter terms, I proposed to do some hammock time, combined with some un-exciting chores that enhance the stability and user-friendliness of Uncomplicate libraries. I completed a lot of chores, and lots of development of the loose ends from the previous round. I am satisfied how I managed to fulfill this for Clojure CPP, ClojureCUDA, Neanderthal, and Fluokitten, fairly satisfied with ClojureCL, and almost satisfied with Deep Diamond (where I had to deal with changes in new DNNL and cuDNN versions while porting to JavaCPP, so I did not manage to iron out RNN).