September 2021 Monthly Update

By Alyssa Parado

A big welcome to new company members Stylitics, Parkside, and doctronic; thanks for your support!

Here are the project updates from the month of September to the first half of October.

Clojure LSP

This was the month with the most changes, updates, and fixes of the project! Also, this is the new clojure-lsp webpage: https://clojure-lsp.io

release 2021.09.13-19.32.00

This release added a new diagnostics feature to clojure-lsp API, making clojure-lsp the only tool that can do most Clojure lint tasks in a single tool! Besides that, it was fixed a lot of issues and improved some defaults to make clojure-lsp a tool that just works out of the box along with some preparations for changing the way clojure-lsp persists the analysis cache.

Here is the changelog of this release:

Clojure LSP

I released version 2021.09.30-12.28.16.

This release focused on improving user UX giving better feedback during clojure-lsp startup both on API and editor. Also, it fixed a lot of issues making clojure-lsp a more mature tool with fewer false positives and more settings to work with more Clojure projects.

Polylith

I’ve released version 0.2.13-alpha of the poly tool, which includes the interactive shell command and a few more issues:

Holy Lambda

I’ve successfully refactored the holy-lambda core. Handlers can now be defined via regular defn and composed freely before passing to the entrypoint macro. The only functionality exposed from the core is entrypoint that now accepts additional parameter init-hook for side effect initialization of dependencies before the runtime loop executes. The following addition is handy if you want to initialize dependencies only once.

I’ve also decoupled holy-lambda from AWS SAM. As a result, tasks are faster, less error-prone, and minimal. My plans for the next few months are to remove problematic tasks: hl:sync & hl:compile and focus almost entirely on runtime. However, I plan no changes around native compilation and native configuration generation.

I’ve also published a brand new documentation, which is available here. Documentation is almost complete, but additional explanations for the native backend regarding the java agent are required.

Some other changes:

I’m also finishing the benchmarks for the new release. I hope the new release to be a much faster one, especially for the subsequent handler calls. The results will be published in the next month of funding.

Next month I’m also planning to release an Ring adapter for HL. The significant progress has been done in that area, so stay tuned.

Typed Clojure

The goal of this project funded by Clojurists Together is to improve static type error messages in Typed Clojure, specifically to replace expanded code in error messages with surface-level syntax.

In the first half of the project, I have concentrated on three main areas:

  1. Increase direct support for problematic clojure.core macros
  2. Improve error messages for inlining functions
  3. Identify classes of implementation quirks in core Clojure macros to prepare for potential typing rules

Increase direct support for problematic clojure.core macros

Problem: Typed Clojure expands macros that it does not have special rules for. This works well when the expansion is simple (eg., binding, future, or delay), but this strategy backfires horribly for complex macros like doseq.

For example, doseq does not have direct support in Typed Clojure 1.0.17 and any usage of it results in an incomprehensible error message (note: t/cf type [c]hecks a [f]orm):

$ clj -Sdeps '{:deps {org.typedclojure/typed.clj.checker {:mvn/version "1.0.17"}}}}'
Clojure 1.10.3
user=> (require '[clojure.core.typed :as t])
nil
user=> (t/cf (doseq [a [nil :a 3]] (inc a)))
Type Error (NO_SOURCE_PATH:1:7)
Loop requires more annotations


in:
(loop*
 [seq_30744
  (clojure.core/seq [nil :a 3])
  chunk_30745
  nil
  count_30746
  0
  i_30747
  0]
 (if
  (clojure.core/< i_30747 count_30746)
  (clojure.core/let
   [a (.nth chunk_30745 i_30747)]
   (do (inc a))
   (recur
    seq_30744
    chunk_30745
    count_30746
    (clojure.core/unchecked-inc i_30747)))
  (clojure.core/when-let
   [seq_30744 (clojure.core/seq seq_30744)]
   (if
    (clojure.core/chunked-seq? seq_30744)
    (clojure.core/let
     [c__6014__auto__ (clojure.core/chunk-first seq_30744)]
     (recur
      (clojure.core/chunk-rest seq_30744)
      c__6014__auto__
      (clojure.core/int (clojure.core/count c__6014__auto__))
      (clojure.core/int 0)))
    (clojure.core/let
     [a (clojure.core/first seq_30744)]
     (do (inc a))
     (recur (clojure.core/next seq_30744) nil 0 0))))))



Execution error (ExceptionInfo) at clojure.core.typed.errors/print-errors! (errors.cljc:274).
Type Checker: Found 1 error

We need explicit support for doseq and similar macros to both improve inference and error messages.

Prior work: Before custom typing rules were possible in Typed Clojure, an alternative macro clojure.core.typed/doseq was provided for compatibility with the type checker. For example, instead of (doseq [a [1]] ..) you would write (t/doseq [a :- Int, [1]] ...).

This macro had some downsides:

  1. all bindings required annotations.
  2. it had poor discoverability as doseq’s error message makes no mention of this alternative.

Approach: Create typing rule for doseq and create or enhance typing rules for other problematic clojure.core macros.

Results:

doseq is now supported and error messages are pleasant. Note the error msg for inc is also new–see next section.

$ clj -Sdeps '{:deps {org.typedclojure/typed.clj.checker {:mvn/version "1.0.19"}}}}'
Clojure 1.10.3
user=> (require '[clojure.core.typed :as t])
nil
user=> (t/cf (doseq [a [nil :a 3]] (inc a)))
Type Error (NO_SOURCE_PATH:1:29)
Function inc could not be applied to arguments:


Domains:
        Number

Arguments:
        (t/U (t/Val 3) (t/Val :a) nil)

Ranges:
        Number




in:
(inc a)



Execution error (ExceptionInfo) at clojure.core.typed.errors/print-errors! (errors.cljc:276).
Type Checker: Found 1 error

Commits:

Improve error messages for inlining functions

Problem: inline functions are an experimental Clojure feature that enables the compiler to treat a var as a macro in operator position and a function in higher-order contexts. Typed Clojure expands inline functions for :tag inference purposes, but if a type error occurs in the inlined expansion, the original form is lost and the expansion is blamed. This results in an unhelpful error message.

For example, inc blames its expansion clojure.lang.Numbers/inc:

$ clj -Sdeps '{:deps {org.typedclojure/typed.clj.checker {:mvn/version "1.0.17"}}}}'
Clojure 1.10.3
user=> (require '[clojure.core.typed :as t])
nil
user=> (t/cf (inc nil))
Type Error (NO_SOURCE_PATH:1:7)
Static method clojure.lang.Numbers/inc could not be applied to arguments:


Domains:
        Number

Arguments:
        nil

Ranges:
        Number




in:
(clojure.lang.Numbers/inc nil)



Execution error (ExceptionInfo) at clojure.core.typed.errors/print-errors! (errors.cljc:274).
Type Checker: Found 1 error

Prior work: There is a similar problem in ClojureScript’s compiler via the js* form. The ClojureScript analyzer added :js-op metadata to the analyzed form so linters like Typed Clojure can infer better error messages. However this only helped marginally as the expanded code was still checked, and the inlining was not always easy to infer (eg., different order of arguments).

A briefly considered approach in fixing this problem was to define a custom typing rule for each of the ~80 clojure.core inline functions. This was discarded in favor of the following once-and-for-all solution.

Approach: Check inlining before expansion, and propagate tag information after type checking. This is not possible if using tools.analyzer (as Typed Clojure did pre-2019), but is relatively straightforward with typed.clj.analyzer (see maybe-check-inlineable for the required juggling).

Results This change improved error messages for around 78 functions in clojure.core. Now inline functions never blame their expansions and unsupported inline functions consistently throw type errors in first- and higher-order contexts (instead of expanding in inline contexts and erroring in higher-order ones).

For example, inc now blames its form instead of its expansion (see in: (inc nil)).

$ clj -Sdeps '{:deps {org.typedclojure/typed.clj.checker {:mvn/version "1.0.19"}}}}'
Clojure 1.10.3
user=> (require '[clojure.core.typed :as t])
nil
user=> (t/cf (inc nil))
Type Error (NO_SOURCE_PATH:1:7)
Function inc could not be applied to arguments:


Domains:
        Number

Arguments:
        nil

Ranges:
        Number




in:
(inc nil)



Execution error (ExceptionInfo) at clojure.core.typed.errors/print-errors! (errors.cljc:276).
Type Checker: Found 1 error

Commits

Identify classes of implementation quirks in core Clojure macros to prepare for potential typing rules

Problem: To improve static type error messages for a macro, a custom typing rule is needed. However, typing rules for macros need to simulate the macro expansion of the original macro accurately in order to be sound. Some macros in clojure.core are known to leak implementation details–this would influence how typing rules are written, so we need to investigate similar issues for other macros.

Prior work:

Approach: Study the definition of macros and try and break them.

Results: I found 5 classes of implementation leakage in core Clojure macros.

  1. In macros that wrap try/finally around a body, catch syntax is leaked to the user. The following macros expand to (try (catch Exception e :foo) (finally ...)). In all of these cases, catch is not bound so we might expect an error instead of the below behavior.
$ clj
Clojure 1.10.3
user=> (binding [] (catch Exception e :foo))
nil
user=> (locking 1 (catch Exception e :foo))
nil
user=> (with-in-str "a" (catch Exception e :foo)) 
nil
  1. In macros that wrap fn around a body, a recur target is available. In all of these cases, a compilation error might be more appropriate. (also plays poorly with :once fns).
$ clj
Clojure 1.10.3
user=> (delay (recur))
#object[clojure.lang.Delay 0x3d7fa3ae {:status :pending, :val nil}]
user=> (future (recur))
#object[clojure.core$future_call$reify__8477 0x5f462e3b {:status :pending, :val nil}]
user=> (do (lazy-seq (recur)) nil)
nil
user=> (let [a (Object.)] @(delay (when a (recur)))) ;; infinite loop? no: `^:once fn*` clears bindings.
nil
user=> (let [a (Object.)] (lazy-seq (when a (recur))))
()
  1. In macros that wrap fn around a body, :pre/:post syntax is leaked to the user.
$ clj
Clojure 1.10.3
user=> (with-bindings [] {:pre [false]} 1)
Execution error (AssertionError) at user/eval164$fn (REPL:1).
Assert failed: false
  1. Double macro expansion and evaluation.
$ clj
Clojure 1.10.3
user=> (vswap! (do (prn "created") (volatile! 0)) inc)
"created"
"created"
1
  1. Unreliable :tag propagation.
$ clj
Clojure 1.10.3
user=> (set! *warn-on-reflection* true)
true
user=> (defmacro id [a] a)
#'user/id
user=> (vswap! (id (identity (volatile! 0))) inc)
Reflection warning, NO_SOURCE_PATH:1:1 - reference to field deref can't be resolved.
Reflection warning, NO_SOURCE_PATH:1:1 - call to method reset can't be resolved (target class is unknown).
1

As a result of this (and some prior) work, the following macros are now known to leak implementation details in some combination of the aforementioned ways and need special handling in potential typing rules:

Reference: