Thinking differently about software testing paradigms

Photo by Simon Berger: https://www.pexels.com/photo/brown-camel-931881/

> "The terms 'unit test' and 'integration test' have always been rather murky, even by the slippery standards of most software terminology." 
> -- [Martin Fowler](https://martinfowler.com/articles/2021-test-shapes.html)

> "...there is no such thing as a Unit Test" 
> -- [Michael Belivanakis](https://blog.michael.gr/2022/10/incremental-integration-testing.html)

## What is a Unit Test?

I typed the above question into a popular search engine, and the first three results I got back were as follows (emphasis mine)

> A unit test is a block of code that verifies the accuracy of a smaller, isolated block of application code, typically a function or method. -- [aws.amazon.com](https://aws.amazon.com/what-is/unit-testing/)

> A unit test is a way of testing a unit - the smallest piece of code **that can be logically isolated** in a system. In most programming languages, that is a function, a subroutine, a method or property. -- [smartbear.com](https://smartbear.com/learn/automated-testing/what-is-unit-testing/)

> Unit is defined as a single behaviour exhibited by the system under test (SUT), usually corresponding to a requirement. While it may imply that it is a function or a module (in procedural programming) or a method or a class (in object-oriented programming) **it does not mean functions/methods, modules or classes always correspond to units**. From the system-requirements perspective only the perimeter of the system is relevant, thus **only entry points to externally-visible system behaviours define units**. -- [Kent Beck via Wikipedia](https://en.wikipedia.org/wiki/Unit_testing)

A problem I often see with modern software testing is that [we lean toward the second definition too often](https://www.reddit.com/r/learnprogramming/comments/nwpzx9/how_can_unit_tests_be_useful_if_refactoring/): a unit is "the smallest piece of code that can be logically isolated in a system". The word "can" is carrying a _lot_ of weight in this sentence. We _can_ logically isolate just about anything.

"Is a unit a file?"

-- "Definitely not", I can hear you say.

"Assuming an object-oriented program, how about a class?"

-- "Probably not", you say with slightly less conviction.

"How about a method?"

-- "Probably", with a bit more confidence, agreeing with the first two results above.

"What if that method is 300 lines of code?"

-- "Ooh, yeah, you should probably break that out into smaller methods."

Suppose we do this. Let's break our 300-line method into, say, 10 methods of 30 lines each, as [some CS professors seem to teach their students that this is a good rule of thumb for function length](https://www.reddit.com/r/learnprogramming/comments/toynah/when_i_was_in_undergrad_they_told_us_no_function/#:~:text=In%20general%2C%20the%20most%20common,should%20only%20do%20one%20thing.).

```scala
// before
def original(x: String, y: Int): Boolean = {

 // ...
 // hundreds of lines of code
 // ...

}
```

```scala
// after
def improved(x: String, y: Int): Boolean = {

 val intermediateValueA = a(x)
 val intermediateValueB = b(intermediateValueA, y)
 val intermediateValueC = c(intermediateValueB)
 // ...
 val intermediateValueH = h(intermediateValueG)
 val intermediateValueI = i(intermediateValueH)
 
 j(intermediateValueI)

}

private def a(x: String): Long = {
 // ...
}

private def b(z: Long, y: Int): Double = {
 // ...
}

// eight more private functions...
```

All of these methods could be `private` (or whatever your language's equivalent of that is). In that case, they can only be accessed by the class which contains them. They used to be all in one method anyway, so we can be sure nobody else is using this logic anywhere else.

But now we face another decision: should we write unit tests for all of these individual methods? For many of us, our gut reaction will be "yes". This could make refactoring more difficult in the future, though, because ["the closer your tests are to the implementation the more sensitive they are to changes"](https://www.effective-software-testing.com/do-unit-tests-make-refactoring-harder).

> Anecdotally, I've worked on codebases with hundreds of tests like this, all tightly coupled to the production implementation. Adding one field to a class meant updating a hundred or more tests which didn't care about this field at all, but needed the new field in order to compile. The test changes regularly took longer to implement than the production changes.

Writing unit tests for these smaller methods might also require us to make them more `public` than they need to be; the world outside this class doesn't care about these individual methods, all it cares about is the one `improved` method which now ties them all together.

The real question is, are these functions "units" of code?

The answer is no.

As Kent Beck would say, these are not "entry points to externally-visible system behaviours".

The only externally-visible entry point in the refactored example above is the `improved` function, just as the `original` function was initially. But these pervasive ideas that...

1. large functions should be broken up into smaller ones, and
2. a "unit test" is a "test of a single function"

...combine to produce a result that is much worse than the sum of its parts: huge suites of tests tightly-coupled to unnecessarily-public production code that take too long to write and are difficult to maintain.

Outcomes like this lead many developers to believe things like...

> "Most Unit Testing is Waste" 
> -- [James O. Coplien](https://wikileaks.org/ciav7p1/cms/files/Why-Most-Unit-Testing-is-Waste.pdf)

## Kinds of Tests

Rather than thinking of tests along the traditional [unit / integration / end-to-end spectrum](https://qase.io/blog/test-pyramid/), I think it's helpful to think along a few other dimensions

1. is this test _fast_ or _slow_?
2. is this a _black-box_ test or a _white-box_ test?
3. is this test _informed by_ development or does _it inform_ development?

### Fast and Slow Tests

Let me start by asserting that "fast" is not synonymous with "good", and "slow" is not synonymous with "bad" in this context.

Fast tests are tests that run in a few seconds, milliseconds, microseconds, or less. Fast tests, therefore, must be entirely in-memory. They [do no disk IO and they make no network calls](https://gist.github.com/jboner/2841832). They can be run every single time a code change is made without being a roadblock to development speed, and should therefore be run as part of the developer's [inner loop](https://www.awwsmm.com/blog/drop-everything-and-review). Every time you compile, you can run these tests.

Slow tests take several seconds, minutes, or hours to run. The dividing line between fast and slow tests is somewhere around 2-5 seconds. Slow tests may require reading large input files from disk, doing lots of computation, or communicating across the network. That is: they are [IO, CPU, or network _bound_](https://stackoverflow.com/q/868568/2925434). [Contract tests](https://testsigma.com/blog/api-contract-testing/) (which often spin up Docker containers) and performance tests (which may run gigabytes of data or thousands of requests through the system) are examples of slow tests. These tests should be run less regularly, as they can impact development speed: before each commit to `main` / `master` is probably fine for tests shorter than a few minutes, daily or weekly might be a good cadence for tests much longer than that.

### Black-Box and White-Box Tests

Black-box tests make no assumptions about the internals of the thing they are testing. They provide inputs and assert on observable outputs, and that's it. The observable output is usually a return value from a method, but a black-box test might instead assert that a side effect has occurred, like that a log line has been written, or that a metric has been recorded, or that some state has been mutated.

White-box tests specifically test the internals of the thing they are testing. They are [_introspective_](https://stackoverflow.com/a/25199156/2925434). Tests which have assertions like "when 'x' happens, function a() should call function b()" are white-box tests. They are explicitly testing _how_ something should happen (_how_ some code is implemented), rather than testing only that it _has_ happened. Tests which rely heavily on mocking frameworks are often white-box tests, asserting that [such-and-such a method has (or hasn't) been called](https://sachithradangalla.medium.com/mockito-verify-how-to-test-method-invocations-8cf15be9bc19) in response to some inputs.

If you don't care about _how_ something is implemented -- just that that it does what it's supposed to do -- you should write a black-box test. This is usually the case, so opt for black-box tests as a default.

### Development-Informed Tests and Development-Informing Tests

Development-informed tests are written _reactively_, in that the production code is written first, and the tests are written afterward. Development-informed tests codify the behaviour of the system as-is. Traditional "unit tests" are almost exclusively development-informed tests.

Development-informing tests are written _proactively_, in that a test is written first, and the production code is written after. [Test-Driven Development (TDD)](https://en.wikipedia.org/wiki/Test-driven_development) is a software development methodology which encourages writing _only_ development-informing tests, ensuring that 100% of the system's behaviour is always codified in tests.

Development-informing tests can also provide confidence that some tricky piece of logic has been implemented correctly. For example, you might write a [regex](https://en.wikipedia.org/wiki/Regular_expression) to parse U.S. phone numbers, and -- at the same time -- add a handful of tests to ensure that you catch things like
- area codes surrounded by parentheses
- spaces vs. no spaces vs. hyphens
- the presence or absence of a `+1` country code

It can be hard to be sure -- just by staring at the regular expression -- that it catches all of these cases. Usually it's more convincing to just write a handful of simple tests to convince yourself that the most common edge cases are being handled correctly.

I always write bug fix tests in a development-informing way, as well. First, I write a test which _should_ pass, but which I expect to fail due to the presence of a bug. Then, I fix the bug in the production code, ensuring that the test now passes. This process shows that -- had the test existed originally -- it would have caught the bug. This gives confidence that the bug should not reappear in the future.

## "Most Unit Testing is Waste"

The three ways of looking at tests outlined above can provide insight into why developers like [James O. Coplien](https://wikileaks.org/ciav7p1/cms/files/Why-Most-Unit-Testing-is-Waste.pdf) believe that most unit testing is a waste of time.

### Most Unit Tests are Development-Informed

In my experience, TDD is not practiced by [_most_](https://www.reddit.com/r/ExperiencedDevs/comments/1g1xdqu/do_you_guys_use_tdd/) developers.

Most tests, therefore, are development-informed. A developer writes some production code and then writes a test, usually to ensure that some [code coverage minimum](https://www.atlassian.com/continuous-delivery/software-testing/code-coverage#:~:text=With%20that%20being%20said%20it,fairly%20low%20percentage%20of%20coverage.) is reached.

These tests are not written to catch bugs, and they are not written to help a developer think through some difficult implementation, and so their value is not immediately apparent.

### Most Unit Tests do not test "Externally-Visible System Behaviours"

As mentioned earlier, the twin practices of (1) breaking large functions up into smaller ones and (2) writing tests for each _function_ rather than for each _externally-visible system behaviour_ leads to a proliferation of tests tightly-coupled to the production implementation. These tests are, by their nature, fragile. They must be updated whenever the smallest implementation detail is changed, even if the externally-visible system behaviour is identical.

> This often happens when using mocking frameworks, since every method called on [a mocked object](https://www.digitalocean.com/community/tutorials/mockito-mock-examples) must be declared, with its return value specified.

In the worst-case scenario, developers will sometimes copy-and-paste the production implementation directly into the test, asserting that the "expected" result from the test implementation equals the "actual" result from the production code. This kind of white-box test unquestionably adds no value, even if it does increase "code coverage".

## A New Test Pyramid

The traditional [test pyramid](https://qase.io/blog/test-pyramid/) aims to emphasise to developers that they should be mostly writing "unit tests", with fewer "integration tests" and only a handful of "end-to-end tests". Although different formulations of the pyramid may use different terms for the latter two levels, almost all agree that the base of the pyramid should be composed of "unit tests". [Google recommends](https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html) a 70% / 20% / 10% split of unit / integration / end-to-end tests.

The idea is that you should cover most of the code's logic in small, fast tests which can be run over and over during the [inner loop of development](https://www.awwsmm.com/blog/drop-everything-and-review). Your integration tests should cover interactions between units; and your end-to-end tests should validate that an end user's actions result in some expected overall outcome.

That advice is fine, provided all developers agree what constitutes a "unit test" or an "integration test". Clearly this is not the case. (See the search engine results at the top of this blog post.) However, we can use the objective criteria above (fast vs. slow, black-box vs. white-box, development-informed vs. development-informing) to construct a New Test Pyramid.

![A New Test Pyramid](/blog/there-is-no-such-thing-as-a-unit-test/pyramid.png)

### The Base

Opt for black-box tests wherever possible. Where external dependencies are required, prefer [fake](https://stackoverflow.com/q/346372/2925434) implementations [rather than mocks](https://testing.googleblog.com/2024/02/increase-test-fidelity-by-avoiding-mocks.html) (and add corresponding contract tests to ensure that external dependency behaves as you think it does). This keeps the entire test in-memory, making it fast enough to run before each commit to `main` / `master`. You will find that most of your tests are these fast, black-box tests.

Note that this is _not_ synonymous with "unit test". As discussed above, traditional "unit tests" are usually fast, but are sometimes white-box tests, and are often development-informed.

### The Middle

Prefer development-informing tests over development-informed tests (prefer a TDD style of development). Development-informed tests are often written [by rote](https://www.merriam-webster.com/dictionary/rote) and offer little value.

Prefer slow black-box tests over fast white-box tests. The former are easier to maintain as they are less tightly-coupled to the production implementation.

Traditional "integration tests" and "end-to-end" tests both fall into the "slow, black-box tests" category.

### The Top

Write as few white-box tests as possible. That is, do as little code introspection as possible. Only test observable outputs.

Write development-informed tests only when necessary. If the production implementation works, it works. If it doesn't, you will find a bug, write a development-informing test, and fix the bug. This process is described above.

## Conclusion

The traditional unit / integration / end-to-end categorization of tests is fuzzy at best. Differing interpretations of what constitutes a "unit test", combined with well-intentioned but misapplied advice on keeping code readable by reducing the number of lines per function, class, etc. has led to a proliferation of hard-to-maintain, low-value test suites that negatively impact developer productivity.

Categorizing tests objectively, using the three criteria described above, can lead to more maintainable tests which provide more value.

There is No Such Thing as a Unit Test

Computer programming as an art form in and of itself

## Digital Art

In almost all circumstances, code is a means to an end.

The phrase "computer programming" itself describes the activity of _programming_ a _computer_ to accomplish a particular task. Often, that task is logical. Most computer programs do something "useful", whether that's calculating the best route from your home to work, balancing a budget, or running your [smart fridge's Twitter client](https://www.cbsnews.com/news/teen-goes-viral-for-tweeting-from-lg-smart-fridge-after-mom-confiscates-all-electronics/).

But many programs are written solely to produce something of aesthetic value.

Recently, Generative AI applications like [Midjourney](https://www.midjourney.com/showcase), which can produce visual art from text prompts, have seen a lot of press. But consider also [dwitter.net](https://www.dwitter.net/top/all), where JavaScript programs of 140 characters or fewer create moving 3D landscapes, gyrating fractals, and even [self-playing games of Pong](https://www.dwitter.net/d/135). And also programmers like [Grzegorz Witczak](https://codepen.io/Wujek_Greg/pen/LmrweG), [Ben Evans](https://codepen.io/ivorjetski/pen/xxGYWQG), and [Diana A. Smith](https://diana-adrianne.com/) who create photorealistic works of art using only [CSS](https://en.wikipedia.org/wiki/CSS), a language otherwise used mainly to style fonts, borders, and backgrounds on web pages.

[Digital art](https://en.wikipedia.org/wiki/Digital_art) is not a new phenomenon. Nearly as long as computers have existed, programmers have been using them to express themselves aesthetically.

> Traditional artists have been inspired by digital art, as well. Consider Dalí's [_Painting of Gala looking at the Mediterranean sea which from a distance of 20 meters is transformed into a portrait of Abraham Lincoln (Homage to Rothko)_](https://www.salvador-dali.org/en/artwork/catalogue-raisonne-paintings/obra/871/painting-of-gala-looking-at-the-mediterranean-sea-which-from-a-distance-of-20-meters-is-transformed-into-a-portrait-of-abraham-lincoln-homage-to-rothko), produced in 1976, which appears to show a pixelated image of Abraham Lincoln when viewed from a distance.

For most digital artists, code is a _medium_, like marble for a sculptor, or a canvas for a painter. Code is the material through which the art is _expressed_. Though it takes time and skill to produce, a canvas itself is not usually seen as art itself. But [could it be](https://www.npr.org/2021/09/29/1041492941/jens-haaning-kunsten-take-the-money-and-run-art-denmark-blank)?

## Readability

Developers often talk about writing ["clean"](https://en.wiktionary.org/wiki/clean_code) or ["elegant" code](https://www.quora.com/What-do-programmers-mean-by-elegant-code). Most programming languages are ["general-purpose"](https://en.wikipedia.org/wiki/General-purpose_programming_language), meaning they can be used to accomplish a variety of tasks. Accordingly, there are often many different ways to accomplish the _same_ task, using the same programming language.

Robert C. Martin's [_Clean Code: A Handbook of Agile Software Craftsmanship_](https://www.oreilly.com/library/view/clean-code-a/9780136083238/) gives many examples of "unclean" code, with tips and rules of thumb for rewriting the code in a "clean" way.

In the overwhelming majority of cases, the most important aspect of code is its functionality -- usually to perform some computational task. A _secondary_ aspect of the code is its _readability_, closely related to the "cleanliness", "elegance", or "aesthetics" of the code.

As a quick example, consider this 3x3 matrix

```scala
val identity = Array(1, 0, 0, 0, 1, 0, 0, 0, 1)
```

Which elements are on the diagonal of this matrix?

It's much more _readable_ if we [manually format it](https://scalameta.org/scalafmt/docs/configuration.html#-format-off) as follows

```scala
val identity = Array(1, 0, 0,
 0, 1, 0,
 0, 0, 1)
```

These two arrays are functionally identical. But _aesthetically_, the source code used to construct the latter array is more "readable" than the former. Functionality is paramount, but aesthetics are a close second.

But what if a program (or an entire programming language) were designed _primarily_ for its aesthetic qualities?

## Unreadability

Formatting code for readability -- or to follow established patterns and idioms -- is the norm. But some programmers do the opposite (intentionally or otherwise).

Every year, the International Obfuscated C Code Contest ([IOCCC](https://www.ioccc.org/)) accepts submissions of C code which accomplish some task, but do it in an unusual, hard-to-decipher way. These programs are functional, but are purposely formatted so as to be as _unreadable_ as possible (they are _obfuscated_).

Some programming languages have been designed specifically to accomplish as much as possible in as few characters as possible. [APL](https://en.wikipedia.org/wiki/APL_(programming_language)#Game_of_Life) is a "serious" (non-"esoteric") and very terse programming language which programmers may be familiar with, but many [code golf](https://en.wikipedia.org/wiki/Code_golf) languages have been created specifically to minimize source code size in a similar way.

And some programmers choose to format their code so that their entire ray-tracing C++ program can [fit on the back of a business card](https://mzucker.github.io/2016/08/03/miniray.html), or so [the source code](https://github.com/mame/quine-relay/blob/master/QR.rb) of their [ouroborous quine](https://en.wikipedia.org/wiki/Quine_(computing)#Ouroboros_programs) written in Ruby looks like a dragon (see the header image of this blog post), or so that they [look like and can be read as poems](https://code-poetry.com/). All of these examples use "serious" programming languages in an "unserious" way.

## Esoteric Languages

There are also many programming languages where the [aesthetics of the source code](https://www.awwsmm.com/blog/20-intriguing-unusual-and-goofy-programming-languages#aesthetics) is more important than what is actually written in that code. As most programming languages are written using ASCII or Unicode character sets, [some of these](https://web.archive.org/web/20240205064805/https://blue.sky.or.jp/grass/) look like [ASCII art](https://en.wikipedia.org/wiki/ASCII_art)...

```
　　　　_, ._
　　（　・ω・）　んも〜
　　○=｛=｝〇,
　 　|:::::::::＼, ', ´
､､､､し ､､､((（.＠）ｗｖｗｗＷＷｗｖｗｗＷｗｗｖｗｗｗｗＷＷＷｗｗＷｗ
ｗＷＷＷＷＷＷｗｗｗｗＷｗｗｖｗＷＷｗＷｗｗｖｗＷＷＷ
```


...though most just look like [jumbled messes of characters](https://esolangs.org/wiki/Labyrinth)

```
 72_1_108::_3+:}
 0 _ """
 11_{78_23_" " "
 4 4 ".{@
 _ " {
 1 }"
 08_100_33"""""
```

Some esoteric programming languages are [non-textual](https://esolangs.org/wiki/Category:Non-textual). The source code of these programs can be visual, as with [Piet](https://www.dangermouse.net/esoteric/piet/samples.html)

!["Hello World" in the Piet programming language](/blog/code-as-art/piet.gif)

or [Nice](https://esolangs.org/wiki/Nice) (which does not yet have a working interpreter)

!["Hello World" in the Nice programming language](/blog/code-as-art/nice.png)

Other languages are auditory, meaning programs are encoded as sound. Usually, these programs can also be written [as individual notes](https://esolangs.org/wiki/Sound), or as [sheet music](https://esolangs.org/wiki/Hello_world_program_in_esoteric_languages_(D-G)#Fugue)

!["Hello World" in the Fugue programming language](/blog/code-as-art/fugue.png)

<figure style="display:block;text-align:center">
 <figcaption>Listen to the above program</figcaption>
 <audio controls src="/blog/code-as-art/fugue-hello-world.mp3"></audio>
</figure>

In many cases, the simplest implementation of a program in these languages is not the most aesthetically pleasing. Programmers will opt for beautiful programs over efficient ones. 

For example, the following Piet program gets a number as input from the user, squares it, and prints the result

![A sample Piet program](https://thepracticaldev.s3.amazonaws.com/i/1clwo2qss6zurm2kisdw.png)

...but so does this one

![A simple Piet program](https://thepracticaldev.s3.amazonaws.com/i/i77oznhfqnyu147wxesz.png)

## Code as Art

> "There is no generally agreed definition of what constitutes art" [[Wikipedia](https://en.wikipedia.org/wiki/Art)]

Even in languages which aren't specifically designed to have "beautiful" source code, some programmers opt to format their code in ways which trade efficiency, practicality, or readability for aesthetic appeal.

There is no single, correct solution to a problem which can be solved by a program. Different programs can accomplish the same task in different ways. Weighing tradeoffs and choosing one solution over another is part of the art of computer programming.

The human desire to be creative and express one's individuality does not end where a keyboard begins. Hundreds of esoteric programming languages can attest to that.

Whether textual, visual, or auditory, code can be used as a medium to create art, but it can also _be_ art itself.

Code as Art

Given only inequality relationships between members of the set

## Sayonara Scala Steward

One of my ongoing projects at work is developing and maintaining an in-house dependency updating solution for our code.

We used to use [Scala Steward](https://github.com/scala-steward-org/scala-steward) to keep our Scala projects up-to-date, but found that it [created a lot of noise](https://medium.com/permutive/scala-steward-without-the-noise-da7424f3c315) -- failed pipelines, merge conflicts, and new versions of software being released where the only change was a patch version bump to a dependency. 

We wanted a cleaner solution
- one update PR per week per repo, containing all of the dependency updates
- PRs with failed pipelines would be assigned to a "repo owner" who could resolve them during the week
- and next week, if that PR was still open, it would be closed in favour of a new one

The last point ensures that at most one dependency update PR is open at any given time for any project. This seriously reduces the mental overhead on the developer (usually, the team lead) tasked with handling these weekly update PRs.

This project is implemented in two parts.

First, there's the [`sbt-dependency-updater`](https://gitlab.com/improving-andrew/sbt-dependency-updater), which adds an `updateDependencies` task to any [`sbt`](https://www.scala-sbt.org/)-managed Scala project (as all of ours are). This task makes it easy to update all of the dependencies for a specific repo. (Note that the way in which these dependencies are updated is pretty primitive and could use some refinement.)

Second, there's a (privately-hosted) `dependency-updater` repo which uses GitLab pipelines and the GitLab API to clone a project, run `updateDependencies` in that project, and -- if there is a nonzero number of updates -- create a branch and MR against `master` / `main`, assigning the team lead to the MR.

It's all been working very smoothly for the past few months. Noise (both cognitive and pipeline) has decreased significantly, and dependencies are as up-to-date as they have always been.

But this blog post is not about that dependency updating software; it's about how we grouped our dozens of repos into "Tiers", based on dependencies, to ensure that everything is always as up-to-date as possible.

## Tiers

A repository of code has some list of dependencies (upstream libraries which it uses to do its job).

Suppose you have some `infrastructure` library which is depended on by most projects, and a `utilities` library which depends on `infrastructure`, and that most projects also depend on `utilities`.

In order to ensure that some project `a` has the latest version of `utilities` and the latest version of `infrastructure`, ideally with no transitive version conflicts, we want to do something like the following

1. update `infrastructure`'s dependencies
2. update `utilities`'s dependencies, which include `infrastructure`
3. update `a`'s dependencies, which include `utilities` and `infrastructure`

A natural step-by-step process has emerged. Here we introduce some terminology and say that projects which can be updated at the same time belong to the same "tier". For instance, if we have another project `b` which also depends on `utilities` and `infrastructure`, then `a` and `b` can be updated simultaneously (in the same CI pipeline) without any problems. But if `b` depends on `a`, then updating `b`'s dependencies should happen _after_ updating `a`'s dependencies. `b` gets bumped into a lower tier.

This begs the question: _what is the minimum required number of tiers for some set of projects with dependencies on each other_? In other words, by analyzing their dependencies, how can we efficiently group a set of projects into tiers such that updates to higher tiers efficiently "flow" into lower tiers during the course of the week's dependency updates?

## Problem Statement

Given a set of `N` values representing projects: `a`, `b`, `c`, `d`, and `e`, let inequality relationships between the values indicate dependencies between projects. For example, `a < b` indicates that `a` depends on `b`.

What is the minimum number of unique values that can be assigned to `a`, `b`, `c`, `d`, and `e`, given some set of `M` unique inequality relationships?

"Unique equality relationships" means `a < b` and `b > a` count as only a single relationship, because they are just two different ways to express the same information.

## Discussion

I'm not planning on _proving_ this in the mathematical sense, but I think we can come up with an intuitive solution by thinking through this problem logically.

Let's start with `M = 0`. This is the case where there is _no_ inequality information. In this case, there is no reason we cannot assume `a = b = c = d = e`, and put all dependencies in the same tier. The minimum number of unique values in this case is `1`, when there is only a single tier. So in the `M = 0` case, we have no inequality information, and `1` tier is sufficient when `N = 5` and `M = 0`.

`1` is the minimum number of unique values when `N = 5` and `M = 0`.

> Note that `a`, `b`, `c`, `d`, and `e` _could_ also have five unique values. This is always possible if we assume that there _exists_ some additional information which we are just not currently aware of. But the problem we are trying to solve is the _minimum_ number of unique values _required_ to satisfy the known information. 

In the `M = 1` case we have a single unique inequality, for example `a < b`. Here, we must have at least two unique values, since `a` and `b` are explicitly not equal; `b` is in tier `1` and `a` is in tier `2`. However, `c`, `d`, and `e` can fall into either tier without contradicting any known inequality information, so `2` is the minimum number of unique values when `N = 5` and `M = 1`.

`M = 2` is where things start to get interesting, because there are two kinds of scenarios here.

First, suppose our two known inequalities are `a < b` and `b < c`. This requires `a`, `b`, and `c` to have three unique values, and so we require three tiers. But if our two known inequalities are `a < b` and `a < c`, then it's possible that `b = c`, and only two tiers are required. In fact, we could have up to four inequalities and still only require two tiers, if `a < b`, `a < c`, `a < d`, and `a < e`. In this case, `b = c = d = e`, `a` is in tier 2, and all other projects are in tier 1.

So `2` is the minimum number of unique values when `N = 5` and `0 < M <= 4`.

But when `M = 5` something changes. If our initial four inequalities are `a < b`, `a < c`, `a < d`, and `a < e`, and we add a fifth unique inequality, we must split tier `1` into two tiers. For example, if `b < c`, then `a`, `b`, and `c` must exist in different tiers, and the minimum required number of tiers is `3`.

But like the `1 < M <= 4` case, we can continue to add inequalities for a while with only three tiers. If `b < d` and `b < e`, then it's possible that `c = d = e`, `a < b`, and `b < c`, and all of our previous inequalities still hold (`a < b`, `a < c`, `a < d`, `a < e`, `b < c`, `b < d`, and `b < e`).

So `3` is the minimum number of unique values when `N = 5` and `4 < M <= 7`.

You may be starting to see where this is going. If we add another inequality, say `c < d`, we require a fourth tier. But four tiers is also enough for _another_ inequality, `c < e`.

`4` is the minimum number of unique values when `N = 5` and `7 < M <= 9`.

Finally, when `N = 5`, we can have at most `M = 10` unique inequalities. For example: `a < b`, `a < c`, `a < d`, `a < e`, `b < c`, `b < d`, `b < e`, `c < d`, `c < e`, and `d < e`. This uniquely orders all projects `a < b < c < d < e` and requires the maximum number of `5` tiers.

`5` is the minimum number of unique values when `N = 5` and `M = 10`.

Let's recap. If `k` is the minimum number of unique values required, then, when `N = 5`

- when `M = 0`, `k = 1`
- when `0 < M <= 4`, `k = 2`
- when `4 < M <= 7`, `k = 3`
- when `7 < M <= 9`, `k = 4`, and
- when `M = 10`, `k = 5`

Writing this another way

- when `M = 0`, `k = 1`, else
- when `M <= 4`, `k = 2`, else
- when `M <= 4 + 3`, `k = 3`, else
- when `M <= 4 + 3 + 2`, `k = 4`, and
- when `M = 4 + 3 + 2 + 1`, `k = 5`

The [triangular numbers](https://en.wikipedia.org/wiki/Triangular_number) have emerged. If `T(N)` is the `N`th triangular number, then

- `T(0) = 0`
- `T(1) = 1`
- `T(2) = 3`
- `T(3) = 6`
- `T(4) = 10`

So we can re-express the above as

- when `M = 0`, `k = 1`, else
- when `M <= T(4) - T(3)`, `k = 2`, else
- when `M <= T(4) - T(2)`, `k = 3`, else
- when `M <= T(4) - T(1)`, `k = 4`, and
- when `M = T(4)`, `k = 5`

Since `M` is bounded by the range `[0, T(4)]` when `N = N`, this covers all possible values of `M` and `k`.

We can postulate that, for any set of `N` values, described only by `M` unique inequalities between values, the set must be composed of at least `k` unique values, where

- `M` is bounded by the range `[0, T(N-1)]`
- when `M = 0`, `k = 1`
- when `M = T(N-1)`, `k = N`, and
- `T(N-1) - T(k) < M <= T(N-1) - T(k-1)`

## Future Work

### Optimal Number of Tiers

The above reasoning was enough to convince me that this is procedure gives the correct _minimum_ number of tiers required for staggered, grouped dependency updates. But the next step is more difficult -- actually determining what those tiers should be.

It's easy to see that when `a < b` and `b < c`, we can construct three tiers...

- tier 1: `c`, `d`, and `e` (these receive dependency updates first)
- tier 2: `b` (then this project receives dependency updates)
- tier 3: `a` (and finally, this one)

...but that `d` and `e` can be put into any tier.

However, it's much harder to reason about what the tiers should be, and where projects should be placed on them, when you have dozens of projects with dozens of dependencies. We know the _minimum_ number of tiers from the above analysis, but we've also seen that this minimum is exceeded when we have dependency chains like `a < b < c`. Does the longest dependency chain determine the "optimal" number of tiers? Can we prove this? Can we turn this process into an algorithm?

### Circular Dependencies

Another possible extension to this problem is the idea of _circular dependencies_. In the real world, it's possible (though an anti-pattern) for `a` to depend on `b` while `b` depends on `a`. Notationally, we could express this with the `<=` operator, rather than `<`. If `a <= b` and `b <= a`, then it must be the case that `a = b`, and these two dependencies should be updated at the same time, on the same tier.

This _is_ the case in my real-world system, and the way I handle this is to update any tiers with circular dependencies _twice_. Here's how that looks

- `a v1` depends on `b v0` and `b v1` depends on `a v0`
- update `a` to depend on `b v1`, producing `a v2`; update `b` to depend on `a v1`, producing `b v2`
- at this point, `a v2` still (circularly) depends on the "old" `a v0` via `b v1`, and vice versa for `b v2`
- so update `a` again to depend on `b v2`, producing `a v3`; and update `b` again to depend on `a v2`, producing `b v3`
- now, we have `a v3 < b v2 < a v1` and vice versa for `b`, and the old dependencies are no longer brought in transitively (at least two levels deep)

How does the possibility of circular dependencies affect the minimum required number of tiers? How do we express that, when `a <= b`, for example, but we have no corresponding `b <= a`, that `a` and `b` should be in different tiers?

There's a bit more to think about here, but I'll leave that for another time.

Solving for the Minimum Number of Unique Values in a Set

On prioritizing code review when developing software

## The Inner and Outer Development Loops

Feedback is critical when developing software; the earlier it can be given, the better. Static code analysis tools will _immediately_ write red squigglies underneath code, as it's being written, to let a programmer know that there's a syntax error. (In fact, as I'm writing this blog post, I'm getting blue underlines from my spellchecker.) Unit tests, similarly, give very quick feedback, validating the logic of any newly-written code, and ensuring that there are no regressions (that the new code doesn't cause previously-written tests to break).

This very fast code-build-test feedback loop is often called the [inner loop of software development](https://sourcegraph.com/blog/developer-productivity-thoughts). It requires no input from other people, and the time for a complete loop can be as short as the time it takes static analysis to run, which might be only a few hundred milliseconds. The only blockers in the inner loop are the speed of the compiler or interpreter and the speed at which the developer can program.

The _outer loop_, by contrast, usually refers to the process of deploying code to a live environment and iterating based on integration with the larger system, acceptance tests, performance tests, and so on (though [different](https://sourcegraph.com/blog/developer-productivity-thoughts) sources [disagree](https://docs.stakater.com/saap/for-developers/explanation/inner-outer-loop.html) about [which steps](https://notes.paulswail.com/public/The+inner+and+outer+loops+of+software+development+workflow) should be included in the [outer loop](https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/yes-you-can-measure-software-developer-productivity)). This loop is much more involved and, accordingly, takes much more time to provide feedback to the developer. Depending on how fast your CI / CD pipeline runs, and whether your QA tests are automated or manual, it could take anywhere from minutes to days.

But I propose that there is a _middle loop_, as well, which often gets incorrectly lumped into the outer loop: the code review loop.

## The Middle Loop

The Middle Development Loop involves getting feedback from developers and other stakeholders, adjusting your code according to that feedback, and then requesting additional feedback. It is distinct from the inner loop, which is traced by a single developer in a single codebase in a single environment, as well as from the outer loop, which is traced by many developers across many environments. The key differentiator here is that this is the first time that multiple stakeholders will be looking at the same new bit of code.

This often happens during code review, but feedback can come in many forms: pair programming, mob programming, [synchronous code review](https://glia.engineering/the-case-for-synchronous-code-reviews-51a19b76b7b7), as well as traditional asynchronous code review. The latter is, in many workplaces, the standard: you create a PR, add some reviewers, and go off and do something else while you wait for feedback.

![https://xkcd.com/303/](/blog/drop-everything-and-review/approvals.png)

This is usually fine, provided you have lots of tasks to work on and can context switch easily from one to another while waiting for reviews. And while this is often the case, in my experience, the opposite happens just as often: you're building some feature across multiple code bases, or in multiple steps, and you need reviews and approvals at each stage. In this case, each time a review is required, it becomes a blocker.

Developers can get frustrated waiting for reviews and glob multiple fixes into a single PR to reduce the number of reviews required, or they might ping other developers directly to ask for reviews. Those pinged might be taken out of their [flow state](https://github.blog/2024-01-22-how-to-get-in-the-flow-while-coding-and-why-its-important), or even feel harassed, if contacted over and over.

For all parties involved, it can be a very frustrating experience.

If you have the power to do so in your organization, push for [synchronous code reviews](https://glia.engineering/the-case-for-synchronous-code-reviews-51a19b76b7b7), which greatly reduce the size of the middle loop, or for synchronous _coding_, like pair or mob programming, which eliminates the middle loop entirely. For many reading this, though, I know this is not an option. You've got async code reviews and you're stuck with them. In that case, let's try to make the best of a less-than-ideal situation: what is it that makes async review frustrating?

The problem with async code review is that it often unnecessarily inflates the duration of the middle loop.

Code review can be a thankless job. When's the last time you heard of someone getting promoted for being the most thorough reviewer on a team? Plus, it's easy to hide in a crowd, and if 10 developers are added as reviewers to a PR which only needs 2 approvals, they may ignore the request, assuming that their teammates will pick up the slack. Even when a developer _does_ review some code, they may delay doing so. Waiting until you're finished your current task, or until after lunch, or even until after your next meeting, to review a PR may not seem like a delay for you. In fact, in may seem downright prompt. But to the developer waiting for a review to continue their work, they are blocked.

The next time _you_ put up a PR, you will _also_ need reviews. And just like your coworkers, you will want them as quickly as possible. So remember that when someone requests a code review, the courteous thing to do is to **Drop Everything and Review (DEAR)**.

## Minimize the Middle Loop, Dear

After handling any active production incidents, the first thing on a developer's `// TODO` list should _always_ be reviewing code.

If you've just sat down at your desk and are looking for something to work on, checking which pull requests you've been assigned to should become a reflex.

Don't start that little bug fix, don't read that next blog post (unless it's this one), don't check your inbox.

Remember **DEAR** and do the courteous thing: Drop Everything and Review.

Drop Everything and Review

Understanding const generics in the Rust programming language with a few examples

[I was working through an example](https://github.com/awwsmm/daily-bevy/tree/time/virtual_time) in [the repo for the Bevy game engine](https://github.com/bevyengine/bevy/blob/v0.13.0/examples/time/virtual_time.rs) recently and came across this code

```rust
/// Update the speed of `Time<Virtual>.` by `DELTA`
fn change_time_speed<const DELTA: i8>(mut time: ResMut<Time<Virtual>>) {
 let time_speed = (time.relative_speed() + DELTA as f32)
 .round()
 .clamp(0.25, 5.);

 // set the speed of the virtual time to speed it up or slow it down
 time.set_relative_speed(time_speed);
}
```

This is a function (`fn`) which takes a `mut`able argument called `time`. The type of `time`, `ResMut<Time<Virtual>>`, comes after the colon, `:`.

The thing that caught my eye here was the generic parameter: `<const DELTA: i8>`. What is that?

Here's another example from Bevy

```rust
pub unsafe fn read<T>(self) -> T {
 let ptr = self.as_ptr().cast::<T>().debug_ensure_aligned();
 // -- snip --
}
```

The `read` function takes a generic type parameter `T` and uses it in two places: in the body of the function, and as a return type. Programmers who are familiar with generics know that an unconstrained `T` is a placeholder that means "any type"; it could be `String` or `bool` or anything else.

> In languages with a global type hierarchy, like Java, a value `t: T` has _some_ operations which can be performed on it, like `.toString()`, because _every_ type `T` in Java extends the base `Object` type. Rust has no such global type hierarchy, and no root `Object` type, so there's not much at all you can do with an instance of an unconstrained type.

Going back to the first example, `const DELTA: i8` clearly already _has_ a type, appearing after the colon, `:`. (It is `i8`, an 8-bit signed integer.) So what is it doing sitting between those angle brackets (`<>`) where generic parameters usually sit?

In this position, `const DELTA: i8` is acting as a [const generic](https://doc.rust-lang.org/reference/items/generics.html#const-generics).

## What Are Const Generics?

Const generic parameters are a new ([ish](https://blog.rust-lang.org/2021/02/26/const-generics-mvp-beta.html)) kind of generic parameter in Rust, similar to type parameters (e.g. `T`) and lifetime parameters (e.g. `'a`). In the same way that a function (or method, `struct`, `enum`, `impl`, `trait`, or `type` alias) can use a generic _type_ parameter, it can also use _const generic_ parameters.

Const generic parameters are what power `[T; N]` type annotation of arrays in Rust. They are why `[T; 3]` (an array of three `T` values) and `[T; 4]` (an array of four `T` values) are _different types_, but different types which can be handled _generically_ as specific implementations of `[T; N]`.

Const generic parameters allow items to be generic over _constant values_, rather than over types.

The difference can be subtle. Here's a simple example

```rust
fn add<const b: i8>(a: i8) -> i8 {
 a + b
}
```

Here, `b` is not a "type parameter"; it is a _value_, and so it can be treated exactly as a value, used in expressions, and so on. But since it is `const`, the value of `b` must be known at compile time. For example, the following will not compile

```rust
fn example(a: i8, b: i8) -> i8 {
 add::(a) // error: attempt to use a non-constant value in a constant
}
```

The logic in this function, of course, could also be expressed like

```rust
fn add(a: i8, b: i8) -> i8 {
 a + b
}
```

...so what's the benefit of const generics? Let's look at some other examples

## Using Const Generics to Enforce Correctness

There are a few examples from linear algebra where const generics are very helpful. For example, the [dot product](https://en.wikipedia.org/wiki/Dot_product) of two vectors `a` and `b`, is defined for any two vectors of any dimensionality (length), provided they have the _same_ dimensionality

```rust
struct Vector<const N: usize>([i32; N]);

impl<const N: usize> Vector<N> {
 fn dot(&self, other: &Vector<N>) -> i32 {
 let mut result = 0;
 for index in 0..N {
 result += self.0[index] * other.0[index]
 }
 result
 }
}
```

We get a compile-time error if we try to find the dot product of two vectors with different numbers of elements

```rust
fn main() {
 let a = Vector([1, 2, 3]);
 let b = Vector([4, 5, 6]);
 
 assert_eq!(a.dot(&b), 32); // ok: a and b have the same length
 
 let c = Vector([7, 8, 9, 10]);
 
 a.dot(&c); // error: expected `&Vector<3>`, but found `&Vector<4>`
}
```

Const generics can be applied to [matrix multiplication](https://amacal.medium.com/learning-rust-const-generics-8c29fc26fad4), as well. Two matrices [can be multiplied](https://en.wikipedia.org/wiki/Matrix_multiplication) only if the first one has `M` rows and `N` columns and the second has `N` rows and `P` columns. The resulting matrix will have `M` rows and `P` columns.

```rust
struct Matrix<const nRows: usize, const nCols: usize>([[i32; nCols]; nRows]);

impl<const M: usize, const N: usize> Matrix<M, N> {
 fn multiply<const P: usize>(&self, other: &Matrix<N, P>) -> Matrix<M, P> {
 todo!()
 }
}
```

Here, we again get a compile-time error if we ignore this constraint

```rust
fn main() {
 let a = Matrix([[1, 2, 3], [4, 5, 6]]); // 2 x 3 matrix
 let b = Matrix([[1, 2, 3, 4], [2, 3, 4, 5], [3, 4, 5, 6]]); // 3 x 4 matrix
 
 a.multiply(&b); // ok: 2 x 4 matrix
 
 let c = Matrix([[1, 2, 3], [2, 3, 4]]); // 2 x 3 matrix
 
 a.multiply(&c); // error: expected `&Matrix<3, <unknown>>`, but found `&Matrix<2, 3>`
}
```

These constraints can be enforced at runtime without const generics, but const generics can help [shift these issues left](https://en.wikipedia.org/wiki/Shift-left_testing), catching them earlier in the development process, tightening [the inner dev loop](https://www.getambassador.io/docs/telepresence/latest/concepts/devloop#what-is-the-inner-dev-loop).

## Using Const Generics to Conditionally Implement `trait`s

(Adapted from [Nora's example here](https://nora.codes/post/its-time-to-get-hyped-about-const-generics-in-rust/).)

Const generics also enable really powerful patterns, like compile-type checks on values in signatures. For example...

```rust
struct Assert<const COND: bool> {}
```

...this `struct` takes a constant generic `bool` parameter, `COND`. If we define a `trait` `IsTrue`...

```rust
trait IsTrue {}

impl IsTrue for Assert<true> {}
```

...we can conditionally implement `trait`s by requiring some `Assert` to `impl IsTrue`, like so

```rust
trait IsOdd<const N: i32> {}

impl<const N: i32> IsOdd<N> for i32 where Assert<{N % 2 == 1}>: IsTrue {}
```

> The above `Assert<{N % 2 == 1}>` requires `#![feature(generic_const_exprs)]` and the `nightly` toolchain. See https://github.com/rust-lang/rust/issues/76560 for more info.

Above, `trait IsOdd` is implemented for the `i32` type, but only on values `N` which satisfy `N % 2 == 1`. We can use this trait to get compile-time checks that constant (hard-coded) `i32` values are odd

```rust
fn do_something_odd<const N: i32>() where i32: IsOdd<N> {
 println!("oogabooga!")
}

fn do_something() {
 do_something_odd::<19>();
 do_something_odd::<42>(); // does not compile
 do_something_odd::<7>();
 do_something_odd::<64>(); // does not compile
 do_something_odd::<8>(); // does not compile
}
```

The above will generate a compiler error like

```rust
error[E0308]: mismatched types
 --> src/main.rs:70:5
 |
70 | do_something_odd::<42>();
 | ^^^^^^^^^^^^^^^^^^^^^^^^ expected `false`, found `true`
 |
 = note: expected constant `false`
 found constant `true`
```

## Using Const Generics to Avoid Complex Return Types

Finally, const generics can be used to make code more readable, and more performant. The example from the beginning of this post comes from Bevy, and the reason const generics are used there is because Bevy is expecting a function pointer as an argument to a method

```rust
fn main() {
 App::new()
 // -- snip --
 .add_systems(
 Update,
 (
 // -- snip --
 change_time_speed::<1>.run_if(input_just_pressed(KeyCode::ArrowUp)),
 // -- snip --
 ),
 )
 .run();
}
```

`change_time_speed::<1>`, above, is a function pointer. We can rearrange this method to take an argument, rather than using a const generic parameter...

```rust
change_time_speed_2(1).run_if(input_just_pressed(KeyCode::ArrowUp)),
```

...but then we would have to change the return type as well

```rust
/// Update the speed of `Time<Virtual>.` by `DELTA`
fn change_time_speed_2(delta: i8) -> impl FnMut(ResMut<Time<Virtual>>) {
 move |mut time| {
 let time_speed = (time.relative_speed() + delta as f32)
 .round()
 .clamp(0.25, 5.);

 // set the speed of the virtual time to speed it up or slow it down
 time.set_relative_speed(time_speed);
 }
}
```

To many, the original function may be more readable

```rust
/// Update the speed of `Time<Virtual>.` by `DELTA`
fn change_time_speed<const DELTA: i8>(mut time: ResMut<Time<Virtual>>) {
 let time_speed = (time.relative_speed() + DELTA as f32)
 .round()
 .clamp(0.25, 5.);

 // set the speed of the virtual time to speed it up or slow it down
 time.set_relative_speed(time_speed);
}
```

Remember, as well, that Rust uses [monomorphization of generics](https://doc.rust-lang.org/book/ch10-01-syntax.html#performance-of-code-using-generics) to improve runtime performance. So not only is the const generic version of this function more readable, but it's possible (though I haven't benchmarked) that it's more performant as well. Either way, it's good to know that there are multiple ways to attack a problem, and to be able to weigh the pros and cons of each approach.

Hopefully this discussion has helped you to understand what const generics are, and how they can be used in Rust.

What Are Const Generics and How Are They Used in Rust?

Let the compiler do the hard work of data validation for you

Penrose Triangle optical illusion created out wafer biscuits. Image by Emmanuel Lefebvre from Pixabay.

Suppose you have a `Person` class in your program, and that a `Person` has an `age`. What type should the `age` be?

## `age` as a `String`

```scala
case class Person(age: String)
```

"Of course it shouldn't be a `String`" you might think. But why? The reason is that we can then end up with code like

```scala
val person = Person("Jeff")
```

If we ever wanted to _do_ anything with an `age: String`, we would need to validate it everywhere

```scala
def isOldEnoughToSmoke(person: Person): Boolean = {
 Try(person.age.toInt) match {
 case Failure(_) => throw new Exception(s"cannot parse age '${person.age}' as numeric")
 case Success(value) => value >= 18
 }
}

def isOldEnoughToDrink(person: Person): Boolean = {
 Try(person.age.toInt) match {
 case Failure(_) => throw new Exception(s"cannot parse age '${person.age}' as numeric")
 case Success(value) => value >= 21
 }
}

// etc.
```

This is cumbersome for the programmer _writing_ the code, and makes it difficult for any programmer _reading_ the code, as well.

We could move this validation to a separate method

```scala
def parseAge(age: String): Int = {
 Try(age.toInt) match {
 case Failure(_) => throw new Exception(s"cannot parse age '$age' as numeric")
 case Success(value) => value
 }
}

def isOldEnoughToSmoke(person: Person): Boolean =
 parseAge(person.age) >= 18

def isOldEnoughToDrink(person: Person): Boolean =
 parseAge(person.age) >= 21
```

...but this is still not ideal. The code is a bit cleaner, but we still need to parse a `String` into an `Int` every time we want to do anything numeric (comparison, arithmetic, etc.) with the `age`. This is often called ["stringly-typed" data](https://en.wiktionary.org/wiki/stringly-typed).

This can also move the program into an illegal state by throwing an `Exception`. If we're going to fail anyway, we should [fail fast](https://www.martinfowler.com/ieeeSoftware/failFast.pdf). We can do better.

## `age` as an `Int`

Your first instinct might have been to make `age` an `Int`, rather than a `String`

```scala
case class Person(age: Int)
```

If so, you have good instincts. An `age: Int` is much nicer to work with

```scala
def isOldEnoughToSmoke(person: Person): Boolean =
 person.age >= 18

def isOldEnoughToDrink(person: Person): Boolean =
 person.age >= 21
```

This 
- is easier to write
- is easier to read
- fails fast

You _cannot construct_ an instance of the `Person` class with a `String` `age` now. That is an invalid state. We have [made it unrepresentable](https://www.cs.rice.edu/~javaplt/411/23-spring/NewReadings/functional_programming_on_Wall_Street.pdf), using the type system. The compiler will not allow this program to compile.

Problem solved, right?

```scala
val person = Person(-1)
```

This is clearly an invalid state as well. A person cannot have a negative age.

```scala
val person = Person(90210)
```

This is also invalid -- it looks like someone accidentally entered their [ZIP code](https://en.wikipedia.org/wiki/Beverly_Hills,_California) instead of their age.

So how can we constrain this type even further? How can we make even more invalid states unrepresentable?

## `age` as an `Int` with constraints

### at runtime

We can enforce _runtime constraints_ in any statically-typed language

```scala
case class Person(age: Int) {
 assert(age >= 0 && age < 150)
}
```

In Scala, `assert` will throw a `java.lang.AssertionError` if the assertion fails.

Now we can be sure that the `age` for any `Person` will always be within the range `[0, 150)`. Both

```scala
val person = Person(-1)
```

and

```scala
val person = Person(90210)
```

will now fail. But they will fail at runtime, halting the execution of our program.

This is similar to what we saw in "`age` as a `String`", above. This is still not ideal. Is there a better way?

### at compile time

Many languages allow _compile-time_ constraints, as well. Usually this is accomplished through macros, which inspect the source code during a compilation phase. These are often referred to as [refined types](https://en.wikipedia.org/wiki/Refinement_type).

Scala has [quite good](https://blog.rockthejvm.com/refined-types/) support for [refined types](https://github.com/Iltotore/iron) across multiple libraries. A solution using the [`refined` library](https://github.com/fthomas/refined) might look something like

```scala
case class Person(age: Int Refined GreaterEqual[0] And Less[150])
```

A limitation of this approach is that the field(s) to be constrained at compile-time must be [_literal_, hard-coded](https://en.wikipedia.org/wiki/Literal_(computer_programming)) values. Compile-time constraints cannot be enforced on, for example, values provided by a user. By that point, the program has already been compiled. In this case, we can always fall back to runtime constraints, which is often what these libraries do.

For now, we'll continue with runtime constraints only, since often that's the best we can do.

## `age` as an `Age` with constraints

From simplest to most complex implementation, we moved left to right in the diagram below

```
String => Int => Int with constraints
```

This increase in complexity directly correlates with the _accuracy_ with which we're modelling this data

> "The problems tackled have inherent complexity, and it takes some effort to model them appropriately." [[source]](https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride)

The move left-to-right above should be driven by the requirements of your system. You should not implement compile-time refinements, for example, unless you have lots of hard-coded values to validate at compile time: otherwise [you aren't gonna need it](https://martinfowler.com/bliki/Yagni.html). Every line of code has a cost to implement and maintain. Avoiding premature specification is just as important as avoiding [premature generalization](https://www.codewithjason.com/premature-generalization), though it's **_always easier_ to move from more specific types to less specific types**, so prefer specificity over generalization.

Every bit of data has a _context_, as well. There is no such thing as a "pure" `Int` value floating around in the universe. An age can be modelled as an `Int`, but it's different from a weight, which could also be modelled as an Int. The labels we attach to these raw values are the _context_

```scala
case class Person(age: Int, weight: Int) {
 assert(age >= 0 && age < 150)
 assert(weight >= 0 && weight < 500)
}
```

There is one more problem for us to solve here. Suppose I'm 81kg and 33 years old

```scala
val me = Person(81, 33)
```

That compiles, but... it shouldn't. I swapped my weight and age!

An easy way to avoid this confusion is to define some more types. In this case, [_newtypes_](https://doc.rust-lang.org/rust-by-example/generics/new_types.html)

```scala
case class Age(years: Int) {
 assert(years >= 0 && years < 150)
}

case class Weight(kgs: Int) {
 assert(kgs >= 0 && kgs < 500)
}

case class Person(age: Age, weight: Weight)
```

The name _newtype_ for this pattern [comes from Haskell](https://wiki.haskell.org/Newtype). This is a simple way to ensure that we don't accidentally swap values with the same underlying type. The following, for example, will not compile

```scala
val age = Age(33)
val weight = Weight(81)

val me = Person(weight, age) // does not compile!
```

We could also use [_tagged types_](https://medium.com/iterators/to-tag-a-type-88dc344bb66c). In Scala, the simplest possible example of this looks something like

```scala
trait AgeTag
type Age = Int with AgeTag

object Age {
 def apply(years: Int): Age = {
 assert(years >= 0 && years < 150)
 years.asInstanceOf[Age]
 }
}

trait WeightTag
type Weight = Int with WeightTag

object Weight {
 def apply(kgs: Int): Weight = {
 assert(kgs >= 0 && kgs < 500)
 kgs.asInstanceOf[Weight]
 }
}

case class Person(age: Age, weight: Weight)

val p0 = Person(42, 42) // does not compile -- an Int is not an Age
val p1 = Person(Age(42), 42) // does not compile -- an Int is not a Weight	
val p2 = Person(Age(42), Weight(42)) // compiles!
val p3 = Person(Weight(42), Weight(42)) // does not compile -- a Weight is not an Age
```

This makes use of the fact that function application `f()` is syntactic sugar in Scala for an `apply()` method. So `f()` is equivalent to `f.apply()`.

This approach allows us to model the idea that an `Age` / a `Weight` is an `Int`, but an `Int` is not an `Age` / a `Weight`. This means we can treat an `Age` / a `Weight` _as_ an `Int` and add, subtract, or do whatever other `Int`-like things we want to do.

Mixing these two approaches in one example, you can see the difference between newtypes and tagged types. You must extract the "raw value" from a newtype. You do not need to do this with a tagged type

```scala
// `Age` is a tagged type
trait AgeTag
type Age = Int with AgeTag

object Age {
 def apply(years: Int): Age = {
 assert(years >= 0 && years < 150)
 years.asInstanceOf[Age]
 }
}

// `Weight` is a newtype
case class Weight(kgs: Int) {
 assert(kgs >= 0 && kgs < 500)
}

// `Age`s can be treated as `Int`s, because they _are_ `Int`s
assert(40 == Age(10) + Age(30))

// `Weight`s are not `Int`s, they _contain_ `Int`s
Weight(10) + Weight(30) // does not compile

// To add `Weight`s, we must "unwrap" them
Weight(10).kgs + Weight(30).kgs
```

In some languages, the "unwrapping" of newtypes can be done automatically. This can make newtypes as ergonomic as tagged types. For example, in Scala, this could be done with an _implicit conversion_

```scala
implicit def weightAsInt(weight: Weight): Int = weight.kgs

// `Weight`s are not `Int`s, but they can be _converted_ to `Int`s
Weight(10) + Weight(30) // this now compiles
```

## Further refinements

The important point of the above discussion is that, as much as possible, we want to _make invalid states unrepresentable_.

- "Jeff" is an invalid age. Age isn't a string, it is a number.
- -1 is an invalid age. Age cannot be negative, it should be 0 or positive, and probably less than about 150.
- My age is not 88. An age should be easily distinguishable from other integral values, like weight.

Everything discussed above implemented these refinements on the concept of "age", one at a time.

We can make further refinements if there is a need for those refinements.

For example, suppose we want to send a "Happy Birthday!" email to a `Person` on their birthday. Rather than an `Age`, we now need a date of birth.

```scala
case class Date(year: Year, month: Month, day: Day)

case class Year(value: Int, currentYear: Int) {
 assert(value >= 1900 && value <= currentYear)
}

case class Month(value: Int) {
 assert(value >= 1 && value <= 12)
}

case class Day(value: Int) {
 assert(value >= 1 && value <= 31)
}

case class Person(dateOfBirth: Date, weight: Weight) {
 def age(currentDate: Date): Age = {
 ??? // TODO calculate Age from dateOfBirth
 }
}
```

The amount of information provided by `dateOfBirth` is strictly greater than the amount of information provided by `Age`. We can calculate someone's age from their date of birth, but we cannot do the opposite.

The above implementation leaves much to be desired, though -- there are lots of invalid states. A better way to implement this would be for `Month` to be an enum, and for `Day` validity to depend on the `Month` (February never has 30 days, for example)

```scala
case class Year(value: Int, currentYear: Int) {
 assert(value >= 1900 && value <= currentYear)
}

sealed trait Month

case object January extends Month
case object February extends Month
case object March extends Month
case object April extends Month
case object May extends Month
case object June extends Month
case object July extends Month
case object August extends Month
case object September extends Month
case object October extends Month
case object November extends Month
case object December extends Month

case class Day(value: Int, month: Month) {
 month match {
 case February => assert(value >= 1 && value <= 28)
 case April | June | September | November => assert(value >= 1 && value <= 30)
 case _ => assert(value >= 1 && value <= 31)
 }
}

case class Date(year: Year, month: Month, day: Day)
```

Always prefer low-cardinality types to [high-cardinality types](https://www.seas.upenn.edu/~sweirich/types/archive/1999-2003/msg01233.html), when possible. It limits the number of possible invalid states. In most languages, `enum`s are the way to go here (in Scala 2, an `enum` can be modelled using a `sealed trait`, as shown above). But there are still invalid states hiding above. Can you find them?

In some cases, stringly-typed data validated using regular expressions can be replaced entirely by `enum`s. Could you model [Canadian postal codes](https://www150.statcan.gc.ca/n1/pub/92-153-g/2011002/tech-eng.htm) such that it's impossible to construct an invalid one?

Use the above knowledge to go forth and _make invalid states unrepresentable_.

Make Invalid States Unrepresentable

Building a Rust IoT MVP over the Holidays

What This Senior Developer Learned From His First Big Rust Project

A Case Study in Building a Website using Next.js


## Trade-Offs in Software Development

> "Where's all my CPU and memory gone?"
>
> -- [thegeomaster](https://news.ycombinator.com/item?id=14868999)

### Software development is -- and has always been -- about trade-offs.

The [CAP theorem](https://en.wikipedia.org/wiki/CAP_theorem) tells us that we need to choose between consistency, availability, and partition tolerance when designing distributed data stores.

[Caches](https://www.indeed.com/career-advice/career-development/cache-pros-and-cons) can make information available more quickly, but it might be out of date, and it will definitely use more storage space than making a fresh request each time.

And, as always, money can be a factor. [SSDs are faster than HDDs](https://reviewed.usatoday.com/laptops/features/ssd-vs-hdd), but they are more expensive per GB. And microservices in a distributed system should each have their own database, but cloud computing costs can mean that shared DBs are more cost-effective.

Weighing the pros and cons of varied solutions to a problem was something I needed to tackle recently when redesigning part of my personal website, [awwsmm.com](https://awwsmm.com). Here's how that went down...

## A Case Study: My Website

### The Setup

My website is a pretty minimal Next.js site, written in TypeScript, hosted [on GitHub](https://github.com/awwsmm/awwsmm.com), built and deployed [by Vercel](https://vercel.com/).

In general, Next.js allows you to have two kinds of pages: static pre-rendered pages, and dynamic pages, rendered "just in time", when a visitor to your website tries to visit that page.

My website has static [blog posts](https://www.awwsmm.com/blog), as well as ["project" pages](https://www.awwsmm.com/projects), which give a quick overview of personal projects that I've been working on lately.

My initial design was for a given project page (for example, [this one for my website itself](https://www.awwsmm.com/projects/awwsmm.com)) to have an up-to-date commit history, interleaved with occasional "log entries", summarizing big sweeping changes made to these projects. (Kind of like release notes, but with more detail behind _why_ certain changes are being made.)

So what was the problem?

### The Problem

The problem was that these project pages were statically rendered; they were built in advance.

I would [request the commit history of a project from GitHub](https://github.com/awwsmm/awwsmm.com/blob/cfec0c66a0cf1ba8cc54459ee77c85b707d21d61/lib/utils/ProjectUtils.ts#L59) and write it to a local cache, saved in the repository.

When [a new PR is opened](https://github.com/awwsmm/awwsmm.com/pull/74) against the repo containing my website, Vercel runs a test deployment. When the test deployment looks good, I hit "merge" and the new changes are added to the repo in a merge commit.

But, because the cache is created only when I'm developing locally, _this merge commit is not a part of the commit history in the cache._ It couldn't possibly be. It would require updating the cache file, which would introduce changes not included in that commit, which would require another commit, ad infinitum.

This means that the commit history for the `awwsmm.com` project page is _always_ at least one commit behind `master`.

This bothered me, and I wanted to see if I could fix it.

### First Attempt: Environment-Aware Caching

[My first attempt at a solution](https://github.com/awwsmm/awwsmm.com/commit/569a6368f7bd115a13aa219b398b4bf23eab04ad#diff-e1998422d9b0a8903593c437a2bf074f2d47e10bede05c333e3a2312797ca486) was (what I'm going to call) "environment-aware caching".

What if, after the deployment passed and the new PR was merged into `master`, Vercel ignored the cache, only using it as a backup? During deployment, we could hit the GitHub API again, which should then have the new commit, right?

This required knowing which environment the build was running on. This is straightforward, as Vercel populates [a `VERCEL_ENV` environment variable](https://vercel.com/docs/concepts/projects/environment-variables/system-environment-variables) to `"production"`, `"preview"`, or `"deployment"`, depending on where the build is running.

But this also required maintaining a cache which we hopefully would never fall back to, which seemed kind of silly.

It also meant that each release would have to be deployed at least twice: once to satisfy the checks before merging the PR, and once after the PR was merged to pick up the new commit on `master`. I'd have to remember to do this "double deployment" to keep things up-to-date.

Finally, there was some human error here in that I tried to use [the same caching mechanism for "last updated" dates on my blog posts](https://github.com/awwsmm/awwsmm.com/commit/cb38f6411859de4586f945eaae6231ebcd87a78f#diff-8779cf4bd89603a2f8a50f4ea24460afbd79e6d8b65ba8575183d02f47cf074b), confusing the issue.

All in all, this solution was pretty complex to maintain (all I want is a static blog), and it put me off of keeping my website up-to-date for a while. When I finally came back here after a few months, I decided that a simpler solution was in order.

### Second Attempt: Server-Side Rendering

So how about trying to [render the project pages "just in time"?](https://github.com/awwsmm/awwsmm.com/commit/5daf1b29cbdcafa7f26f6fed340adbfc6482df2e#diff-dc938814c5900cd48c67f3c5b743ac02c3d0730245e9fa633d02e9fef4cdb801R104)

Vercel's [server-side rendering (SSR)](https://vercel.com/blog/nextjs-server-side-rendering-vs-static-generation) also generates static pages, but it renders them only when the user navigates to the page, not during deployment.

"This is great!" I thought. I could just request the commit history when the page is requested, and it would always be up-to-date.

Unfortunately, requesting and processing 100 commits from GitHub seemed to be too much to ask. I was consistently waiting about 3 seconds for the page to load, which is [really bad](https://web.dev/ttfb/). Project pages with shorter commit histories loaded a bit faster, but there was still a noticeable delay. Sending the request to GitHub, awaiting a response, processing the response, generating the resulting page, and displaying it just took too much time.

This approach would also send a request to GitHub _every time_ a user loaded that page.

This second issue could be solved by [fine-tuning the `Cache-Control`](https://github.com/awwsmm/awwsmm.com/commit/d6eb611eb5d9c746670cceeffc0fecfe32fd0817#diff-dc938814c5900cd48c67f3c5b743ac02c3d0730245e9fa633d02e9fef4cdb801R105-R123) header sent along with the request, such that I could guarantee that the 5000-requests-per-hour (authenticated) limit would never be exceeded.

But the first issue remained a problem.

### Third Attempt: Non-Blocking Server-Side Rendering

"Maybe I'm the problem" I thought.

"Maybe it's the way I (think I) am blocking inside of `getServerSideProps`."

[I rearranged this method](https://github.com/awwsmm/awwsmm.com/commit/ea8ec00ed8905f55d6bb780df5535dcb459af638) to not `await` anywhere, but return a `Promise` which is a result of other `Promise`s chained together with `then`, and nothing else.

I was hoping that, with everything done in a non-blocking way in the return value, Vercel could work some magic to speed up the call. (Maybe it could run the request as soon as the user _hovered over_ the link to the page?)

But it didn't help. I was still stuck at ~3 seconds of loading time. Worth a try, at least.

### Fourth Attempt: GraphQL

"Maybe the response from GitHub is taking so long because it's returning too much data?"

I wasn't using _most_ of the response anyway; all I cared about was the commit hash, the message, and the date. I'd never used GraphQL before, but I knew that it could be used in situations like this, where you wanted to request only _particular_ data from an endpoint.

So I learned enough about GraphQL and GitHub's API to [request only the commit data I cared about](https://github.com/awwsmm/awwsmm.com/commit/f6260bbc9bd1d181fbf3f613a73008f1cfe53e4c#diff-93b7ed2c3a46038993bf9d2c1478cc9dbb3f14e3beeafd30cb21983fbfc75befR67-R79).

This sped up the page a bit _locally_ but not in production on Vercel. It was still taking about 3 seconds.

### Fifth Attempt: Timeout / Fallback to Cache

"Well, if it's loading fast enough locally, but not remotely, maybe I can set a timeout threshold?"

My thought was that I could [cache](https://www.google.com/search?q=time+is+a+flat+circle) a "stale" version of the page to display if the "live" version of the page took too long to load. Maybe longer than 500ms or so.

So I would now [generate a cache when building locally](https://github.com/awwsmm/awwsmm.com/commit/927db29fb8cdfb53402d8a7aeff047503ec36ff1#diff-93b7ed2c3a46038993bf9d2c1478cc9dbb3f14e3beeafd30cb21983fbfc75bef), and save the cache to the repo. In production, I would attempt to request fresh data from GitHub, but if it took longer than 500ms, I would fall back to the cache.

...but it still took 3 seconds to render the page remotely.

"Why?" I thought, pulling my hair out in frustration.

Other project pages loaded quickly... maybe Vercel was just (somwehow?) slow to process this 600-line JSON file, generate the hundreds of components for it, style all of them, and display them.

I found [issues on the `vercel` repo](https://github.com/vercel/vercel/discussions/7961) of people complaining about similar problems. Maybe it just took a few seconds for Vercel to spin up a runner to render the page? (That didn't seem to square with the load times of the other project pages.)

I took solace in the fact that other people on the Internet _also_ thought that [`getServerSideProps` is weird](https://www.youtube.com/watch?v=1er7Zqs_h9k) and unintuitive.

I assessed other possible avenues for investigation: [timing the response?](https://web.dev/custom-metrics/?utm_source=devtools#server-timing-api), [actually reading the docs? (no thanks)](https://vercel.com/blog/nextjs-server-side-rendering-vs-static-generation), incremental static generation?, [edge functions?](https://vercel.com/docs/concepts/functions/edge-functions).

I was beginning to lose the will to live... all I wanted was a blog.

## Regrouping

I took a breather and came back to it all after a little while.

"What is it I actually want?"

- up-to-date commit histories
- fast page loads

The problem was that those two things were -- if not mutually exclusive, at least -- in competition with each other. Up-to-the-second commit histories would require a request to GitHub as soon as the user requested the project page. Which meant that all of the data fetching, and processing, and rendering would have to be done _quickly_.

This didn't seem like too big of an ask to me, but apparently it was. Maybe it was my code, maybe it was the Lambda cold start times, maybe it was something on Vercel's end... whatever it was, it was pinning me to 3-second page load times.

"Maybe I can request and process _most_ of the history in advance, then only request _recent_ history when a user clicks on a page?"

This sounded overcomplicated, though. (Unlikely, but) what if a project had had 100 new commits since I last deployed my website (and generated the cache)? Then I would have the same problem: trying to request and render 100 commits onto this history page.

[So I compromised](https://github.com/awwsmm/awwsmm.com/commit/34ee82581280b28da6041b4966da0d423665b885), and came up with a simpler solution:

1. generate `project/` pages statically

   This solves the load time issue -- by the time we get to production, the pages will have already been generated, so nothing needs to be done except displaying them.
   
   But this means that we will not have an up-to-date commit history.

2. accept that the commit history will not be up-to-date

   I'd already sunk so much time into this, I was ready to compromise.

   Nobody coming to my website will care if my commit histories are slightly out of date. So I decided to just put a disclaimer at the top and bottom of the histories, saying something like "for the most up-to-date commit history, see GitHub".

Now, we can generate pages in advance, keep load times to basically zero, and we don't have to worry about complex caching solutions, or what environment the build is running in, or GitHub rate limits, or anything.

My compromise was that my time and energy were worth more to me than having this one feature on my website be exactly the way I had envisioned it initially.

## Software Development is About Compromise

In my case, I tried and tried to get my website to do what I wanted it to, but my two requirements: up-to-date commit histories and fast loading times, were in direct competition with each other.

Having an up-to-date history will require a request when the user clicks the button, and processing the result of that request, which will take time.

Surely there are more complex solutions which balance these two better, but in the end, having these pages be up-to-date is not critical for my blog. Choosing where to use my time is another trade-off. I learned a lot about Vercel, GraphQL, and Promises in JavaScript / TypeScript during this process, but ultimately, "see GitHub" is good enough for me. And that's a compromise I'm willing to accept.

Software Development is About Compromise

Ergonomic Multi-Line Strings for bash, zsh, and Other Shells

## The State of Multi-Line Strings

Multi-line strings in shells are a pain.

Suppose you want to create a file, using a shell script, which contains the following content

```ts
export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
}
```

How can this be achieved?

### Some Methods

#### Method 1: a multi-line variable

This simple solution works when the variable definition is not indented at all

```sh
var="export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
}"
```

```
$ echo $var
export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
}
```

But what if we're defining `$var` within a function, and we want it indented along with the rest of the function body?

```sh
function my_function() {
 var="export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
 }"
 echo $var
}
```

```sh
$ my_function
export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
 }
```

Oh, well, that's obviously not what we want.

So, simple variable assignment: it works in a very limited subset of cases, when the variable definition is not indented at all. Let's try another method.

#### Method 2: a single-line variable with '`\n`'s for line breaks

In this method, we replace all of the line breaks in the multiline string with `\n` line break characters:

```sh
function my_function() {
 var="export default class Greeter {\n greet(name: string) { return 'Hello, ' + name + '!'; }\n}"
 echo $var
}
```

```sh
$ my_function
export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
}
```

The result looks good, but the method is messy. What if we want to reformat this like

```ts
export default class Greeter {
 greet(name: string) {
 return 'Hello, ' + name + '!';
 }
}
```

That would involve adding more `\n` characters, and spaces to match the indentation. It's not extremely straightforward:

```sh
function my_function() {
 var="export default class Greeter {\n greet(name: string) {\n return 'Hello, ' + name + '!';\n }\n}"
 echo $var
}
```

So, explicit line break characters: this works if the text you want formatted won't change often, and if the readability of the implementation doesn't matter. If you want the text-generating code itself to be readable or maintainable, this is not a great solution.

So what else can we do?

#### Method 3: Heredocs

Multiline strings are what [Heredocs](https://en.wikipedia.org/wiki/Here_document) were made for:

> In computing, a here document (here-document, here-text, heredoc, hereis, here-string or here-script) is a file literal or input stream literal: it is a section of a source code file that is treated as if it were a separate file. The term is also used for a form of multiline string literals that use similar syntax, preserving line breaks and other whitespace (including indentation) in the text.

So let's see how well they work for our problem

```sh
function my_function() {
 var=$(cat <<EOF
 export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
 }
 EOF)
 echo $var
}
```

```sh
$ my_function
/Users/andrew/test.sh:8: parse error near `var=$(cat <<EOF'
```

Oh, uh, yeah, obviously the delimiter sequence (`EOF` in this case), cannot be indented, and must appear on a line by itself, so we have to write

```sh
function my_function() {
 var=$(cat <<EOF
 export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
 }
EOF
 )
 echo $var
}
```

...which is fine, but sort of breaks indentation of the rest of the function body. It also doesn't work:

```sh
$ my_function
 export default class Greeter {
 greet(name: string) { return 'Hello, ' + name + '!'; }
 }
```

Just like Method #1, this method adds to the output the whitespace we used to indent the function body, which we don't want.

So how can we preserve _only_ the indentation _we want_ (and maybe get rid of that ugly heredoc delimiter)?

### My Method

Here's how I do it

```sh
function my_function() {
 var="$(sed -e 's/^[ ]*\| //g' -e '1d;$d' <<'--------------------'
 | 
 | export default class Greeter {
 | greet(name: string) { return `Hello, ${name}!`; }
 | }
 | 
--------------------
 )"
 echo $var
}
```

I use pipe characters `|` to define a "margin", which I then strip out using `sed`. `sed -e 's/^[ ]*\| //g'` will remove any number of space characters (`[ ]*`) at the beginning of the line (`^`), followed by a pipe (`|`), followed by one space character (`[ ]`).

> This "margin" method was inspired by [Scala's `String#stripMargin` functionality](https://www.oreilly.com/library/view/scala-cookbook/9781449340292/ch01s03.html), which behaves in a very similar way.

The second `sed` expression, `-e '1d;$d'`, removes the first and last line. I add blank lines to provide a bit of visual whitespace around the content I want to write to the variable. If you don't want one or both of these blank lines, remove them with this slight variation on my method

```sh
function my_function() {
 var="$(sed -e 's/^[ ]*\| //g' <<'--------------------'
 | export default class Greeter {
 | greet(name: string) { return `Hello, ${name}!`; }
 | }
--------------------
 )"
 echo $var
}
```

I don't mind the line of hyphens, either, as the `EOF` replacement, because it sort of acts like the top and bottom margin of the content. But, if you put the heredoc delimiter in quotes, as I have above, you can also include whitespace in it. So you could do something like

```sh
function my_function() {
 var="$(sed -e 's/^[ ]*\| //g' <<' +'
 | export default class Greeter {
 | greet(name: string) { return `Hello, ${name}!`; }
 | }
 +
 )"
 echo $var
}
```

Though I personally think this leaves a bit too much whitespace under the content. Also, many syntax highlighting algorithms have trouble with this.

With any of these variations, you can indent the content to whatever level you like

```sh
function my_function_1() {
 var="$(sed -e 's/^[ ]*\| //g' <<'--------------------'
 | export default class Greeter {
 | greet(name: string) { return `Hello, ${name}!`; }
 | }
--------------------
 )"
 echo $var
}

function my_function_2() {
 var="$(sed -e 's/^[ ]*\| //g' <<'--------------------'
| export default class Greeter {
| greet(name: string) { return `Hello, ${name}!`; }
| }
--------------------
 )"
 echo $var
}

function my_function_3() {
 var="$(sed -e 's/^[ ]*\| //g' <<'--------------------'
 | export default class Greeter {
 | greet(name: string) { return `Hello, ${name}!`; }
 | }
--------------------
 )"
 echo $var
}
```

```sh
$ my_function_1; my_function_2; my_function_3
export default class Greeter {
 greet(name: string) { return `Hello, ${name}!`; }
}
export default class Greeter {
 greet(name: string) { return `Hello, ${name}!`; }
}
export default class Greeter {
 greet(name: string) { return `Hello, ${name}!`; }
}
```

The flexibility -- combined with the visual aesthetics -- of this method is why it's recently become my go-to for multi-line strings in the shell.

Elegant Multi-Line Shell Strings

Debugging a New Feature in a Vercel TypeScript GitHub Repo

**Part III: The Deep End**

At this point, I wasn't sure where to go next.

Why should [these blog posts now have a seemingly random publication date](https://www.awwsmm.com/blog/whats-wrong-this-time-part-2), which was not the current date, and was not the date they were first added to the repo? What was going on?

So I did what most programmers do in a situation like this: I added some `console.log()`s.

## Digging Deeper

I wanted to know what commits were available, and what information they had. In other words, I wanted better [_observability_](https://dzone.com/articles/rethinking-programming-automated-observability) into what was going on in this bit of the codebase.

My first idea was to just [print out the dates of every line returned from `git log`](https://github.com/awwsmm/awwsmm.com/commit/a43bf7c356aa4b649ddfbf26528a454ae94bd807). The output of that (for a given blog post) looked something like

```
lines returned from git log
2022-01-09T20:48:23+00:00
2022-01-06T09:18:41+00:00
```

...okay, not extremely useful. Why are there only two commits here? This is just the same information we had on the website. Maybe [printing out the hash of each commit](https://github.com/awwsmm/awwsmm.com/commit/0477f384c0cb604c8cd4b64307c1b37b150575dc) would be a bit more helpful? Then I could check if there was anything unusual about these commits

```
lines returned from git log
69e038a919e448251fa2211a9fcf3fda914812fe @ 2022-01-09T20:48:23+00:00
d5cf8fbc05891ac9d8d7067b5cb1fb195dc2cf99 @ 2022-01-06T09:18:41+00:00
```

Now we can search GitHub for that commit [`dc2cf99`](https://github.com/awwsmm/awwsmm.com/commit/d5cf8fbc05891ac9d8d7067b5cb1fb195dc2cf99).

But this commit doesn't add or update _any_ blog posts... so why is it being returned from `git log <path/to/blogpost>`?

What if I `git log`ged a file that has _definitely_ been around since the first commit, like `index.tsx`. I tried printing out every log line for this file and saw the following on Vercel

```
lines returned from git log index.tsx
69e038a919e448251fa2211a9fcf3fda914812fe @ 2022-01-09T20:48:23+00:00
88c420835d35a008de808b7cef04980a15b029bc @ 2022-01-09T12:55:49+00:00
0a882cf5062e4c0ac4505ed609ca77f14b35a76a @ 2022-01-08T20:15:44+00:00
d4a9a360c38398cdd41825aa0fe193e8176cb4fd @ 2022-01-07T22:41:52+00:00
3acb76c1f6c6d1b4cdb76939496e251220aa29ea @ 2022-01-06T20:09:17+00:00
```

It only goes back five commits! The commit history looked the same for other long-lived files as well. Only ever going back to that last commit on January 6.

Running the same code on my local machine gives many more commits, going back all the way to the first commit on January 2.

What gives?

## The Shallow End

At this point, I wasn't sure how much more debugging I could do. So I started doing a bit of research.

And I found [this issue ("How to unshallow repo?")](https://github.com/vercel/vercel/discussions/5737) on the Vercel GitHub repo

> _"Hello, I need to define a variable at build time that depends on `git describe` which depends on git history, but it seems the repo in vercel build enviroment is a shallow clone with only few last commits."_

That sounds like my problem! And it sounds like it's caused by Vercel making [a _shallow clone_ of my git repo](https://www.perforce.com/blog/vcs/git-beyond-basics-using-shallow-clones) before building. I'd never encountered shallow cloning _in the wild_ before, but I knew of it as a concept, which is how I found that GitHub issue.

So how can we work around this? We simply won't have the information available at build time to determine the correct "published" and "last updated" dates for a given blog post.

But there's always a way to work around these kinds of limitations. In this case, that involves a cache.

## Cache Rules Everything Around Me

There are a few ways we could solve this problem. We could, for instance, use the GitHub API to pull commit information from the repo hosted on GitHub.com. I chose not to do this as I preferred to keep the solution self-contained: we have all the information available at build time _when running locally_, so how could we make that information available when building _for production_ (on Vercel), as well? (Where we'll have a shallow clone of the repo.)

Rather than make API calls over the internet for information which is available locally, I thought we could simply save this information in a cache, and then use that cache when building on Vercel.

The workflow I came up with for writing blog posts (and caching the important git info) looked something like this

1. draft a `wip-` post (these are [ignored for version control by my `.gitignore`](https://github.com/awwsmm/awwsmm.com/blob/4281812447b58b08bcd613530e4d8db2b1855be6/.gitignore#L23))
2. when the draft is ready, `git commit` it to the `development` branch and push to Vercel
3. for...
 - new blog posts (where the only commit in `git log` is the current commit), Vercel assumes the post is brand-new and uses the date of the current commit for the "published" and "last updated" times
 - old blog posts (where more than one commit references this blog post), Vercel looks for cached "published" and "last updated" times, and throws an error if it doesn't find any

There are a few small problems with this.

First, when do we update the cache? You'll notice that there is no step in the workflow above for ensuring that the cache is up-to-date. Since we only have access to the required information when building locally, we have to update the cache when building locally. But when does this information get pushed to the remote repo? We have to enforce that, as well.

Second, the above workflow has a problem when we merge the `development` branch into the `master` branch when promoting a new release to production -- the merge commit itself means that the "new" blog post is now in two commits. As outlined above, this will cause Vercel to throw an error if the post isn't in the cache (it won't be).

## So... What Now?

I've got some hacky fixes for the above problems implemented.

For instance, I've got a [pre-push git hook](https://github.com/awwsmm/awwsmm.com/blob/4281812447b58b08bcd613530e4d8db2b1855be6/git-hooks/pre-push) which runs a build before each `git push`. This means that -- in theory -- the cache is always up to date. But of course, I need to make sure to `git add` it in the _next_ commit.

As for the "merging creates a new commit" issue, I've tried two solutions so far.

The first was to distinguish between commits on the `development` branch and commits on the `master` branch. Only blog posts with commits on `master` should be considered as "old". This works great when running locally, but the clone that Vercel creates seems to _rename_ this `development` branch to `master` when building a preview deployment. So that's a no-go.

The second solution (which I'm currently using) is to simply [ignore merge commits](https://github.com/awwsmm/awwsmm.com/blob/4281812447b58b08bcd613530e4d8db2b1855be6/lib/blog/SlugFactory.ts#L28).

So far, the above appears to be working. But it feels like an overly complex and fragile solution, and I hope to improve upon it in the future. Maybe just querying GitHub for the commit history is easier than going through all of this cache trouble.

## Conclusion

So that's it! The goal was simple: get rid of arbitrary "published" times on blog posts and pull that data directly from the project's git history. But the solution ended up being much more complex and nuanced than I had initially planned.

But along the way, I learned some new tools and tricks, I learned a bit more about how my repo is built and deployed on Vercel, and I have some ideas for how I can make things more streamlined in the future. And that's what this is all meant to be, really, a learning experience.

In the future, maybe I'll do away with this overly-complex caching mechanism, but I do want to get the "published" and "last updated" dates from the repo's git history. This initial solution, while messy, does the job for now.

What's Wrong This Time? Part III: The Deep End

**Part II: The Bugs**

There's an old programming joke that goes [something like](https://martinfowler.com/bliki/TwoHardThings.html)

> _There are only two hard problems in computer science: cache invalidation, naming things, and off-by-one errors._

I think we should add a third (fourth?) problem to that list: sorting things.

## Sorting Things

There are [lots of different ways to sort things](https://en.wikipedia.org/wiki/Sorting_algorithm) in computer science. C.S. students learn about time and space complexity of these sorting algorithms, YouTubers make [cool visualisations](https://www.youtube.com/watch?v=BeoCbJPuvSE) of them, and occasionally, [a guy named Tim](https://en.wikipedia.org/wiki/Timsort) will invent a new one.

But there's one aspect of sorting algorithms that -- for me, at least -- seems completely impossible: remembering in which direction things are sorted.

If you say to a group of people: "okay, everyone, stand in a single-file line, ordered by height", the next question you might ask is "okay, but in which direction?" Who should stand at the _front_ of the line? The shortest person or the tallest person?

In programming, we define comparison functions, which describe how to order whatever objects we're interested in.

Some comparison functions seem obvious. For example, in TypeScript, using the default `string` comparison...

```ts
const array: string[] = ["cherry", "apple", "banana"]
array.sort()
//...
```

...we would expect `array` to be sorted alphabetically, with `apple` as the first (`0`th) element of the sorted array

```ts
//...
console.log(array) // [ 'apple', 'banana', 'cherry' ]
console.log(array[0]) // apple
```

> Note that `<array>.sort()` in JavaScript sorts the array "in place", so that the original, unsorted `array` no longer exists afterward. In some languages, and for some sorting algorithms, arrays are not sorted in place, and a new array will be returned. This new array should be assigned to a new variable.

But often we will be working with objects more complex than `string`s, and we will need to define custom comparison functions. These are functions which take two elements of type `T` and return a `number`, and are used to sort arrays of type `T`:

```ts
type T = string

const newArray: T[] = ["cherry", "apple", "banana"]

function comparison(t1: T, t2: T): number {
 return t1.charCodeAt(0) - t2.charCodeAt(0)
}

newArray.sort(comparison)

console.log(newArray) // ?
console.log(newArray[0]) // ?
```

Without [reading the docs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort), will the `console.log()`s above give the same result as the earlier ones? How about something a bit simpler -- sorting an array of `number`s:

```ts
type T = number

const newArray: T[] = [42, 2112, 19]

function comparison(t1: T, t2: T): number {
 return t2 - t1
}

newArray.sort(comparison)

console.log(newArray) // ?
console.log(newArray[0]) // ?
```

Will the first element above be `19`? Or `2112`? Are you sure?

I understand the utility of sorting algorithms, and I understand the need for a ternary (greater than, less than, or equal) return value, and hence `number` as the return type instead of `boolean`, but comparison functions are just one of those things that I've always had to test every time. Sometimes in development, and sometimes in production.

## So What Happened?

With what we learned above, you should now be able to see what went wrong with [my initial code](https://github.com/awwsmm/awwsmm.com/commit/69e038a919e448251fa2211a9fcf3fda914812fe). The problem was here

```ts
 // get the blog post date from its git commit date
 const gitLog = SlugFactory.git.log({ file: `blog/${slug.params.slug}.md` });

 return gitLog.then(lines => {
 const dates = lines.all.map(each => each.date);

 // if blog post hasn't been committed yet, use current date
 const date = dates[0] ?? new Date().toISOString();

 return new FrontMatter(slug.params.slug, title, description, date, rawContent);
 });
```

`git log` returns commits sorted by date, such that _newer_ commits come first and _later_ commits come afterward. So `dates[0]`, above, is the _newest commit_ returned from `git log`, and each blog post was being given a "publication" date of the most recent commit in which that post was modified.

When were these blog posts most recently modified? Well, all of them were modified in [that same commit](https://github.com/awwsmm/awwsmm.com/commit/69e038a919e448251fa2211a9fcf3fda914812fe), because the point of the commit was to remove the `date` parameter from the front matter. Essentially, I was mixing up the `lastUpdated` date and the `published` date. One of these is the first element in the list (`dates[0]`) and one of them is the last element in the list (`dates[dates.length-1]`).

So like I said, there are four hard problems in computer science.

## On To The Next One

With that fixed, we're off to the races, right?

![Screenshot from awwsmm.com showing blog posts with incorrect dates](https://i.imgur.com/ME7yZQ9.png)

Oh... well, that's not right.

Those two posts were both committed on January 2 ([Hello, World!](https://github.com/awwsmm/awwsmm.com/commit/ec96618d38c71134f4a9ed14d6ae6d7a2b5c9e59) and [Git Hooks](https://github.com/awwsmm/awwsmm.com/commit/b2e504f52e4df0ddf77c662903e39b4aaf12f242)), not on January 6. So why did they both have the wrong date?

That's right, it's another bug... Or is it?

Find out in [the _thrilling_ final installation of this debugging mystery!](https://www.awwsmm.com/blog/whats-wrong-this-time-part-3)

What's Wrong This Time? Part II: Electric Bugaloo

**Part I: The Idea**

## _"I have an idea"_*, I said to myself.

"What if, instead of adding arbitrary dates to blog posts as I write them, I just got the dates from git?"

> [Git](https://git-scm.com/) is a [version control system](https://www.atlassian.com/git/tutorials/what-is-version-control) used by software developers which tracks changes to files, as well as creation and deletion of files. Every change, addition, or deletion is timestamped and has a [hash](https://en.wikipedia.org/wiki/Hash_function) which uniquely identifies those changes, like a fingerprint.

"[Each git commit](https://github.com/awwsmm/awwsmm.com/commits/master) has a date, so I should be able to programatically determine when a file in my [repo](https://www.gitkraken.com/learn/git/tutorials/what-is-a-git-repository) was added, or edited."

That was my idea for simplifying the metadata ("front matter") that I add to each of my blog post files on [my website](https://github.com/awwsmm/awwsmm.com). Before this adventure, that metadata looked [something like this](https://raw.githubusercontent.com/awwsmm/awwsmm.com/ec96618d38c71134f4a9ed14d6ae6d7a2b5c9e59/blog/hello-world.md):

```txt
---
title: 'Hello, World!'
description: 'Rewriting My Personal Website in Next.js and TypeScript'
date: '2022-01-01'
---
```

What I wanted to do was get rid of that arbitrary `date` field I was adding to every blog post, and just let git tell me when I published or updated a file. I could pull this information from the history of file changes that git keeps track of.

Sounds relatively simple, right?

Well, there were a few roadblocks to getting this up and running. Before we get into them, let's take a look at my initial, quick-and-dirty solution.

## First Attempt

### Git background

When working locally (on your personal computer), you can interact with git using the command line. One command, [`git log`](https://git-scm.com/docs/git-log), gives a list of all of the changes saved ("committed") to the project

```txt
$ git log --oneline -n 5
b6eea85 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #15 from awwsmm/development
bc222b3 (origin/development, development) automatically date blog posts using git history
88c4208 order blog posts on index page reverse chronologically
9368f24 new blog post about nominal, structural, and duck typing
6bbe307 Merge pull request #1 from awwsmm/development
```

The options I passed to `git log`, `--oneline` and `-n 5`, tell git that I only want one line of minimal information for each commit (just the hash and the first line of the commit message), and that I only want the five most recent commits. By default, git shows something like

```txt
$ git log
commit b6eea85256fe54f34013f17aca4e892aba0778dd (HEAD -> master, origin/master, origin/HEAD)
Merge: 88c4208 bc222b3
Author: Andrew <aww@awwsmm.com>
Date: Fri Jan 14 19:01:11 2022 +0000

 Merge pull request #15 from awwsmm/development

 automatically date blog posts using git history
```

...with the commit hash, the commit message, the date of the commit, the author, and so on. But lots of other options can be set for `git log`, for instance, specifying a hash will show only that commit and its ancestors (commits that came before it), and `--name-only` shows the names of all files modified in that commit

```txt
$ git log bc222b3 --name-only -n 1
commit bc222b37913ec2cdd7ff5f152dfaca06d33ec9f3 (origin/development, development)
Author: Andrew Watson <aww@awwsmm.com>
Date: Fri Jan 14 18:54:39 2022 +0000

 automatically date blog posts using git history

.gitignore
README.md
blog/cache.json
blog/git-commit-hooks.md
blog/hello-world.md
blog/nominal-vs-structural-vs-duck-typing.md
lib/blog/FrontMatterFactory.ts
lib/blog/SlugFactory.ts
package-lock.json
package.json
pages/blog/[slug].tsx
pages/index.tsx
```

You can also pass a file path to `git log` and it will only show you commits in which that file was modified

```txt
$ git log blog/hello-world.md
commit bc222b37913ec2cdd7ff5f152dfaca06d33ec9f3 (origin/development, development)
Author: Andrew Watson <aww@awwsmm.com>
Date: Fri Jan 14 18:54:39 2022 +0000

 automatically date blog posts using git history

commit ec96618d38c71134f4a9ed14d6ae6d7a2b5c9e59
Author: Andrew Watson <aww@awwsmm.com>
Date: Sun Jan 2 12:11:54 2022 +0000

 attempting to deploy to Vercel
```

That's all well and good, but I wanted to get this information when building my website, which is written in TypeScript. So I would need a TypeScript or a JavaScript library to be able to do this easily. That's where [`simple-git`](https://www.npmjs.com/package/simple-git) comes in.

### Git in TypeScript

`simple-git` is a JavaScript library which [allows you to run git commands](https://github.com/steveukx/git-js) from within TypeScript / JavaScript code.

For instance, instead of the above `git log blog/hello-world.md`, we can write something like

```ts
import simpleGit from 'simple-git'

const git = simpleGit()
const logResult = await git.log({ file: 'blog/hello-world.md' })
const commits = logResult.all
```

Then, we can loop over all commits and easily access each commit's data

```ts
commits.forEach(commit => { // example:
 console.log(commit.author_email) // aww@awwsmm.com
 console.log(commit.author_name) // Andrew Watson
 console.log(commit.body) //
 console.log(commit.date) // 2022-01-14T18:54:39+00:00
 console.log(commit.diff) // null
 console.log(commit.hash) // bc222b37913ec2cdd7ff5f152dfaca06d33ec9f3
 console.log(commit.message) // automatically date blog posts using git history
 console.log(commit.refs) // origin/development, development
})
```

So that should be it: when compiling the project, we should be able to use this library to get all log lines which refer to a specific file. The most recent one should be the `lastUpdated` date of the file, while the least recent one should be the `published` date of the file.

[I quickly coded this up](https://github.com/awwsmm/awwsmm.com/commit/69e038a919e448251fa2211a9fcf3fda914812fe), ran it locally, and everything looked fine. I thought that would be the end of the story.

But then I pushed it to production.

## First Problem

The first problem I hit was that all of my blog posts now had the same date (which at the time was the current date):

![Screenshot of awwsmm.com showing multiple blog posts with the same date](https://i.imgur.com/C0WGWdp.png)

Do you see any problems with [the code I wrote here?](https://github.com/awwsmm/awwsmm.com/commit/69e038a919e448251fa2211a9fcf3fda914812fe)

What's going on here?

Check out the answer in [the second part of this series](https://www.awwsmm.com/blog/whats-wrong-this-time-part-2).

---

> \* This is the traditional phrase for conjuring bugs in your programs. See also: _"I'll just do a little refactoring"_.

What's Wrong This Time? A Debugging Mystery in Three Parts

Nominal, Structural, and Duck Typing in Java, TypeScript, and JavaScript


## Nominal Typing

Most major programming languages with objects use [nominal typing](https://en.wikipedia.org/wiki/Nominal_type_system), where the _name_ (or [fully qualified class name, FQCN](https://en.wikipedia.org/wiki/Fully_qualified_name)) of an object determines whether or not it is equal to another object, or assignable to a variable of a particular type. [For example, in Java](https://www.mycompiler.io/view/BDKXhWg)

```java
class Dog {
  public String name;
  public Dog (String name) {
    this.name = name;
  }
}

class Person {
  public String name;
  public Person (String name) {
    this.name = name;
  }
}
```

If you create a `Dog` and a `Person` with the same `name`, [Java will tell you that they're not the same thing](https://www.mycompiler.io/view/JJSt7fK), even though both classes have the same _structure_ (a single `String` field named `name`) and the same internal state (`name` is `"Fido"`)

```java
Dog dog = new Dog("Fido");
Person person = new Person("Fido");

// System.out.println(dog == person); // error: incomparable types: Dog and Person
System.out.println(dog.equals(person)); // false
System.out.println(person.equals(dog)); // false
```

And [you cannot pass a `Dog` to a method which expects a `Person`](https://www.mycompiler.io/view/88L7lwy)

```java
class Greeter {
    public static void greet (Person person) {
        System.out.println("Hello, " + person.name + "!");
    }
}

// ...

Greeter.greet(person); // Hello, Fido!
// Greeter.greet(dog); // error: incompatible types: Dog cannot be converted to Person
```

## Structural Typing

In contrast, TypeScript allows for _structural typing_ in some scenarios, where the structure of the objects is all that matters (and class names are immaterial). If we had the following two classes in TypeScript, we will see -- similar to Java -- that [they are not equal when compared with `==`](https://www.typescriptlang.org/play?#code/MYGwhgzhAEAiD2BzaBvAUNaA7MBbApgFzQQAuATgJZaIbTDxZnkCuwp850AFDgcc2qIAlKjqZSAC0oQAdH3zQAvNjz4A3HQC+aHWlCQYABXzkIjMZgUCKQugyYU2HLrzU2qNUekwTpchWVVAk1MHT0HMmgAEyQgrHwAdzgkbgAiADFKWLThTUjSaAAHU3MseKToEzNGdKycvLR9RnMQfFkQVNjkJRUSmqw86AB6YegAMzAQCHwgA)

```ts
class Dog {
  name: string
  constructor (name: string) {
    this.name = name;
  }
}

class Person {
  name: string
  constructor (name: string) {
    this.name = name;
  }
}

const dog = new Dog("Fido");
const person = new Person("Fido");

console.log(dog == person); // false
```

But suppose [we write our `Greeter` class in TypeScript as](https://www.typescriptlang.org/play?#code/MYGwhgzhAEAiD2BzaBvAUNaA7MBbApgFzQQAuATgJZaIbTDxZnkCuwp850AFDgcc2qIAlKjqZSAC0oQAdH3zQAvNjz4A3HQC+aHWlCQYABXzkIjMZgUCKQugyYU2HLrzU2qNUekwTpchWVVAk1MHT0HMmgAEyQgrHwAdzgkbgAiADFKWLThTUjSaAAHU3MseKToEzNGdKycvLR9RnMQfFkQVNjkJRUSmqw86AB6YegAMzAQCHwmgyhoAHFyfHxSU0sSUjBSSmBoRBW1nkPV7YAjNuIUYKItz2Qtb3F6Fvg2jtSAAwAJfBBOgAaaAAEhQpzWYEu7QUWgAhF9GmFdE1lmdTLIIaRuP0ykNRtA-gD4MD6vA4Wg0WsMVjuN18WMiUDoGS4UA)

```ts
class Greeter {
  static greet (greetable: { name: string }) {
    console.log(`Hello, ${greetable.name}!`);
  }
}

Greeter.greet(person); // Hello, Fido!
Greeter.greet(dog); // Hello, Fido!
```

TypeScript simply checks that the object passed to `greet()` has a `name: string` field, because that's the type we specified for `greetable`: an object with a `name: string` field. It doesn't care what class `greetable` might be, [or if it has other fields and methods as well](https://www.typescriptlang.org/play?#code/MYGwhgzhAEAiD2BzaBvAUNaA7MBbApgFzQQAuATgJZaIbTDxZnkCuwp850AFDgcc2qIAlKjqZSAC0oQAdH3zQAvNjz4A3HQC+aHWlCQYABXzkIjMZgUCKQugyYU2HLrzU2qNUekwTpchWVVAk1MHT0HMmgAEyQgrHwAdzgkbgAiADFKWLThTUjSaAAHU3MseKToEzNGdKycvLR9RnMQfFkQVNjkJRUSmqw86AB6YegAMzAQCHwmgyhoAHFyfHxSU0sSUjBSSmBoRBW1nkPV7YAjNuIUYKItz2Qtb3F6Fvg2jtSAAwAJfBBOgAaaAAEhQpzWYEu7QUWgAhF9GmFdE1lmdTLIIaRuP0ykNRtA-gD4MD6vA4Wg0WsMVjuN18WMiUDoGSKfpwAsAEKUcjRTYMTrkDx2Kzue4i16OVjsTg8AWcYU0YHWcVeTZ+GSyeVcFTa0K+KSawIqBT68JzFqFc48vkmyrc3npFbRNLAtKc+AATxdjSp63ImKO2OtvIZhP+zI93rhQA)

```ts
class Bird {
  color: string
  name: string
  constructor (color: string, name: string) {
    this.color = color;
    this.name = name;
  }
}

const bird = new Bird("red", "Boyd");
Greeter.greet(bird); // Hello, Boyd!
```

## Duck Typing

In JavaScript, we might [rewrite the above TypeScript classes like](https://jsfiddle.net/y5vts2Lk/)

```js
class Dog {
  constructor (name) {
    this.name = name;
  }
}

class Person {
  constructor (name) {
    this.name = name;
  }
}

class Bird {
  constructor (color, name) {
    this.color = color;
    this.name = name;
  }
}

const dog = new Dog("Fido");
const person = new Person("Fido");
const bird = new Bird("red", "Boyd");
```

But, as JavaScript doesn't specify types using the `:` as TypeScript does, we also have to [rewrite our `Greeter` slightly](https://jsfiddle.net/y5vts2Lk/1/)

```js
class Greeter {
  static greet (greetable) {
    console.log("Hello, " + greetable.name + "!");
  }
}

Greeter.greet(person); // Hello, Fido!
Greeter.greet(dog); // Hello, Fido!
Greeter.greet(bird); // Hello, Boyd!
```

In this case, there are no type or structural constraints at all on `greetable`. JavaScript is using [duck typing](https://en.wikipedia.org/wiki/Duck_typing) here, where field and method accesses are only checked at runtime (and not at compile time, because JavaScript isn't compiled). If a `greetable` has all of the required fields, no errors will be thrown.

However, [if a field is missing...](https://jsfiddle.net/bjgz7cuo/)

```js
class Complimenter {
  static compliment (target) {
    console.log("Hello, " + target.name + "!");
    console.log("What a nice shade of " + target.color + " you are!");
  }
}

Complimenter.compliment(person); // Hello, Fido! What a nice shade of undefined you are!
Complimenter.compliment(dog); // Hello, Fido! What a nice shade of undefined you are!
Complimenter.compliment(bird); // Hello, Boyd! What a nice shade of red you are!
```

...we can get `undefined` results.

> The name "duck typing" comes from the phrase ["If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck"](https://en.wikipedia.org/wiki/Duck_test). When using duck typing, we run the program as long as we can until we hit a runtime error -- until the "duck" no longer looks like a duck.

What's the Difference Between Nominal, Structural, and Duck Typing?

Rewriting My Personal Website in Next.js and TypeScript


New Year's Day is as good a time as any to roll out a new blog, I suppose.

As one of my New Year's Resolutions this year, I'm going to revamp my website, and I'm going to do it in public.

I've spent the last few weeks working with Next.js for the first time (and brushing up on my TypeScript) in order to start hosting my technical blog on my own website. (In addition to mirroring the posts [on Dev.to](https://dev.to/awwsmm), where I've been writing for a few years now.)

In addition to the ol' blog, I've also created **/project** pages for a few ongoing things I've got in the works (including this very website). The project pages synthesize groups of commits from GitHub with manual "update" log posts to give an idea as to the current status of each of my ongoing projects. I think this is a bit more dynamic than a static "portfolio" page, and it reflects a tenet of my own personal philosophy that [_un ouvrage n’est jamais achevé... mais abandonné_](https://quoteinvestigator.com/2019/03/01/abandon/).

Follow along here, [on Dev.to](https://dev.to/awwsmm), or [on Twitter](https://twitter.com/awwsmm_dot_com) for continuing updates.

Thanks for reading!

Hello, World!

Writing Git Commit Hooks in Bash, Java, and Scala

## Writing a Git Hook in Bash

[Git Hooks](https://www.atlassian.com/git/tutorials/git-hooks) can be a very useful addition to a developer's workflow.

For instance, as I'm working on [rewriting my new website using Next.js](https://www.awwsmm.com/blog/hello-world), I've set up [eslint](https://eslint.org) and have pretty strict linting rules when running Next builds.

Unfortunately, Git doesn't follow these rules by default. So it's easy to forget to run the linter, run a `git push`, and end up with a failing production build on [Vercel](https://vercel.com).

To avoid this, I've set up a simple Git Hook which runs `eslint` before every `git push`, and fails the `push` if the linter returns any `error`s:

```sh
# .git/hooks/pre-push

# pipe output (with colors) to .temp file
RESULT=$(script -q .temp npm run lint)

# if .temp file contains "error", fail
if grep --quiet error .temp; then
 printf "\n !! Will not push -- found linting errors:\n"
 sed '1,5d' .temp | cat
 rm .temp
 exit 1
else
 rm .temp
 exit 0
fi
```

The above script runs `npm run lint`, captures the output ([with text colors](https://stackoverflow.com/a/3515296/2925434)) to a `.temp` file, and then `grep`s for `error` in that file. If it finds any instances of the word "error", it won't `push`, but will instead print the output of the `npm run lint` command:

```sh
$ git push

 !! Will not push -- found linting errors:

/Users/andrew/Git/awwsmm.com/components/CommitGroupComponent.tsx
 2:1 error Imports should be sorted alphabetically sort-imports
 24:4 error Missing semicolon semi

✖ 2 problems (2 errors, 0 warnings)
 1 error and 0 warnings potentially fixable with the `--fix` option.

error: failed to push some refs to 'github.com:awwsmm/awwsmm.com.git'
```

This prevents useless builds on Vercel which will fail because of linting errors.

**Note:** make sure your `.git/hooks/<file>` is executable (`chmod +x .git/hooks/<file>`), or it will not prevent the particular action from occurring. It's probably best to play around with a `pre-commit` hook rather than a `pre-push` one, when testing.

## Writing a Git Hook in Java

We can do a similar thing using Java instead of bash scripting. For this example, we'll look at writing a `pre-commit` hook in Java. Instead of writing Java code to the `pre-commit` file directly, we'll add a bit of indirection.

First, create a file called `PreCommit.java` in the `.git/hooks/` directory and add some pre-commit logic to it. In this case, we're just going to fail every commit:

```java
public class PreCommit {
 public static void main (String[] args) {
 System.out.println("Java says: no committing please.");
 System.exit(1);
 }
}
```

The `System.exit(1)` gives the same result as `exit 1` in the Bash script -- the commit will fail to complete.

Next, we'll make the `.git/hooks/pre-commit` file executable with

```sh
$ chmod +x .git/hooks/pre-commit
```

...and modify its contents to read:

```sh
#!/bin/sh

# get the path to this script file
DIR=$(dirname "$0")

# Java >= 11
exec java $DIR/PreCommit.java "$@"
```

> If you're using Java 11+, you can also write this [all as a single file](https://stackoverflow.com/a/61899256/2925434), though I personally don't recommend this method as it combines Bash and Java syntax in a single file, which can be difficult for IDEs to handle in terms of syntax highlighting, etc.

With this in place, we'll now see a message like

```sh
$ git commit -m "update gitignore, more hook testing"
Java says: no committing please.
```

Note that in Java versions &lt; 11, we need to compile this script first with `javac`, so `pre-commit` should instead look something like

```sh
#!/bin/sh
DIR=$(dirname "$0")

# Java < 11
javac $DIR/PreCommit.java "$@"
exec java -classpath $DIR PreCommit
!#
```

## Writing a Git Hook in Scala

The process for [writing a Git Hook in Scala](https://alvinalexander.com/scala/scala-shell-script-example-exec-syntax/) is very similar to writing one in Java. First, define a `pre-commit.scala` file with the following contents

```scala
object Hook {
 def main (args: Array[String]): Unit = {
 System.out.println("Scala says: no committing please.");
 System.exit(1);
 }
}
```

(Note that, in Scala, the name of the class doesn't have to match the name of the file.)

Then, modify the contents of `pre-commit` to read

```sh
#!/bin/sh
DIR=$(dirname "$0")
exec scala $DIR/pre-commit.scala "$@"
!#
```

Finally, try to commit:

```sh
$ git commit -m "update gitignore, more hook testing"
Scala says: no committing please.
```

That's all there is to it.

As it takes a while for the JVM to boot up when running Git Hooks in Java or Scala, it's probably not extremely practical to do so... unless you have a lot of logic in your hook which is easier to read in Java / Scala than in a Bash script.

But anyway, it's good to know that it can be done.

Eliminate Unnecessary Builds With Git Hooks

Quickly and Easily Publish Your Scala Code for Use in Other Projects

## Intro

Say what you will about JavaScript, but [npm](https://www.npmjs.com/) has made it incredibly easy for anyone to publish code. In just a few clicks, you can share a library to make others' lives easier, or publish a little snippet like [`left-pad`](https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/) to cause endless headaches.

Java has long had the [Maven repository](https://mvnrepository.com/) (and accompanying build tool of the same name), which serves a similar purpose: sharing packaged code.

But it can be intimidating for a newbie to publish to Maven -- that's where Sonatype comes in.

Sonatype, Inc.'s [Central Repository](https://central.sonatype.org/) (a complement to Apache's Maven repository) makes it super easy to share code. With just a few lines and a few manual steps, you can publish your package to Maven, and others can easily import it and use it.

sbt's [Using Sonatype](https://www.scala-sbt.org/release/docs/Using-Sonatype.html) guide and Sonatype's [Deployment Guide](https://central.sonatype.org/publish/publish-guide/) cover probably 95% of what you need to know for this process, but there are a few _gotchas_ that I'll highlight below using the ❗ exclamation point emoji.

> Note: this guide assumes that your source code is hosted in a GitHub or GitLab repository.

## Steps

### 1. Create a Sonatype Jira issue

Sonatype syncs with Maven, and so to publish to Maven, you can publish via Sonatype. To do so, you'll need a Sonatype Jira account to create an issue. Do that by [clicking here to create an account](https://issues.sonatype.org/secure/Signup!default.jspa) and then [clicking here to create a Jira issue](https://issues.sonatype.org/secure/CreateIssue.jspa?issuetype=21&pid=10134).

As an example, [here is the ticket I opened](https://issues.sonatype.org/browse/OSSRH-71079) to publish a project of mine to Maven. If you want to publish multiple packages to Maven using Sonatype, you'll need to create multiple Jira tickets, but you only need the one user account.

The fields in the Jira are easy to fill out:

#### Summary

This should include the name of the project. In my case, I'm publishing a small project hosted at https://github.com/awwsmm/zepto, so I've included the name "zepto" in the Jira summary.

#### Group Id

If your project is hosted on GitHub, this should be something like `io.github.awwsmm`, or similar for your GitHub username. See [this guide](https://central.sonatype.org/publish/requirements/coordinates/) for more information on package coordinates.

> ❗ **GOTCHA:** this should _not_ be something like `com.github.awwsmm`. `com.github` coordinates are no longer supported. It _can_ be your personal website, like `com.awwsmm`, but [a bot will comment on your Jira asking you to add a DNS record to prove that you own that website, like here on another one of my Jiras](https://issues.sonatype.org/browse/OSSRH-95101).

#### Project URL

This is where users can go on the web to find information about your project. For me, that's `https://github.com/awwsmm/zepto`, but it could be a `github.io` page or a page on your personal website, or whatever.

#### SCM url

This is the URL that your source code is hosted at. For me, that's `https://github.com/awwsmm/zepto.git`.

### 2. Generate and distribute a PGP key

#### On macOS

On macOS, I recommend that you use https://gpgtools.org/ to create and distribute your PGP key. Simply download the installer and follow the instructions there.

To run `sbt publishSigned` later in this guide, you will also need `gpg2` and `pinentry` installed. You can get those on macOS via `brew` with

```bash
$ brew install gpg2
$ brew install pinentry-mac
```

Note that you need `gpg2` here, not `gpg`. You should [check if you have both installed and `brew uninstall gpg` if it exists](https://stackoverflow.com/a/45932796/2925434)

```bash
$ which gpg gpg2
```

Once you've done that, check that `gpg-agent` is correctly configured, and restart it by `--kill`ing it, per [these instructions](https://superuser.com/a/1628810/728488).

#### On Linux

We'll use a [PGP key](https://en.wikipedia.org/wiki/Pretty_Good_Privacy) to sign this package, so users can be confident that you are the person who created it. To do this, you can mostly follow the instructions from [scala-sbt.org](https://www.scala-sbt.org/release/docs/Using-Sonatype.html)...

Install [GnuPG](https://www.gnupg.org/download/) (on macOS, you can do this via `brew install gnupg`) and verify that it's installed by checking the version

```bash
$ gpg --version
gpg (GnuPG) 2.3.1
...
```

Next, generate a key...

```bash
$ gpg --gen-key
```

...list your keys...

```bash
$ gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2023-07-11
/Users/andrew.watson/.gnupg/pubring.kbx
---------------------------------------
pub ed25519 2021-07-11 [SC] [expires: 2023-07-11]
 1234517530FB96F147C6A146A326F592D39AAAAA
uid [ultimate] Andrew Watson <my@email.com>
sub cv25519 2021-07-11 [E] [expires: 2023-07-11]
```

...and distribute the key to some GPG keyserver, like

```bash
$ gpg --keyserver keyserver.ubuntu.com --send-keys 
1234517530FB96F147C6A146A326F592D39AAAAA
gpg: sending key A326F592D39AAAAA to hkp://keyserver.ubuntu.com
```

> ❗ **GOTCHA:** If you get an error like `keyserver send failed: No name` from the command above, it means the keyserver you're trying to send to is no longer active. [You can instead use any other major GPG keyserver](https://stackoverflow.com/a/68132500/2925434), like the Ubuntu one I used above.

[Leonard notes here](https://leonard.io/blog/2017/01/an-in-depth-guide-to-deploying-to-maven-central/) that synchronising these keys around the various keyservers can take some time, but for me it worked almost instantly.

Once you publish your key, you can check that it's visible on the public keyservers by searching for it here: https://keys.openpgp.org/.

### 3. Enable `sbt-pgp`

[As outlined here](https://www.scala-sbt.org/release/docs/Using-Sonatype.html), you'll next want to add the `sbt-pgp` plugin to your project. You can add it to your project's `plugin.sbt` file, or you can add it to `~/.sbt/1.0/plugins/gpg.sbt` to enable it for all sbt projects. The line you should add is

```scala
addSbtPlugin("com.github.sbt" % "sbt-pgp" % "2.2.1")
```

The latest version of this plugin can be found [at the GitHub page for the project](https://github.com/sbt/sbt-pgp).

### 4. Add your Sonatype credentials to your project

Next, add your credentials to your project by following the instructions in [Step 3](https://www.scala-sbt.org/release/docs/Using-Sonatype.html) here. I'll reproduce them to be clear...

Create a file at

```bash
$HOME/.sbt/1.0/sonatype.sbt
```

And add the following line to it

```scala
credentials += Credentials(Path.userHome / ".sbt" / "sonatype_credentials")
```

Then, create the credentials file at

```bash
~/.sbt/sonatype_credentials
```

Add the following information to that file

```
realm=Sonatype Nexus Repository Manager
host=s01.oss.sonatype.org
user=<your username>
password=<your password>
```

The `user` and `password` here are the same username and password you used when creating your account on Sonatype's Jira server in Step 1. You don't need to worry about putting anything in quotes here -- spaces are fine. But...

> ❗ **GOTCHA:** ...be careful! Some older guides say to set `host=oss.sonatype.org`, but this has recently changed to `host=s01.oss.sonatype.org`. More information about this change [can be found here](https://central.sonatype.org/publish/release/).

### 5. Set up your `publish.sbt`

The penultimate step in this process is to configure your project for publishing, by adding a bunch of information to either your project's `build.sbt` (or, as I recommend) `publish.sbt` file.

Replace `username` with your GitHub / GitLab username below, `project` with the name of the project (in my case, these are `awwsmm` and `zepto`, respectively), and all the other obvious fill-in bits, and paste it into a `publish.sbt` file at the root of your project

```scala
ThisBuild / organization := "io.github.username"
ThisBuild / organizationName := "username"
ThisBuild / organizationHomepage := Some(url("https://www.yourwebsite.com"))

ThisBuild / scmInfo := Some(
 ScmInfo(
 url("https://github.com/username/project"),
 "scm:git@github.username/project.git"
 )
)

ThisBuild / developers := List(
 Developer(
 id = "username",
 name = "Your Name Goes Here",
 email = "youremail@address.com",
 url = url("https://www.yourwebsite.com")
 )
)

ThisBuild / description := "Describe your project here..."
ThisBuild / licenses := List("The Unlicense" -> new URL("https://unlicense.org/"))
ThisBuild / homepage := Some(url("https://github.com/username/project"))

// Remove all additional repository other than Maven Central from POM
ThisBuild / pomIncludeRepository := { _ => false }

ThisBuild / publishTo := {
 val nexus = "https://s01.oss.sonatype.org/"
 if (isSnapshot.value) Some("snapshots" at nexus + "content/repositories/snapshots")
 else Some("releases" at nexus + "service/local/staging/deploy/maven2")
}

ThisBuild / publishMavenStyle := true

ThisBuild / versionScheme := Some("early-semver")
```

You can set the `licenses` to whatever you feel is appropriate. Here is GitHub's guide to [choosing a license for your repo](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/licensing-a-repository) -- though be aware that they [probably ignore it](https://news.ycombinator.com/item?id=27736650) anyway.

> ❗ **GOTCHA:** Notice that we've also used `s01.oss.sonatype.org` above!

### 6. Publish!

Once you've got all of that set up, you should be able to publish your project from the `sbt` shell with the `publishSigned` command

```scala
sbt> publishSigned
```

On macOS, this should open up the GPGTools UI, where you can enter the password you associated with your PGP key. The output will then look something like

```bash
[info] Wrote /local/path/to/zepto_2.13-0.4.0.pom
[info] 	published zepto_2.13 to https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/io/github/awwsmm/zepto_2.13/0.4.0/zepto_2.13-0.4.0-sources.jar
[info] 	published zepto_2.13 to https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/io/github/awwsmm/zepto_2.13/0.4.0/zepto_2.13-0.4.0-javadoc.jar
[info] 	published zepto_2.13 to https://s01.oss.sonatype.org/service/local/staging/deploy/maven2/io/github/awwsmm/zepto_2.13/0.4.0/zepto_2.13-0.4.0-javadoc.jar.asc
...
```

You can use your Sonatype username and password to log in to https://s01.oss.sonatype.org/ where you should now see your (pre-release) project!

Note that if you try to click on one of the `.jar` links above, you may get an error like

```
Staging of Repository within profile ID='...' is not yet started!
```

You can [follow along with Sonatype's guide here](https://central.sonatype.org/publish/release/) to release the artifacts to the public.

![Screenshot of my pre-release project listed in my account on sonatype.org](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ue4pc8kh7bmewm41qndl.png)

Select the project, click `Close` from the menu, and then `Release`. You should see your project [on Maven](https://repo1.maven.org/maven2/io/github/awwsmm/zepto_2.13/0.4.0/) almost immediately, though it will take a few hours to show up [in the search engine](https://search.maven.org/artifact/io.github.awwsmm/zepto_2.13).

...and that's it!

You should now be able to import your published library into other projects by adding

```
"io.github.awwsmm" %% "zepto" % "0.4.0"
```

(or whatever is appropriate) to the `libraryDependencies` in the `build.sbt` of another project.

Blog