git / monorepo
At a devtool startup, my team wrote and maintained multiple programs. We worked incrementally, used technologies suited to their tasks and our preferences, and wrote custom tooling when needed.
New repo
I initialized a monorepo on GitHub, org/repo:
.
└── README.md
I wrote a README.md:
Add to your shell profile:
# Set environment variable to monorepo path
```sh
export ORG="$HOME/org"
```
# Prepend monorepo scripts
```sh
export PATH="$ORG/bin:$PATH"
```
Clone:
```sh
git clone https://github.com/org/repo.git $ORG
cd $ORG
```
The $ORG environment variable is used throughout the codebase.
Design
The architecture:
- a "Dashboard" web interface for customers (HTML, CSS, JavaScript)
- SDKs for customers (Go, JavaScript, Ruby)
- an HTTP server (Go)
- a Postgres database backing the HTTP API
.
├── ...
├── cmd
│ └── serverd
│ └── main.go
├── dashboard
├── sdk
│ ├── go
│ ├── js
│ └── ruby
└── server
└── http_handler.go
Commit messages
I used commit message conventions with a subject line prefix:
$ git log
dashboard: redirect to new project form after sign up
sdk/js: enable Keep-Alive in HTTP agent
server: document access to prod db
The prefix refers to the module that is changing. It often matches a filesystem directory. Sometimes the change will touch files across modules, but the prefix should be "where the action is." Try to keep the subject line under 50 characters and err on the side of brevity.
Dependencies
To aid local development and help onboard teammates,
I added a setup.sh script for each module.
In turn, each setup.sh script might invoke more general $ORG/bin/setup-* scripts.
Since everyone has $ORG/bin on their $PATH,
scripts in bin/ are available in everyone's shells
without a directory prefix.
.
├── ...
├── bin
│ ├── setup-go
│ ├── setup-js
│ ├── setup-postgres
│ └── setup-ruby
├── dashboard
│ └── setup.sh
├── sdk
│ ├── go
│ │ └── setup.sh
│ ├── js
│ │ └── setup.sh
│ └── ruby
│ └── setup.sh
└── server
└── setup.sh
Formatting
Our Go code was already formatted with gofmt, but our JavaScript and Ruby code was less consistent. I added config files for Prettier and Rubocop at the top of the file hierarchy:
.
├── ...
├── .rubocop.yml
└── .prettierrc.yml
CI
To move quickly without breaking things, we wrote automated tests and ran them continuously as we integrated changes.
Hosted CI services were too slow to start,
so I wrote our own CI service, cibot:
.
├── ...
├── cibot
│ ├── Checkfile
│ ├── farmer
│ │ └── main.go
│ └── worker
│ └── main.go
├── cmd
│ └── cibot
│ └── main.go
├── dashboard
│ └── Checkfile
├── sdk
│ ├── go
│ │ └── Checkfile
│ ├── js
│ │ └── Checkfile
│ └── ruby
│ └── Checkfile
└── server
└── Checkfile
cibot runs checks within seconds of opening a pull request.
See cmd / cibot for the architecture,
Checkfile format, and job scheduling.
Testing across services
To make it convenient to test across service boundaries,
I wrote a with-serverd script that:
- installs the
serverdbinary - migrates the database
- creates a team and credential
- runs
serverd servewithout blocking programs passed as arguments towith-serverd
I placed this script in $ORG/bin to make it available on our $PATH.
This script can be used in Checkfiles:
$ cat $ORG/dashboard/Checkfile
tests: cd $ORG/dashboard && with-serverd ./test.sh
$ cat $ORG/sdk/go/Checkfile
tests: cd $ORG/sdk && with-serverd go test -cover ./...
Tests that depend on with-serverd
can avoid mocking on HTTP boundaries,
making actual requests to the backing service
and generating observable logs.
To broadly cover the product's surface area,
I moved the Go SDK's Checkfile
to the top of the file hierarchy
so its test jobs run on all pull requests:
.
├── ...
├── Checkfile
├── bin
│ └── with-serverd
└── sdk
└── go
Backwards compatibility
As customers used the software, we understood their needs better and began to design v2.
To do this well, we added new interfaces and deprecated old ones in the v1.x series of the SDKs. The server continued to support the v1.x series for a well-communicated time period past the release of v2.0 (such as 90 days).
As we released minor versions v1.1 and v1.2, patch versions v1.2.1 and v1.2.2, and eventually v2, we wanted to carefully ensure backwards compatibility.
To do this,
I wrote with-go-sdk, with-js-sdk, and with-ruby-sdk scripts:
.
├── ...
└── bin
├── with-go-sdk
├── with-js-sdk
└── with-ruby-sdk
These scripts run processes using a given version of our SDKs:
with-go-sdk [version] [command]
with-js-sdk [version] [command]
with-ruby-sdk [version] [command]
The version can be a released version to a registry such as NPM or Rubygems
or a special value monorepo to use the current $ORG source code.
For example:
with-ruby-sdk 1.0 ruby -e "require 'org-sdk'; puts OrgSDK::VERSION"
I updated the Checkfile at the top of the file hierarchy
to test supported releases
and the current source code (pre-release)
on every pull request:
$ cat $ORG/Checkfile
gohead: with-serverd with-go-sdk monorepo go test -cover ./...
gov1: with-serverd with-go-sdk 1.5 go test -cover ./...
gov2: with-serverd with-go-sdk 2 go test -cover ./...
# ... etc.
rubyv2: with-serverd with-ruby-sdk 2 $ORG/sdk/ruby/test.sh
The SDK scripts are composable with the with-serverd script.
In addition to running on CI, they are useful for quickly testing bug reports from customers on a particular version, and hopefully providing a great customer experience for them.
Deploy
We deployed serverd and cibot to
a host like Render.
Each program is a Go binary with its own web service
or background worker, configured with environment variables.
Conclusion
Working in a monorepo can encourage a feeling of "tight integration", where service boundaries are well-defined and less likely to be mocked out. Some tasks are a particular pleasure, such as searching across projects for callsites of a function or RPC.
For other tasks, it can help to write custom tools. Writing those custom tools offers an opportunity to design an ideal experience for the engineering team.
The final directory structure looks like this:
.
├── .prettierrc.yml
├── .rubocop.yml
├── .tool-versions
├── README.md
├── Checkfile
├── bin
│ ├── setup-go
│ ├── setup-js
│ ├── setup-postgres
│ ├── setup-ruby
│ ├── with-go-sdk
│ ├── with-js-sdk
│ ├── with-ruby-sdk
│ └── with-serverd
├── cibot
│ ├── Checkfile
│ ├── farmer
│ │ └── main.go
│ ├── setup.sh
│ └── worker
│ └── main.go
├── cmd
│ ├── cibot
│ │ └── main.go
│ └── serverd
│ └── main.go
├── dashboard
│ ├── Checkfile
│ └── setup.sh
├── sdk
│ ├── go
│ │ └── setup.sh
│ ├── js
│ │ ├── Checkfile
│ │ └── setup.sh
│ └── ruby
│ ├── Checkfile
│ └── setup.sh
└── server
├── Checkfile
├── http_handler.go
└── setup.sh