Client/Server code sharing in Typescript monorepos
--
With the popularization of node.js since 10+ years ago, it’s becoming more and more usual to find the same programming language in both sides of web applications. Using node.js in the server side allows for re-using a good amount of logic but, how can we effectively share code between client and server in Typescript monorepos? Let’s have a look at different alternatives.
Most modern web applications are characterized by the following two traits when it comes to code repetition:
- The same validation logic is executed on both client and server side. On the client side, to detect validation errors without having to communicate with the server, and on the server side, to protect ourselves from API faulty calls.
- Very similar data models are used in both client and server side. Since we deal with the same application data on both ends, it’s only logic to define symmetric data models.
In node.js monorepos we can easily extract the duplicated code away and require it using relative paths. But the extraction comes with some challenges when Typescript is in the mix. Let’s use a sample Typescript web app to better illustrate those challenges and how to resolve them effectively. We can call it Weather Now.
On one hand we have a React UI (i.e. weather-client
) that fetches weather data for a given city from a web API and displays it in a simple layout. On the other hand we have an Express web API (i.e. weather-server
) with a single endpoint that returns weather data (i.e. /api/weather
) and serves the static React UI files in the root url.
And, to fit the purpose of this article, the monorepo sure has some duplicated code: validateCityName
, the function which validates that a city name has been provided, and the Validation
, Weather
and WeatherIcons
types.
Now that we have a concrete example, let’s proceed to extract that duplicated code into a common folder (e.g. weather-common
) and require it through relative path imports (see base-code-extraction branch).
Here is where the challenges start: Typescript does compile successfully but node fails to start the server.
$ npm run start:server
> start:server
> cd weather-server && npm run start
> start
> node distribution/index.js
internal/modules/cjs/loader.js:905
throw err;
^
Error: Cannot find module '/.../weather-monorepo/weather-server/distribution/index.js'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
at Function.Module._load (internal/modules/cjs/loader.js:746:27)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)
at internal/main/run_main_module.js:17:47 {
code: 'MODULE_NOT_FOUND',
requireStack: []
}
In Javascript monorepos, where the code files maintain their relative location at runtime, this type of code extraction works straight away. In Typescript repositories however, where the compiled files generally change their location (i.e. the distribution folder), further configuration is necessary to perform the code extraction, since we are now importing code from outside the Typescript rootDir .
It is possible to keep the compiled files in the same relative location as the Typescript files (i.e. by removing the outDir
property in tsconfig.json
), but we generally want to compile them to a separate location, so we only include the distribution files when deploying the code to production.
Importing code from outside the scope of a Typescript project forces the Typescript compiler to replicate the file system hierarchy inside the distribution folder, so it can guarantee that the referenced files will be compiled and available from within the distribution folder (note that Typescript never changes the relative paths in compiled files). In Weather Now this causes the index.js
server file’s location to change, causing the start
npm script to fail.
.
···
├─ weather-common
| ···
| utils.ts
└─ weather-server
├─ distribution
| ├─ weather-common
| | ···
| | utils.js
| └─ weather-server
| ···
| index.js
└─ source
···
index.ts
···
package.json
tsconfig.server.json
···
package.json
tsconfig.base.json
Fortunately there are several ways of fixing this problem 👍 Let’s have a look at different solutions.
1. Relative paths fixes
The quick and dirty approach to get things working again. We just need to adapt the code to support the changes in the distribution folder. For Weather Now this means changing the references to compiled files in package.json
(i.e. main
and start
npm script) as well as the paths to certain resources in the code files (see 1-path-fixes branch).
Changing those paths might not look like a major deal but it adds unnecessary complexity to our code. Moreover, it forces us to introduce additional logic to, for example, run the code without compiling (e.g. using nodemon
or ts-node
). For Weather Now this means introducing an environment variable and setting the express.static
path dynamically.
It works, but we can do better. Why changing the code to match the distribution file system hierarchy when we can change the distribution file system hierarchy to match the code?
2. common npm project
By turning the common folder into an npm project itself, Typescript will expect the compiled files to exist in the referenced paths and it will stop replicating the file system hierarchy inside the distribution folder 💪 Let’s go ahead and add both package.json
and tsconfig.json
files to the common folder, move the code files into a source
subfolder and define an npm script to compile the Typescript code (see 2-npm-project branch).
Note that as a result of extracting the code to a separate project:
- We are introducing a build dependency. We will always need to compile the common project separately before compiling the projects that depend on it (i.e. update the
build
npm script on the rootpackage.json
).
- Typescript looses access to the types definitions. We will need to generate type declaration files for the common project (i.e. setting the
declaration
property to true intsconfig.json
) and let Typescript know where to locate those files (i.e. setting thetypes
property in the commonpackage.json
).
Generating type declaration files allows Typescript to compile but will as well cause the IDE symbols navigation to go the declaration files instead of the source code. Unfortunately we can’t fix this issue when using relative path imports.
- Hot reloading will no longer detect changes happening outside the project
source
folder. We will need to compile the common project in watch mode as well as tweaking nodemon (throughnodemon.json
) to watch for changes in the common project’s source folder.
Now the compiled common code will remain inside the common project, and the distribution folders of the other projects will remain unchanged 🎉 Much better than 1. Relative paths fixes, but still can be improved.
.
···
├─ weather-common
| ├─ distribution
| | ···
| | utils.js
| └─ source
| ···
| utils.ts
| ···
| package.json
| tsconfig.common.json
└─ weather-server
├─ distribution
| ...
| index.js
└─ source
···
index.ts
···
package.json
tsconfig.server.json
···
package.json
tsconfig.base.json
3. common npm project + npm local dependencies
After turning the common folder into a separate npm project we can use npm local dependencies (natively supported since npm 2.0) to install the common npm project as a dependency of the other npm projects, get rid of the relative paths in the cross-project import statements* and let npm natively resolve the dependency instead.
All we need to do is install the common project as a local dependency in all the projects that are importing code from it in the following fashion. This will register the local dependency in both package.json
and package-lock.json
in the dependent projects.
cd weather-client && npm instal --save ../weather-common
cd ../weather-server && npm install --save ../weather-common
Under the hood npm creates symbolic links to the local dependency folder inside the dependent projects’ node_modules
folder:
We can now go ahead and replace all the relative path imports with regular npm dependency imports (see 3-local-dependency branch, or 3-local-dependency-namespace if you prefer to namespace your packages).
Another advantage from this alternative is that it allows us to restore the IDE source code navigation we lost when extracting the duplicated code to a shared folder. We can use the Typescript paths property to tell the IDE where to look for type definitions. For Weather Now this is:
{
"compilerOptions": {
"paths": {
"weather-common": ["./weather-common/source"]
}
},
"extends": "./tsconfig.base.json"
}
Note that using Typescript paths to resolve the relative path imports without registering the local npm dependencies will result in runtime errors since, as previously mentioned, Typescript does not modify the imports path on compilation.
4. common npm project + npm workspaces
After turning the common folder into a separate npm project, we can use npm workspaces (introduced in npm 7.0) to simplify the management of the different projects in the monorepo. The main advantage of using npm workspaces is being able to run, directly from the root folder, npm scripts defined in the package.json
of any project in the monorepo.
In order to enable workspaces in an existing npm monorepo we need to move the different projects into a specific folder (e.g. projects
) and specify that folder through the workspaces
property in the root package.json
. npm workspaces come with a set of dependencies so we will need to re-install npm dependencies after updating the package.json
, which will update package-lock.json
as well.
Note that after the re-install, apart from new npm workspaces specific dependencies (e.g. @nodelib/fs.scandir
), the package-lock.json
file will also contain each of the existing projects in the workspace folder as dependencies. Additionally, npm will create symbolic links in the root node_modules
folder.
We can now install the common project as a local dependency in all the projects that are importing code from it, this time using workspaces (i.e. the --workspace
, or -w
, argument).
npm install --save ./projects/weather-common -w weather-client
npm install --save ./projects/weather-common -w weather-server
This will register the local dependency in the root package.json
and the dependent project’s package.json
. It should update the dependent projects’s package-lock.json
as well, but there is currently an npm known issue which causes the following install command to fail, leaving the file unchanged.
$ npm install --save ./projects/weather-common -w weather-client
npm ERR! Cannot set properties of null (setting 'dev')
npm ERR! A complete log of this run can be found in:
npm ERR /.../.npm/_logs/2022-05-30T05_57_51_827Z-debug.log
Fortunately there is a simple workaround to that issue: re-installing the dependencies of the dependent project (e.g. running an npm install
in the project folder). We can after replace all the relative paths in the cross-project import statements* with regular npm dependency imports (see 4-npm-workspaces branch, or 4-npm-workspaces-namespace if you prefer to namespace your packages).
Just like when using npm local dependencies, we can as well restore the IDE source code navigation we lost when extracting the duplicated code to a shared folder by using Typescript paths property to tell the IDE where to look for type definitions. For Weather Now this is:
{
"compilerOptions": {
"paths": {
"weather-common": ["./projects/weather-common/source"]
}
},
"extends": "./tsconfig.base.json"
}
One more advantage of having npm workspaces in place is that we can simplify, or even remove, the “shortcut” npm scripts. “shortcut” scripts are a common practice in node.js monorepos: since npm doesn’t detect scripts defined in “nested” folders, the only way to run “nested” scripts from the root folder is to define additional scripts which change the folder and then run the corresponding npm script:
Since we have landed npm workspaces to our monorepo we can replace the cd
instructions in our root package.json
npm scripts with the corresponding -w
argument, slightly speeding up the scripts execution time:
Note that certain npm scripts must remain intact. We need, for example, to build the projects in a certain order (i.e. weather-common
before weather-server
), so using the --ws
(all workspaces) for the build
script could run into compilation errors. Or, trying to use npm install -ws
in the postinstall script would result into an endless install loop (the workspaces arguments only apply on install when adding/updating a dependency), so we need to keep the original postinstall script.
* What’s the fuss with cross-project relative paths?
Relative paths in cross-project import statements are not necessarily a problem. The value of replacing them is in making the dependent projects agnostic of the common code file system hierarchy. This way, if we were to make changes in the common npm project, the dependent projects would remain unchanged (as long as the common code API is kept the same).
Not that, while both 3. npm local dependencies and 4. npm workspaces allow for relative paths removal, they introduce a potential minor drawback: not being able to install public packages with the same name as the local dependencies. For example, naming a project common would prevent us from installing the common package from the npm registry.
A convenient way of working around these conflicts is to namespace the packages. In fact, you might have noticed that some popular packages use the @organization/package
format on their names: @types/node
, @react-native-community/slider
,@angular/core
, etc. By using that convention is easier to avoid name conflicts, since your local dependencies will have very specific names.
To namespace your packages you need to:
- Move the packages to a
@namespace
subfolder. - Update paths to files outside the project folders accordingly (e.g. the
extends
property intsconfig.json
files referencing the roottsconfig.base.json
). - Update root
package.json
npm scripts that contain project paths. - Modify the
name
property in thepackage.json
file of your npm projects to include the namespace, and updatepackage-lock.json
by runningnpm install
. - In case you are renaming a project (e.g. from
weather-common
to@weather/common
), you will need to rename the project folder and update the corresponding relative import paths as well as potential npm scripts in the rootpackage.json
.
See the corresponding example branch in each section for specific implementation details.
Conclusions
Quoting the popular reference to feline taxidermy, “there is more than one way to skin a cat”. The four described approaches will help you removing duplicated code. The more implementation effort, the more advantages you get. Give them a try and decide which one works better for you. Happy coding! ⌨️