Client/Server code sharing in Typescript monorepos

Writing a web server on node.js means using the same programming languages on both sides of a web app. Which should allow for sharing a good amount of code between the client and the server apps. Which should be easy when both apps are part of the same monorepo. Then, why do things get complicated when we add Typescript to the mix?

The “replication” problem

In Javascript monorepos, node.js runs the very same files where the source code is written. Therefore, we can we can easily extract the duplicated code to shared files and require them using relative paths 👍 We will need to decide how we ship the shared files to production environments but that’s a separate story.

In Typescript monorepos however, node.js runs a set of transpiled files, which are usually in a different location from the source code files (e.g. a lib or dist folder). This set of transpiled files is generated by turning the Typescript files in the source folder into Javascript files in the distribution folder, replicating the same folder structure. For example:

root
📂 public
📂 src
|  📂 utils
|  |  📝 arrays.ts
|  📝 main.ts        # import a from './utils/array';
|  📝 express.ts     # express.static(path.join(__dirname, '..', 'public'));
📂 dist
|  📂 utils
|  |  📝 arrays.js
|  📝 main.js        # const a = require('./utils/array'); ✅
|  📝 express.js     # express.static(path.join(__dirname, '..', 'public')); ✅
📝 package.json      # "npm run start": "node dist/main.js" ✅
📝 tsconfig.json

It is possible to keep the transpiled files in the same relative location as the source files, but we usually compile them to a separate location for easier manipulation of the distribution files.

This “replication” strategy is meant so that Typescript can guarantee, for each relative import/require statement, that the corresponding file will be available at the same relative path. Note that the “replication” strategy hereby defines an important property of Typescript: import/require paths are never modified during transpilation.

In the example above, the import a from './utils/array;' import statement in main.ts will be transpiled into const a = require('utils/array'); in main.js. Because the folder structure is the same in both cases, the relative paths are correct for both Typescript and node.js. So far so good.

The “replication” strategy however has unexpected side effects when importing code, via relative paths, from outside the Typescript project root folder (usually the folder where tsconfig.json is located). In those cases, the only way to guarantee the relative paths after transpiling is to include the folders outside the project in the structure of the distribution folder. For example:

root
📂 utils
|  📂 src
|     📝 arrays.ts
📂 server
   📂 public
   📂 src
   |  📝 main.ts              # import a from '../../utils/src/arrays';
   |  📝 express.ts           # express.static(path.join(__dirname, '..', 'public'));
   📂 dist
   |  📂 utils
   |  |  📂 src
   |  |     📝 arrays.js
   |  📂 server
   |     📂 src
   |        📝 main.js        # const a = require('../../utils/src/arrays'); ✅
   |        📝 express.js     # express.static(path.join(__dirname, '..', 'public')); ❌
   📝 package.json            # "npm run start": "node dist/main.js" ❌
   📝 tsconfig.json

So, in this second example, the import statement import a from '../../utils/arrays'; in main.ts becomes const a = require('../../utils/arrays'); in main.js. Still correct for both Typescript and node.js, because the structure of the distribution folder has been specifically generated to support the import relative paths.

However paths that fall outside Typescript’s scope (the start script in the package.json and the path to the public folder where the static assets are stored) are not aware of the change in the folder structure and will lead into runtime errors due to files not existing at the expected locations. That’s not good. And that’s exactly what I call the “replication” problem.

Note that Typescript’s “replication” strategy also applies to client side projects. It usually doesn’t introduce problems however, because usually the client side files are bundled together in a single file. That single “bundle” file contains all the code, so it doesn’t need to import/require any other file via relative paths.

A sample app

Before looking at different ways to solve the “replication” problem, let’s create a sample Typescript web app to better illustrate each different approach that we can use. I will call such web app Weather Now:

• On one side we have a React UI (i.e. weather-client) that fetches weather data for a given city from a web API and displays it in a simple layout.
• On the other side we have an Express web API (i.e. weather-server) with a single endpoint that returns weather data (i.e. /api/weather). It also serves the static UI files in the root url.

Weather Now UI

And just like most modern web applications it contains two types of duplicate code:

• Validation logic. validateCityName is a function which validates that a city name has been provided. It’s executed on both the client side, to improve the error detection user experience, and on the server side, to protect ourselves from API faulty calls.
• Data models. The same Validation, Weather and WeatherIcons types are used in both client and server side. Since we deal with the same data types on both ends, it’s only logic to define symmetric data models.

Sample duplicated code in Weather Now

Now let’s introduce the “replication” problem. By extracting that duplicated code into a common folder (e.g. weather-common) and requiring it through relative path imports, we can observer how the structure of distribution folder changes after transpiling the code (see base-code-extraction branch).

Extraction of duplicated code to common folder

The code extraction leads to the npm start script failing and, if we run the transpiled file in the new file path (i.e. “node weather-server/distribution/weather-server/source/index.js”), the express server fails to serve the static assets. Great, we have introduced the “replication” problem. Now let’s fix it using different solutions 🛠️

Solution 1. Relative paths fixes

The quick and dirty approach to get things working again. We just need to adapt all paths and references outside Typescript import statements to match the changes in the distribution folder. Doable, but can be challenging on large projects, specially because some errors will only appear on runtime.

Branch: 1-path-fixes.

Implementation:

• Change the references to compiled files in package.json: the main property and the start npm script.
• Change the path to the static assets folder. Because we want to keep the ability to run the code without compiling we will need to define this path based on the “execution mode”. This means, for example, introducing an environment variable (e.g. process.env.NODEMON) to distinguish between node and nodemon/ts-node executions.

Adaptations for the structure of the distribution folder

Solution 2. Common npm project

By turning the common folder into an npm project itself, Typescript assumes that the compiled files will exist in the relative paths and it will stop including the outside folders in the distribution folder 💪 Note that as a result of turning the common folder into a separate npm project:

• We are introducing a “build dependency”. We will need to compile the common project separately before compiling the projects that depend on it.
• Typescript looses access to the types definitions. We will need to generate type declaration files for the common project (i.e. setting the declaration property to true in tsconfig.json) and let Typescript know where to locate those files (i.e. setting the types property in the common package.json) in order to compile the dependent projects.
• Note that, at least in VSCode, the type definitions will cause the IDE symbols navigation to resolve to the declaration files instead of the source code. Unfortunately we can’t fix this issue when using relative path imports.

VSCode types definition resolution through declaration files

• Hot reloading will no longer detect changes happening outside the project source folder. We will need to compile the common project in watch mode as well as using nodemon (or a similar tool) to watch for changes in the common project’s source folder.

Branch: 2-npm-project.

Implementation:

• Move the duplicate code to a source subfolder in the common folder and add a tsconfig.json file to set up the Typescript compilation. Set the declaration property to true, in order to generate type declaration files.

Extraction of duplicated code to common npm project

• Add a package.json file to the common folder. Define an npm script to compile the Typescript code (e.g. build). Set the types property, so that Typescript knows where to locate the type declaration files.
• Change the compilation npm scripts in dependant projects to compile the common project as well.

Modifications to build npm script to compile common project

• To maintain the hot reloading in dependent projects, change the development npm scripts to compile the common project in watch mode and set nodemon to watch for changes in the common project’s source folder, via nodemon.json.

Hot reloading necessary changes after extracting duplicated code

Solution 3. Local dependencies

Building on top of the “Common npm project” approach we can use npm local dependencies (natively supported since npm 2.0) to install the common project as a dependency of other npm projects that import code from it, letting node find the common code via node_modules. When using local dependencies:

• npm creates a symbolic link to the local dependency project inside each dependent project’s node_modules folder.
• We no longer need relative paths to import code from outside the project; we can use the common project name instead. We have greater freedom to refactor the common project, without having to reflect the changes in any of the import statements of the dependent projects.
• Importing via project name also allows us to restore the IDE symbols navigation (at least, in VSCode) that we lost when extracting the code into a separate npm project. It is done via the paths property in tsconfig.json.
• We introduce a potential drawback: not being able to install public packages with the same name as the local dependencies. For example, naming a project common would prevent us from installing the common package from the npm registry.
• A convenient way of working around these conflicts is to namespace the packages. In fact, you might have noticed that some popular packages use the @organization/package format on their names: @types/node, @react-native-community/slider, @angular/core, etc. By using that convention is easier to avoid name conflicts, since your local dependencies will have very specific names.

Branch: 3-local-dependency (or 3-local-dependency-namespace for namespaced projects).

Implementation:

• Install the common project as a local dependency in all the projects that are importing code from it, using the common project relative path:

cd weather-client
npm install --save ../weather-common
cd ../weather-server
npm install --save ../weather-common

• This will register the local dependency in both package.json and package-lock.json in the dependent projects.

node.js generated symbolic links

• Replace all the relative path imports with external dependency imports, using the name of the common project.

npm local dependency import

• To restore the IDE symbols navigation, set the Typescript paths property:

{
  "compilerOptions": {
    "paths": {
      "weather-common": ["./weather-common/source"]
    }
  },
  "extends": "./tsconfig.base.json"
}

VSCode types definition resolution through Typescript paths

Note that using Typescript paths to resolve the relative path imports without registering the local npm dependencies will result in runtime errors since, as previously mentioned, Typescript does not modify the imports path on compilation.

Wrong usage of Typescript paths

Solution 4. Workspaces

Building on top of the “Common npm project” approach we can use npm workspaces (introduced in npm 7.0) to simplify the management of the different projects in the monorepo. When using workspaces:

• All dependencies are installed in the root folder’s node_modules, removing the need for each project folder to install its own dependencies. As a result, no package-lock.json files are needed in the project folders.
• Just like in the “Local dependencies” approach, we can use npm local dependencies to install the common project as a dependency of other npm projects that import code from it. Note that this time the local dependency is managed from the root repository, so relatives paths and symbolic links change slightly, but the rest of benefits from local dependencies are kept just the same.
• npm scripts defined in nested projects can be run from the root folder, using the -w or -ws arguments, which allows removing “soft link” npm scripts.
• Because, without workspaces, npm doesn’t detect scripts defined in nested projects, a common way to run nested scripts from the root folder is to create additional scripts in the root package.json which change the folder (thus the “soft link” term) and then run the corresponding npm script from a nested package.json.

Branch: 4-npm-workspaces (or 4-npm-workspaces-namespace for namespaced projects).

Implementation:

• Move the different projects into a specific folder (e.g. projects) and specify that folder through the workspaces property in the root package.json (e.g. ["./projects/*"]).

Enabling npm workspaces in package.json

• Remove the package-lock.json file and the node_modules folder in each project.
• If you have installed local dependencies in any of your projects, you will need to remove them for now, as the relative paths will change when using workspaces. You can later re-install them using the -w argument (see below). Failing to do so can lead to the following error:

npm ERR! Cannot set properties of null (setting 'dev')

npm ERR! A complete log of this run can be found in:
npm ERR /…/.npm/_logs/2022–05–30T05_57_51_827Z-debug.log

• Remove the node_modules folder in the root folder and re-install npm dependencies from the root folder. This will update the package-lock.json as well, since npm workspaces come with an additional set of dependencies (e.g. @nodelib/fs.scandir).
• Install the common project as a local dependency in all the projects that import code from it, using the common project relative path and the -w (or--workspace) argument to specify the target project:

npm install --save ./projects/weather-common -w weather-client
npm install --save ./projects/weather-common -w weather-server

This will register the local dependency in the root package.json and package-lock.json, as well as in the corresponding project’s package.json.

node.js generated symbolic links, through workspaces

• Replace all the relative path imports with external dependency imports, using the name of the common project.

npm local dependency import, through workspaces

• To restore the IDE symbols navigation, set the Typescript paths property:

{
  "compilerOptions": {
    "paths": {
      "weather-common": ["./projects/weather-common/source"]
    }
  },
  "extends": "./tsconfig.base.json"
}

VSCode types definition resolution through Typescript paths

• To remove “soft link” npm scripts, replace the cd instructions in the root package.json npm scripts with the corresponding -w or -ws argument.

Simplification of root folder npm scripts through npm workspaces

Conclusion

We have described the “replication” problem that can occur in Typescript monorepos when sharing code between different projects. We have seen how the structure of the distribution folder changes when not adapting Typescript to support such code sharing, and how that can break some relative paths within the application.

We have also seen different approaches to resolve the “replication” problem, each providing additional advantages for increasing amounts of implementation effort. Quoting the popular reference to feline taxidermy, “there is more than one way to skin a cat”. Give them a try and decide which one works better for you. Happy coding!