Package management in monorepos [eng]

Hi, my name is Zoltan Kochan. In my presentation, I want to discuss package management in Mono Repos, addressing pain points and sharing strategies for effective package management in such environments. Currently, I am employed at BEAT, where I oversee dependency management tasks as a Lead Software Engineer. Additionally, I serve as the Lead Maintainer of the PNPM open-source project, a GIS package manager. Prior to joining BEAT, I worked at JustAnswer, where we managed a large Mono Repo containing hundreds of components. Installation with NPM in that Mono Repo typically took 30 minutes, prompting me to contribute to PNPM as a faster alternative. With PNPM, installation time was reduced to approximately 90 seconds.

Let's briefly delve into the history of Mono Repo tooling for GIS projects. Babel, a significant project in the JavaScript ecosystem, was one of the earliest popular open-source projects to adopt the Mono Repository model. The creators of Babel developed the first Mono Repository, which utilized Lerna for dependency management in 2015. However, installation with Lerna was notably slow. Recognizing the need for package managers to support Mono Repos seamlessly, both PNPM and YARN introduced Mono Repo support by 2017. YARN termed this feature "workspaces installation," while PNPM referred to it simply as "workspace." Subsequently, NPM also incorporated workspaces support. As of now, three popular Node.js package managers offer built-in support for Mono Repos.

Package managers can organize dependencies in a Mono Repo using two approaches: hoisted and isolated. All three package managers support both layouts by default. YARN and NPM typically employ a hoisted approach, where direct and indirect dependencies are placed in the root Node.modules directory. In cases of multiple versions of the same dependency, one version is hoisted to the root, while the others are nested. On the other hand, PNPM utilizes an isolated Node.modules layout, where each package's dependencies are nested individually. This approach ensures that packages only have access to their own dependencies, unlike the hoisted layout where all projects share dependencies.

Despite the superior developer experience offered by Mono Repos, managing dependencies in such environments can be challenging. For instance, overlooking dependencies, as demonstrated in the example where the "cookie" package is used in "App1" without being listed in its dependencies, can lead to issues. Similarly, improper dependency categorization, as seen with "Lodash" being listed in dev dependencies but used in production code in "main.js" in "App1," can cause problems. To address these issues, special linting rules, such as ESLint's "noextraneous dependencies" rule from the import plugin, can be configured. This rule helps identify imported dependencies not declared in package.json and mismatches between dependencies and their usage contexts, thereby enhancing code quality and reliability.

Working with peer dependencies in Mono Repos can indeed be challenging, particularly ensuring that peer dependencies remain singleton instances during runtime. Ideally, utilizing a single version of the peer dependency across all workspace packages is recommended to maintain consistency and avoid potential issues. As illustrated in the example provided, both "card" and "button" components referencing the same version of React should work seamlessly. This practice of standardizing dependency versions across projects, regardless of whether dealing with peer dependencies, can help mitigate issues and optimize package sizes.

Presently, only YARN natively supports syncing versions of dependencies using constraints, while PNPM plans to introduce this feature through workspace catalogs. Alternatively, third-party tools such as SyncPack can be utilized to identify version duplicates and ensure consistency. In cases where managing multiple versions of a peer dependency becomes necessary within large Mono Repos, PNPM offers a feature known as injected dependencies. This feature allows workspace packages to run with different versions of the peer dependency by enabling the copying of packages. However, it's important to note that modifying components may necessitate rerunning the installation process to update copied instances.

Transitioning to another tool, "bit," addresses Mono Repo challenges by streamlining package management. Although not a package manager itself, bit efficiently manages dependencies and other packages. Functioning as a comprehensive toolchain for building composable software, bit acts as a version control system, dependency manager, and package publisher. Notably, a bit workspace resembles a PNPM workspace, consisting of a collection of components. However, unlike traditional setups, all dependencies for components are declared in a single manifest located at the workspace's root. This centralized approach simplifies dependency management by automatically analyzing code to determine dependencies and their usage contexts.

During the presentation, a demonstration of a bit workspace showcased its functionality within the VSCode environment. Utilizing the bit VSCode extension, a new bit workspace was initialized, and components were created without the need for separate package files. The streamlined process of adding dependencies to components was demonstrated, illustrating bit's ability to conduct code analysis and manage dependencies efficiently across the workspace. Overall, transitioning to bit offers a simplified approach to package management within Mono Repos, enhancing development workflows and ensuring consistency across projects.

In the provided example, the process of managing dependencies using "bit" is demonstrated, highlighting its efficiency and automation capabilities. When executing the command to find import statements and install missing dependencies, "bit" internally utilizes PNPM to handle dependency installation. This ensures that all necessary dependencies, such as "lowdash" and "ramda," are added and resolved correctly, as evidenced by their presence in the root node modules and the workspace's dependencies in workspace.jsonc.

Unlike traditional package management setups, "bit" abstracts away package files, consolidating dependency information in a single manifest located at the workspace's root. Despite this, essential package details, such as dependencies and their usage contexts, are readily accessible within the workspace's components' details. Additionally, "bit" seamlessly manages dependency transitions, such as moving "lowdash" from runtime to dev dependencies, without requiring manual adjustments to package.json files.

The demonstration further showcases "bit's" capability to handle the compilation of packages within the workspace, facilitating the support for multiple versions of a peer dependency. By creating separate runtime environments for components relying on different versions of React, "bit" ensures compatibility and consistency across the workspace. Components are loaded from their respective environments, ensuring that dependencies resolve to the correct version of React specified for each component.

In practical terms, "bit" creates subdirectories within the node modules directory for each environment used in the workspace, such as "react16" and "react17." Components are copied into these directories, with their dependencies resolved accordingly. This setup allows components to utilize the specified version of React within their respective environments, ensuring smooth and reliable execution. Overall, "bit" streamlines package management within Mono Repos, offering automated dependency handling, efficient version control integration, and support for multiple versions of peer dependencies. Its intuitive approach and seamless compatibility make it a valuable tool for managing complex Mono Repo setups and optimizing development workflows.

Certainly, any modifications made to the components will automatically propagate to the corresponding environment directories. This ensures that changes made to the components are consistently reflected across the workspace, maintaining integrity and coherence within the Mono Repo setup. With this, I conclude my presentation. Thank you for the opportunity to share insights on monorepos and the tools available for managing them. I hope you found the information valuable and perhaps consider exploring "bit" for your Mono Repo needs. Have a great day.