cover

Introduction

When Go was first released in 2009, it did not have a module or package management system. In the early days, Golang adopted a GOPATH-based workflow that required developers to organize code of all their projects in the directory pointed to by the GOPATH environment variable. This approach created a global namespace where all the Golang projects would live on the developer's workspace. This special folder also dictated its own folder structure:

  1. $GOPATH/src: This directory contained all the Go projects on which the user is working or the libraries fetched using go get.
  2. $GOPATH/bin or $GOBIN: This folder would contain all the compiled Go binaries.
  3. $GOPATH/pkg: This folder acted as a cache for compiled packages. If multiple projects depended on the same package and if the pre-compiled package was available in this folder, it would speed up compilation and build time for the Go projects.

Problems with $GOPATH approach

Imagine working on Go projects in the above structure. Although simple, this forced developers into a file system structure that was global for all the Golang projects users had checked out. Checking out Go projects outside of the GOPATH would create compilation and build problems. There was also no versioning for the Go packages cache in the $GOPATH/pkg folder.

The biggest problem was that there was no standard way of managing the dependencies of Go projects, forcing users to use community-built tools like godep, glide, and eventually dep to handle versioning and dependency management. These tools often came with their own opinions on how to manage dependencies and created problems for Go developers.

To streamline, simplify, and standardize dependency management for Go Projects, the Go team introduced Go Modules with the release of Go 1.11 in 2018.

Understanding Go Packages and Go Modules

What are Go Packages?

In Go projects, code is organized into packages. A package is a collection of Go source files in the same directory. All files in a package share the same package declaration at the top of each file.

What are Go Modules?

According to the official Go blog on modules, "A Go Module is a collection of Go packages stored in a file tree with a go.mod file at its root."

A Go Module is a collection of Go packages stored in a file tree with a go.mod file at its root. This go.mod file defines the module's name, version, its dependencies, and the minimum Go version needed for building the module. Let's take a look at the go.mod of a popular Go library, Cobra, for creating CLI applications:

module github.com/spf13/cobra

go 1.15

require (
	github.com/cpuguy83/go-md2man/v2 v2.0.4
	github.com/inconshreveable/mousetrap v1.1.0
	github.com/spf13/pflag v1.0.5
	gopkg.in/yaml.v3 v3.0.1
)
  1. The first line module github.com/spf13/cobra declares that the name of this module is github.com/spf13/cobra.
  2. The go directive indicates the minimum Go version needed for building this module.
  3. The require directive identifies the other modules that this module depends on.

Go modules can also be versioned. For versioning, modules follow semantic versioning, consisting of Major.Minor.Patch (e.g., v1.2.3). Now, if you look at the go.mod file above, you will not see any version. So, how does a Go module get its version? We will get to that soon. Let's first understand another file, go.sum, that works alongside go.mod.

go.mod: Defines the module's dependencies and their versions.

go.sum: Contains the expected cryptographic hashes of the content of specific module versions.

Each line in go.sum follows this format:

<module> <version> <hash>
<module> <version>/go.mod <hash>

When you add or update a dependency, Go downloads the module and computes hashes of its content. These hashes are saved in the file go.sum. Two hashes are typically generated for each module dependency:

  1. A hash of the module's zip file.
  2. A hash of the module's go.mod file.

Every time you build your project, Go verifies that the hashes of your dependencies match those in go.sum. If a mismatch is detected, the build fails, protecting against unexpected changes in dependencies. go.sum ensures that everyone working on the project uses exactly the same version of each dependency. It guards against silent, potentially malicious changes to dependencies. Following is the go.sum of the Cobra library at the time of this writing:

github.com/cpuguy83/go-md2man/v2 v2.0.4 h1:wfIWP927BUkWJb2NmU/kNDYIBTh/ziUX91+lVfRxZq4=
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

go.sum is updated automatically when you run commands like go get, go mod tidy, or build your project with new/updated dependencies. Both go.mod and go.sum should be committed to version control. This ensures that all developers (and CI systems) use the same dependency versions.

Note on GO111MODULE Environment Variable

The GO111MODULE environment variable was introduced to facilitate the transition from the GOPATH mode:

  • GO111MODULE=off: Forces GOPATH mode.
  • GO111MODULE=on: Forces module-aware mode.
  • GO111MODULE=auto: Default setting, behaves based on the presence of go.mod file.

Since the introduction of Go modules, the relevance of GO111MODULE has mostly diminished, with module-aware mode becoming the de facto way of managing Go applications and libraries.

Getting Started with Go Modules

Now that we understand what Go modules are, let's understand how to use Go tooling for working with Go modules.

Initializing a new module: go mod init

Navigate to the project directory in your terminal where you want to create a Go module and run the following command to initialize a new module:

go mod init github.com/yourusername/yourproject

This creates a go.mod file in your project root. The module path (e.g., github.com/yourusername/yourproject) should typically match your project's repository location.

Converting an existing project to use Go Modules

To convert an existing project:

  1. Navigate to your project root.
  2. Run go mod init as described above.
  3. Run go mod tidy to add dependencies and clean up unused ones.

Understanding go mod tidy

The primary purpose of go mod tidy is to ensure that the go.mod file accurately reflects the dependencies used in your Go project.

  1. It scans all .go files in your module and analyzes import statements in Go code to determine which packages are actually used.
  2. It adds any dependencies required by the code but not listed in go.mod. This includes transitive dependencies (dependencies of dependencies). Transitive dependencies are marked as // indirect in go.mod.
  3. It removes any dependencies listed in go.mod that are no longer used in the code.
  4. It also updates the go.sum file to ensure it contains the correct cryptographic hashes for all the dependencies.

In short, go mod tidy manages your dependency list and keeps it relevant.

Adding dependencies

To add a dependency, execute go get {module}.

This command does three things:

  1. It downloads the dependencies and stores them in GOPATH.
  2. It updates the go.mod file and adds the module fetched as the dependency.
  3. It updates go.sum with the checksums of the downloaded modules.

To understand GOPATH, refer to the GOPATH section.

Removing dependencies

To remove a dependency, simply remove all the imports of the dependency from your code and run go mod tidy.

Upgrading and downgrading modules

  1. Upgrade to latest: go get -u github.com/pkg/errors
  2. Downgrade/upgrade to specific version: go get github.com/pkg/errors@v0.9.0

Vendoring

Vendoring copies all the project dependencies into the vendor folder in the project root. This is an optional step for projects that choose to vendor dependencies. It's useful for ensuring reproducible builds or when working in environments with limited internet access.

For vendoring, execute the command go mod vendor.

A typical workflow for working with Go modules might look like this:

  1. Add new imports to your code.
  2. Run go get if you need a specific version of a new dependency.
  3. Run go mod tidy to clean up go.mod and go.sum.
  4. (Optional) Run go mod vendor if you're vendoring dependencies.
  5. Commit changes to version control.

go install

go install compiles and installs binaries without modifying go.mod. It's typically used for installing tools, while go get is for managing dependencies. By default, binaries are installed to $GOBIN if set, or $GOPATH/bin if GOBIN is not set. If neither is set, it typically defaults to $HOME/go/bin.

Misc Commands and Notes

# Download a specific module
go get github.com/example/module@v1.2.3

# Update all dependencies
go get -u ./...

# Download dependencies without updating go.mod
go mod download

It's possible that the GOMODCACHE environment variable might be set, and if this is the case, there might be a deviation from the $GOPATH/pkg/mod default location. Like GOPATH, to determine this location, inspect the value of this variable in go env and in your shell environment.

GOPATH

Go stores downloaded modules in a central module cache at $GOPATH/pkg/mod location and the downloaded binaries in $GOPATH/bin.

To determine the GOPATH location, you have to look in the following order of precedence (from highest to lowest):

a. GOPATH set with go env -w GOPATH=path b. GOPATH set in the shell environment c. Default GOPATH (usually $HOME/go on Unix systems)

If GOPATH is set in go env, then it takes precedence over the GOPATH variable set in the SHELL's (bash, zsh, etc.) environment. If both are not set, then Go uses the default value.

With the GOPATH location known, you can find the cached Go Modules in $GOPATH/pkg/mod (fetched by go get) and the installed binaries in $GOPATH/bin (installed by go install).

Additionally, the Go module cache folder has the following structure:

$GOPATH/pkg/mod/{git_provider}/{git_org}/{module}@{version}

With this structure, different versions of the same module are stored separately in the Go module cache. As a result, different projects can use different versions of the same dependency without conflicts. Developers familiar with Java and the Maven build tool can find a striking resemblance between the Go Module Cache and Maven local repository.

Once modules are in the cache, you can build your project offline using these cached versions as Go does not need to fetch the dependency from remote sources.

If you want to clean or purge the cache, you can use go clean -modcache to remove the entire module cache. This is sometimes useful for troubleshooting, but generally not necessary for regular development.

Version selection and semantic versioning

Go Modules use Semantic Versioning (SemVer). Version format: vMAJOR.MINOR.PATCH

  • MAJOR: Incompatible API changes
  • MINOR: Backwards-compatible new features
  • PATCH: Backwards-compatible bug fixes

Advanced Go Module Concepts

Releasing new version of a module

Go uses Semantic Versioning (or SemVer) for the modules in the format v{MAJOR}.{MINOR}.{PATCH} (e.g., v1.2.3). For establishing a version, Go uses the git tag model, and publishing a new version is as simple as executing the following 2 commands:

git tag v{MAJOR}.{MINOR}.{PATCH}
git push origin v{MAJOR}.{MINOR}.{PATCH}
  1. Initial Versioning (Optional but Recommended): Initial versioning is optional but recommended as it sets up the versioning history which helps in tracking the module's history.

Note: Typically v0 and v1 are considered initial versions.

  1. Publishing New Major Versions

Creating a new Major Version requires:

a. Update of module path in go.mod (v2, v3, etc.):

module github.com/{gitOrg}/{module}/v2

b. Commit changes:

git commit -am "Prepare for v2.0.0 release"

c. Create and push git tag:

git tag v2.0.0
git push origin v2.0.0

Note: Since the module path is changing, this will require module clients to update their import paths, and hence it's a breaking change.

  1. Publishing Minor and Patch Versions does not need a change in go.mod but it still needs a new git tag to be published.

Following are some best practices for Module Authors:

  1. Create Git Tags for every release, including minor and patch versions.
  2. Follow Semantic Versioning:
  • Major (v1.0.0, v2.0.0): Breaking changes, require update to import path (v2+).
  • Minor (v1.1.0, v1.2.0): New features, backwards compatible.
  • Patch (v1.0.1, v1.0.2): Bug fixes, backwards compatible.
  1. Only change the module path in go.mod for v2 and above. v0 and v1 use the same path without version suffix.
  2. Consider maintaining a CHANGELOG.md to document changes and release notes across versions.

Minimal version selection explained

Go uses Minimal Version Selection (MVS) to resolve dependencies. It selects the minimum version that satisfies all requirements in the dependency graph, ensuring reproducibility and avoiding version conflicts.

Module queries and version constraints

Go supports various version queries:

  • latest: Latest version
  • v1.2.3: Exact version
  • >v1.2.3: Greater than version
  • <v1.2.3: Less than version

These can be used with go get to specify desired versions.

The module graph and visualization tools

The module graph represents dependencies and their relationships. You can visualize this using:

go mod graph | modgraphviz | dot -Tpng -o graph.png

This requires the graphviz tool and the modgraphviz Go package.

Direct vs. indirect dependencies

  • Direct dependencies are those explicitly required by your module.
  • Indirect dependencies are required by your dependencies.

Both are listed in the go.mod file, but indirect dependencies are marked with // indirect.

Handling incompatible module versions

When faced with incompatible versions, you have two main options:

  1. Use the replace directive to substitute one module version with another.
  2. Update your code to accommodate changes in newer versions.

The replace directive in Go Modules is a powerful feature that allows you to substitute one module for another. It comes in handy in the following situations:

  1. Local Development: When you're working on a dependency locally and want to use your local version instead of the published one.
  2. Forked Repositories: If you've forked a dependency and need to use your fork temporarily.
  3. Dependency Not Yet Published: When you're using a dependency that hasn't been published or tagged yet.
  4. Fixing Bugs: To quickly patch a bug in a dependency without waiting for an official update.
  5. Testing Compatibility: To test your module with a different version of a dependency.

To use it, you can edit your go.mod file directly, adding the replace directive:

module myproject

go 1.17

require github.com/example/library v1.2.3

replace github.com/example/library => ../my-fork-of-library

Or you can perform the same thing by executing:

go mod edit -replace=github.com/example/library=../my-fork-of-library

Working with private Git repositories

To work with private repositories:

  1. Set up Git authentication (e.g., SSH keys, personal access tokens, etc.) for the private Git repository.
  2. Modify the GOPRIVATE environment variable to bypass the public Go proxy for private repos:
export GOPRIVATE=github.com/mycompany/*,gitlab.com/myteam/*

NOTE: GOPRIVATE can be a comma-separated list of multiple Git repos.

Module retraction: When and how to use it

Module retraction in Go allows module authors to mark specific versions of their module as invalid or unsuitable for use. This is done by adding a retract directive in the go.mod file of a newer version of the module. Retracted versions are not automatically removed from users' builds, but Go tools will warn users and avoid selecting these versions unless explicitly requested.

Key points:

  1. Retractions are specified in the go.mod file.
  2. They can be for single versions or version ranges.
  3. Retracted versions are still accessible but flagged as problematic.
  4. Users are warned when attempting to use retracted versions.

Example of retraction in go.mod:

retract (
    v1.0.0 // Critical bug
    [v1.1.0, v1.2.0] // Security vulnerability
)

Retractions are crucial for maintaining the health of the Go ecosystem by allowing module authors to effectively communicate and manage issues with published versions.

For more detailed information, refer to the official Go documentation on module retraction: Module retraction in Go

Best Practices and Common Patterns

  1. Use semantic versioning: Follow SemVer principles when versioning your modules.
  2. Keep your go.mod tidy: Regularly run go mod tidy to clean up dependencies.
  3. Commit go.mod and go.sum: These files should be version controlled.
  4. Use minimal version selection: Let Go's MVS algorithm handle version resolution.
  5. Avoid vendoring unless necessary: Modern Go versions handle dependencies well without vendoring.
  6. Use go.mod for tool dependencies: Manage development tools as module dependencies.
  7. Be cautious with replace directives: Use them sparingly and mainly for temporary solutions.

References

  1. The Go Blog. (2019). "Using Go Modules". Go Programming Language.
  2. go.mod of Cobra Library for building CLI applications
  3. go.sum of Cobra Library
  4. Module retraction in Go