Vanity URL for Go packages

A walk through on how to setup a vanity URL for your organization's Go packages.

Dec 02, 2020

If you have been working with Go programming language for a while, you would have noticed that a lot of open source packages that you import start with "github.com/…". You would then use go get command to download the package and add it to your go.mod file. For instance:

$ go get -u github.com/abvarun226/goiplookup

What if you did not want this dependency on Github and rather wanted to host your own git server? Or may be you want to move your code to Gitlab or Bitbucket but you are worried about your users having to fix their import paths. These problems can be solved with a vanity URL.

With vanity URL, the idea is to use a custom domain (for example: gopkg.in/yaml.v3) to import packages rather than using github.com in the package name. Your code can still be hosted on Github, but you’d need a service that this vanity URL resolves to, that will provide a way for go get command to checkout the source code from.

How does go get work?

For gopkg.in/yaml.v3, the go get command basically makes a HTTP GET request to https://gopkg.in/yaml.v3?go-get=1 URL. The response is an HTML content with certain <meta> tags that indicate to go get command about where it can clone the source code from. Here is an example of the response:

$ http --body "https://gopkg.in/yaml.v3?go-get=1"
<html>
<head>
<meta name="go-import" content="gopkg.in/yaml.v3 git https://gopkg.in/yaml.v3">
<meta name="go-source" content="gopkg.in/yaml.v3 _ https://github.com/go-yaml/yaml/tree/v3{/dir} https://github.com/go-yaml/yaml/blob/v3{/dir}/{file}#L{line}">
</head>
<body>
go get gopkg.in/yaml.v3
</body>
</html>

The content in go-import meta tag indicates where and how to download the source code. For instance, you can clone the yaml.v3 repository using git.

$ git clone https://gopkg.in/yaml.v3
Cloning into 'yaml.v3'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 1643 (delta 0), reused 2 (delta 0), pack-reused 1637
Receiving objects: 100% (1643/1643), 1.52 MiB | 1.37 MiB/s, done.
Resolving deltas: 100% (1069/1069), done.

Server Implementation details

Based on the details of how go get command works, all we need to do is return HTML page with appropriate <meta> tags. For our implementation, let’s say we want the vanity URL ayada.dev and the package we are interested in is yaml. So we want to be able to import the package using ayada.dev/pkg/yaml. Now, we must host the source code on Github, let’s say at github.com/abvarun226/yaml. So the HTTP response from https://ayada.dev/pkg/yaml?go-get=1 should look like this:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="ayada.dev/pkg/yaml git ssh://git@github.com/abvarun226/yaml">
</head>
</html>

If you have very few packages, you can simply use a static file server to serve this html content. If you have more than a few packages or multiple organizations that you would want to have a vanity URL for, then it makes sense to setup a web server that can handle this for you.

We can have a config file describe the import path and the corresponding repository path in the version control system (vcs). The server will use this config file to render the HTML content with appropriate go-import meta tag. It will also automatically handle sub-packages for you.

Let’s say we have two different users in Github with different Go packages. Below is one such config file (P.S. these repo path don’t exist, just an example).

[
    {
        "importroot": "ayada.dev/pkg",
        "vcs": "git",
        "reporoot": "https://github.com/abvarun226"
    },
    {
        "importroot": "ayada.dev/paas",
        "vcs": "git",
        "reporoot": "https://github.com/ayadapaas"
    }
]

For example, package with import path ayada.dev/pkg/mypkg will be redirected to https://github.com/abvarun226/mypkg repository path.

And package with import path ayada.dev/paas/mypkg will be redirected to https://github.com/ayadapaas/mypkg repository path.

The server handler code that does this redirect looks like this:

package handler

import (
    "bytes"
    "net/http"
    "strings"
    "text/template"
)

// VanityServer redirects browsers to godoc or go tool to VCS repository.
func (h *Handler) VanityServer(w http.ResponseWriter, r *http.Request) {
    // Only allow GET method
    if r.Method != http.MethodGet {
        status := http.StatusMethodNotAllowed
        http.Error(w, http.StatusText(status), status)
        return
    }

    pkgName := r.Host + r.URL.Path

    // If go-get param is absent, redirect to godoc URL.
    if r.FormValue("go-get") != "1" {
        if h.opts.GodocURL == "" {
            w.Write([]byte(nothingHere))
            return
        }
        url := h.opts.GodocURL + pkgName
        http.Redirect(w, r, url, http.StatusTemporaryRedirect)
        return
    }

    // go-import mapping rules.
    var importRoot, vcs, repoRoot string
    for _, rule := range h.opts.MappingRules {
        if strings.HasPrefix(strings.ToLower(pkgName), strings.ToLower(rule.ImportRoot)+"/") {
            repoName := strings.Replace(strings.ToLower(pkgName), strings.ToLower(rule.ImportRoot), "", -1)
            repoName = strings.Split(repoName, "/")[1]

            importRoot = rule.ImportRoot + "/" + repoName
            repoRoot = rule.RepoRoot + "/" + repoName
            vcs = rule.VCS

            break
        }
    }

    // Create HTML template with go-import <meta> tag
    d := struct {
        ImportRoot string
        VCS        string
        RepoRoot   string
    }{
        ImportRoot: importRoot,
        VCS:        vcs,
        RepoRoot:   repoRoot,
    }

    var buf bytes.Buffer
    err := tmpl.Execute(&buf, &d)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    w.Header().Set("Cache-Control", "public, max-age=900")
    w.Write(buf.Bytes())
}

var tmpl = template.Must(template.New("main").Parse(`<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="{{.ImportRoot}} {{.VCS}} {{.RepoRoot}}">
</head>
</html>
`))

const nothingHere = `
<html>
<head></head>
<body><h5>Nothing here. Move along.</h5></body>
</html>
`

The response from this server looks like this:

$ http --body "http://ayada.dev/paas/yaml/util?go-get=1"
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="ayada.dev/paas/yaml git https://github.com/ayadapaas/yaml">
</head>
</html>

Without the go-get=1 GET parameter, the server will either redirect to a Godoc URL or display “Nothing here. Move along” message.

With this response, go get command should be able to download the source from the given VCS path.

Final thoughts

In organizations that host their own git server, it makes sense to have a vanity URL for the Go packages. Open source developers can host their Go packages on any VCS. It is pretty easy to setup a server that handles vanity URLs for Go packages, and let the server direct go get command about which VCS to download the source code from. This makes moving Go packages between different VCS straightforward without worrying about changing the import paths in dependent packages.

If you are interested, the complete code for the vanity server is available here: https://github.com/abvarun226/vanity-server

I hope this was helpful and if you find any errors or issues in this post, please do let me know in the comments section below.

In case you would like to get notified about more articles like this, please subscribe.

Greycell's Newsletter

Discussion about this post