In Go, a string value is a sequence of bytes. A string value could be empty and the number of bytes is called the length of the string and is never negative. Strings are immutable: once created, it is impossible to change the contents of a string.
The predeclared string type is string; it is a defined type.

Normally, the length of a string s can be discovered using the built-in function len. This function returns the number of bytes in a string, not the number of characters. This distinction is important because strings in Go support unicode encoding. So a character in a UTF-8 string can be more than one byte. For instance, this is a valid string in Go. Some of the characters here are not even valid UTF-8.

const sample = "\xbd\xb2\x3d\xbc\x20\xe2\x8c\x98"

Printing this string to stdout will produce the following output.

��=� ⌘

Length of a string

Like we discussed in the previous section, the built-in function len returns the number of bytes in a string. So, for instance, if our sample string is “Well done 👍🏼”, then len will return 18.

s := "Well done 👍🏼"
fmt.Printf("len(s) = %d\n", len(s))

The output is:

len(s) = 18

Now, instead of number of bytes, if we wanted to count the number of characters in the string, then we would need to use a different function. May be the standard RuneCountInString function from utf8 might come in handy. Let's try that here:

s := "Well done 👍🏼"

fmt.Printf("# of characters in s = %d\n", utf8.RuneCountInString(s))

The output is:

# of characters in s = 12

This function here returned 12 as the output, but we have only 11 characters in the string s. This is because the function RuneCountInString returns the number of runes in the string. Each unicode code point in a string is represented as a rune literal.

Let's see what the rune literals are here:

s := "Well done 👍🏼"
for _, i := range []rune(s) {
    fmt.Printf("%v ", i)
}

The output is:

87 101 108 108 32 100 111 110 101 32 128077 127996

As expected, we have 12 rune literals in the string s. Note that each english alphabet is a single rune literal, which is equivalent to it’s respective ASCII value, but the emoji character requires two rune literals 128077 and 127996. This is because the emoji used here requires a separate rune or byte to represent the skin tone. So then, how do we actually figure out the number of characters?

There is an open source library github.com/rivo/uniseg that provides a function to do just this. It provides a function GraphemeClusterCount that returns the number of characters that is present in this string. Check out the source code on Github for details about the author and the implementation.

Now let's try this library and see if we can get the actual number of characters in the given string.

import "github.com/rivo/uniseg"

s := "Well done 👍🏼"

fmt.Printf("uniseg.GraphemeClusterCount(s) = %d\n", uniseg.GraphemeClusterCount(s3))

The output is:

uniseg.GraphemeClusterCount(s3) = 11

Finally, we have a function that provides the right value for the number of the characters in the given string. This works in almost all cases that I have tested. Please do run your own tests before using this.

String Conversion

Before we end this article, let's look at a common mistake that beginners to Go might encounter when working with strings. Let's look at a snippet here.

s1 := string(65)
fmt.Printf("length of s1 = %d\n", len(s1))

s2 := strconv.FormatInt(65, 10)
fmt.Printf("length of s2 = %d\n", len(s2))```

Here the output would look like this:

length of s1 = 1
length of s2 = 2

This is because the string(65) returns the character that is represented by integer 65 in unicode. In this case, that value is character A. So the length of s1 is 1. To actually convert the integer to string, use the function FormatInt from strconv package. Here strconv.FormatInt(65, 10) converts base 10 representation of 65 to a string. Hence the variable s2 is set to "65" and the length of s2 is 2.

References

• Strings, bytes, runes and characters in Go https://blog.golang.org/strings

• String types https://golang.org/ref/spec#String_types

• rivo/uniseg package https://github.com/rivo/uniseg

• strconv package https://pkg.go.dev/strconv

I hope this was helpful and if you find any errors or issues in this post, please do let me know in the comments section below.

In case you would like to get notified about more articles like this, please subscribe.

Subscribe now

$ go get -u github.com/abvarun226/goiplookup

What if you did not want this dependency on Github and rather wanted to host your own git server? Or may be you want to move your code to Gitlab or Bitbucket but you are worried about your users having to fix their import paths. These problems can be solved with a vanity URL.

With vanity URL, the idea is to use a custom domain (for example: gopkg.in/yaml.v3) to import packages rather than using github.com in the package name. Your code can still be hosted on Github, but you’d need a service that this vanity URL resolves to, that will provide a way for go get command to checkout the source code from.

How does go get work?

For gopkg.in/yaml.v3, the go get command basically makes a HTTP GET request to https://gopkg.in/yaml.v3?go-get=1 URL. The response is an HTML content with certain <meta> tags that indicate to go get command about where it can clone the source code from. Here is an example of the response:

$ http --body "https://gopkg.in/yaml.v3?go-get=1"
<html>
<head>
<meta name="go-import" content="gopkg.in/yaml.v3 git https://gopkg.in/yaml.v3">
<meta name="go-source" content="gopkg.in/yaml.v3 _ https://github.com/go-yaml/yaml/tree/v3{/dir} https://github.com/go-yaml/yaml/blob/v3{/dir}/{file}#L{line}">
</head>
<body>
go get gopkg.in/yaml.v3
</body>
</html>

The content in go-import meta tag indicates where and how to download the source code. For instance, you can clone the yaml.v3 repository using git.

$ git clone https://gopkg.in/yaml.v3
Cloning into 'yaml.v3'...
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 1643 (delta 0), reused 2 (delta 0), pack-reused 1637
Receiving objects: 100% (1643/1643), 1.52 MiB | 1.37 MiB/s, done.
Resolving deltas: 100% (1069/1069), done.

Server Implementation details

Based on the details of how go get command works, all we need to do is return HTML page with appropriate <meta> tags. For our implementation, let’s say we want the vanity URL ayada.dev and the package we are interested in is yaml. So we want to be able to import the package using ayada.dev/pkg/yaml. Now, we must host the source code on Github, let’s say at github.com/abvarun226/yaml. So the HTTP response from https://ayada.dev/pkg/yaml?go-get=1 should look like this:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="ayada.dev/pkg/yaml git ssh://git@github.com/abvarun226/yaml">
</head>
</html>

If you have very few packages, you can simply use a static file server to serve this html content. If you have more than a few packages or multiple organizations that you would want to have a vanity URL for, then it makes sense to setup a web server that can handle this for you.

We can have a config file describe the import path and the corresponding repository path in the version control system (vcs). The server will use this config file to render the HTML content with appropriate go-import meta tag. It will also automatically handle sub-packages for you.

Let’s say we have two different users in Github with different Go packages. Below is one such config file (P.S. these repo path don’t exist, just an example).

[
    {
        "importroot": "ayada.dev/pkg",
        "vcs": "git",
        "reporoot": "https://github.com/abvarun226"
    },
    {
        "importroot": "ayada.dev/paas",
        "vcs": "git",
        "reporoot": "https://github.com/ayadapaas"
    }
]

For example, package with import path ayada.dev/pkg/mypkg will be redirected to https://github.com/abvarun226/mypkg repository path.

And package with import path ayada.dev/paas/mypkg will be redirected to https://github.com/ayadapaas/mypkg repository path.

The server handler code that does this redirect looks like this:

package handler

import (
    "bytes"
    "net/http"
    "strings"
    "text/template"
)

// VanityServer redirects browsers to godoc or go tool to VCS repository.
func (h *Handler) VanityServer(w http.ResponseWriter, r *http.Request) {
    // Only allow GET method
    if r.Method != http.MethodGet {
        status := http.StatusMethodNotAllowed
        http.Error(w, http.StatusText(status), status)
        return
    }

    pkgName := r.Host + r.URL.Path

    // If go-get param is absent, redirect to godoc URL.
    if r.FormValue("go-get") != "1" {
        if h.opts.GodocURL == "" {
            w.Write([]byte(nothingHere))
            return
        }
        url := h.opts.GodocURL + pkgName
        http.Redirect(w, r, url, http.StatusTemporaryRedirect)
        return
    }

    // go-import mapping rules.
    var importRoot, vcs, repoRoot string
    for _, rule := range h.opts.MappingRules {
        if strings.HasPrefix(strings.ToLower(pkgName), strings.ToLower(rule.ImportRoot)+"/") {
            repoName := strings.Replace(strings.ToLower(pkgName), strings.ToLower(rule.ImportRoot), "", -1)
            repoName = strings.Split(repoName, "/")[1]

            importRoot = rule.ImportRoot + "/" + repoName
            repoRoot = rule.RepoRoot + "/" + repoName
            vcs = rule.VCS

            break
        }
    }

    // Create HTML template with go-import <meta> tag
    d := struct {
        ImportRoot string
        VCS        string
        RepoRoot   string
    }{
        ImportRoot: importRoot,
        VCS:        vcs,
        RepoRoot:   repoRoot,
    }

    var buf bytes.Buffer
    err := tmpl.Execute(&buf, &d)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    w.Header().Set("Cache-Control", "public, max-age=900")
    w.Write(buf.Bytes())
}

var tmpl = template.Must(template.New("main").Parse(`<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="{{.ImportRoot}} {{.VCS}} {{.RepoRoot}}">
</head>
</html>
`))

const nothingHere = `
<html>
<head></head>
<body><h5>Nothing here. Move along.</h5></body>
</html>
`

The response from this server looks like this:

$ http --body "http://ayada.dev/paas/yaml/util?go-get=1"
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="ayada.dev/paas/yaml git https://github.com/ayadapaas/yaml">
</head>
</html>

Without the go-get=1 GET parameter, the server will either redirect to a Godoc URL or display “Nothing here. Move along” message.

With this response, go get command should be able to download the source from the given VCS path.

Final thoughts

In organizations that host their own git server, it makes sense to have a vanity URL for the Go packages. Open source developers can host their Go packages on any VCS. It is pretty easy to setup a server that handles vanity URLs for Go packages, and let the server direct go get command about which VCS to download the source code from. This makes moving Go packages between different VCS straightforward without worrying about changing the import paths in dependent packages.

If you are interested, the complete code for the vanity server is available here: https://github.com/abvarun226/vanity-server

I hope this was helpful and if you find any errors or issues in this post, please do let me know in the comments section below.

In case you would like to get notified about more articles like this, please subscribe.

Subscribe now

Micro gateway server

For this walk through, we want to be able to make HTTP calls to the service and the service is expected to provide JSON data. We’ll need a micro server that acts as the gateway and all services will register themselves with this server. The micro gateway can be made to authenticate or ratelimit requests by defining plugins, but let’s save this for a future post. For now, all we want is run a micro server that acts as the gateway and a service that actually has the business logic.

Now in order to run our micro gateway, we need to build the micro binary and run it. All of this is done on my local computer, without any docker containers, to make it simple.

$ git clone https://github.com/micro/micro.git

$ cd micro

$ git checkout v3.0.2

# Ensure Go is installed on your computer.
$ go version
go version go1.14.9 darwin/amd64

# Let's build the micro server binary.
$ go build -o micro .

# Note the location of this binary. We'll need it later.
$ cp ./micro /usr/local/bin/micro

Now that the micro server binary is ready, let’s start the micro gateway server.

$ ./micro server
2020-11-30 22:54:20  file=user/user.go:45 level=info Loading config key from /Users/greycell/.micro/config_secret_key
2020-11-30 22:54:20  file=user/user.go:80 level=info Loading keys /Users/greycell/.micro/id_rsa and /Users/greycell/.micro/id_rsa.pub
2020-11-30 22:54:20  file=server/server.go:86 level=info Starting server
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering network
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering runtime
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering registry
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering config
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering store
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering broker
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering events
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering auth
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering proxy
2020-11-30 22:54:20  file=server/server.go:111 level=info Registering api
2020-11-30 22:54:20  file=server/server.go:198 level=info Starting server runtime
2020-11-30 22:54:20  file=user/user.go:80 level=info Loading keys /Users/greycell/.micro/id_rsa and /Users/greycell/.micro/id_rsa.pub
2020-11-30 22:54:20  file=service/service.go:190 level=info Starting [service] server
2020-11-30 22:54:20  file=grpc/grpc.go:923 level=info Server [grpc] Listening on [::]:10001
2020-11-30 22:54:20  file=grpc/grpc.go:753 level=info Registry [mdns] Registering node: server-f0fd58e3-a6cc-4b3c-9fe3-4ae4506ce358

Now that our micro gateway has started successfully, let’s create a new micro service, may be a comment service.

Comment microservice

The micro gateway communicates with the comment service using grpc. It will use protocol buffers as a medium to communicate with the service. So, let’s first define our protocol buffers for comment service.

syntax = "proto3";

package comment;

service Comment {
    rpc New(NewRequest) returns (NewResponse) {}
    rpc List(ListRequest) returns (ListResponse) {}
}

message Item {
    string id = 1;
    string post = 2;
    string author = 3;
    string message = 4;
    int64 created = 5;
}

message NewRequest {
    string post = 1;
    string author = 2;
    string message = 3;
}

message NewResponse {
    Item comment = 1;
}

message ListRequest {
    string post = 1;
}

message ListResponse {
    repeated Item comments = 1;
}

Our comment service supports only two endpoints, one for creating a new comment and another to list comments in a post.

We also create a protocol buffer message called Item which basically defines the structure of a single comment. The rpc handler New is for creating a new comment. It accepts NewRequest as the input. The NewRequest message contains the information needed to create a new comment. It returns NewResponse which basically contains the comment data defined in Item.

The rpc handler List is for listing all the comments in a post. Naturally it’ll require the post id which is provided by the message ListRequest. The response is an array of comment Item which is provided by ListResponse message. Let’s save this file under a directory called proto. Our directory structure for the comment service will look like this.

.
├── Makefile
├── README.md
├── go.mod
├── go.sum
├── handler
│   └── comment.go
├── main.go
└── proto
    ├── comment.pb.go
    ├── comment.pb.micro.go
    └── comment.proto

In order to generate the protocol buffers, we’ll need a couple of packages. Let’s compile the list of all the commands we need into a Makefile.

GOPATH:=$(shell go env GOPATH)
MODIFY=Mgithub.com/micro/micro/proto/api/api.proto=github.com/micro/micro/v3/proto/api

.PHONY: init
init:
	go get -u github.com/golang/protobuf/proto
	go get -u github.com/golang/protobuf/protoc-gen-go
	go get github.com/micro/micro/v3/cmd/protoc-gen-micro

.PHONY: proto
proto:    
	protoc --proto_path=. --micro_out=${MODIFY}:. --go_out=${MODIFY}:. proto/comment.proto

.PHONY: build
build: proto
	go build -o service *.go

.PHONY: test
test:
	go test -v ./... -cover

Let’s run the init target to install the packages we need and then run the proto target to generate the micro service interfaces.

$ make init proto

The proto target will create two files under proto directory called comment.pb.micro.go and comment.pb.go. These two files will contain the Go structs for the protocol buffer messages we defined above and also the service handlers required to create and list comments.

Now, let’s define the business logic for our service handlers, New and List, in the file comment.go under handler directory.

// file: ./handler/comment.go

package handler

import (
    "context"
    "time"

    pb "github.com/abvarun226/comment/proto"
    "github.com/google/uuid"
)

// Comment struct.
type Comment struct{}

// NewComment creates a new comment handler.
func NewComment() *Comment {
    return &Comment{}
}

// New handler to create a new comment.
func (c *Comment) New(ctx context.Context, req *pb.NewRequest, rsp *pb.NewResponse) error {
    rsp.Comment = &pb.Item{
        Id:      uuid.New().String(),
        Post:    req.Post,
        Author:  req.Author,
        Message: req.Message,
        Created: time.Now().UTC().Unix(),
    }
    return nil
}

// List handler to list comments in a post.
func (c *Comment) List(ctx context.Context, req *pb.ListRequest, rsp *pb.ListResponse) error {
    rsp.Comments = []*pb.Item{
        {
            Id:      uuid.New().String(),
            Post:    req.Post,
            Author:  "greycell",
            Message: "this is a comment",
            Created: 1604650806,
        },
    }
    return nil
}

Now that we have our handlers to handle the business logic, let’s create the service and register it with the micro gateway server so that the gateway knows how to route requests to comment service.

// file: ./main.go

package main

import (
    "github.com/abvarun226/comment/handler"
    "github.com/micro/micro/v3/service"
    "github.com/micro/micro/v3/service/logger"
)

const (
    // ServiceName is the name of this service.
    ServiceName = "comment"

    // ServerAddress is the registry address.
    ServerAddress = "127.0.0.1:8445"
)

func main() {
    // Create comment service.
    srv := service.New(
        service.Name(ServiceName),
        service.Address(ServerAddress),
    )

    // Register handlers.
    srv.Handle(handler.NewComment())

    // Run service.
    if err := srv.Run(); err != nil {
        logger.Fatal(err)
    }
}

And this is it, the comment service is ready. Next, we’ll need to start the comment service, but before that, using the same micro binary, log into the micro server. Then, start the comment service.

# username is admin, password is micro
$ /usr/local/bin/micro login
Enter username: admin
Enter password: micro

# run the comment service
$ /usr/local/bin/micro run .

# check if the service is running.
$ /usr/local/bin/micro status
NAME    VERSION SOURCE STATUS  BUILD UPDATED METADATA
comment latest  ...    running n/a   4s ago  owner=admin...

If the status shows as error, then check the service logs.

$ /usr/local/bin/micro logs comment

Now, let’s see if the service responds to user HTTP requests. Note that micro gateway is running on localhost port 8080.

# List all comments in post `post1`
$ http --json --body \
  'http://localhost:8080/comment/list' post='post1'
{
    "comments": [
        {
            "author": "greycell",
            "created": "1604650806",
            "id": "830e9eef-251e-472a-b324-41545529ad76",
            "message": "this is a comment",
            "post": "post1"
        }
    ]
}


# Create a new comment.
$ http --json --body POST \
  'http://localhost:8080/comment/new' \
  post='post1' author='greycell' message='test comment'
{
    "comment": {
        "author": "greycell",
        "created": "1606763037",
        "id": "6c68e596-0e38-4ff2-80ad-48e6f45758a1",
        "message": "test comment",
        "post": "post1"
    }
}

Here, the path in the API starts with /comment/ to indicate to the micro gateway that the user is requesting a response from the comment service. The gateway automatically unmarshalled the json body in the request appropriately into the protobuf request message. The gateway also mapped the request to appropriate business handler in the comment service based on the endpoints /comment/new or /comment/list.

Final Thoughts

In this post, we briefly discussed about Micro and how we can easily setup micro gateway and a simple micro service. In the next post, we’ll look at how to setup custom plugins and profiles in micro gateway in order to provide custom features like authentication and rate limits for our HTTP endpoints.

If you’d like to learn more about Micro, checkout https://micro.mu

You can access the full code for this chapter here: https://github.com/abvarun226/microservice/tree/chapter01

I hope this was helpful and if you find any errors or issues in this post, please do let me know in the comments section below.

In case you would like to get notified about more articles like this, please subscribe.

Subscribe now

Ristretto is a high performance, concurrent, memory-bound Go cache. It is contention-proof, scales well and provides consistently high hit-ratios. It is really easy to use with very good documentation and a detailed blog post about it’s architecture and benchmarks.

This article provides a brief implementation of caching objects by tags using Ristretto. Let me first explain the strategy that I’ll use to keep track of tags and the cache keys that are tagged with a specific tag name.

Implementation details

Let’s say we want to tag a cache key with tag name tag1. What we want to do first is to create a set called tag1. We will then add the cache key to be tagged with tag1 to this set. This way, when we want to invalidate a cache key by it’s tag name tag1, we will delete all cache keys that are members of the set tag1.

But unlike Redis, Ristretto is a very simple key-value store. There is no set data structure that the library itself provides. But it should be trivial to create a set data structure in Go using a map. Note that we only want to store strings in the set since cache keys are almost always going to be strings. So let’s create a simple set data structure that accepts strings as it’s members.

// file: ./ds/set.go

package ds

import "sync"

// New StringSet.
func New() *StringSet {
    s := StringSet{
        d: make(map[string]struct{}),
    }
    return &s
}

// We will set the value of the key in the map to this.
var special = struct{}{}

// StringSet struct.
type StringSet struct {
    l sync.RWMutex
    d map[string]struct{}
}

// Add method to add a member to the StringSet.
func (s *StringSet) Add(member string) {
    s.l.Lock()
    defer s.l.Unlock()

    s.d[member] = special
}

// Remove method to remove a member from the StringSet.
func (s *StringSet) Remove(member string) {
    s.l.Lock()
    defer s.l.Unlock()

    delete(s.d, member)
}

// IsMember method to check if a member is present in the StringSet.
func (s *StringSet) IsMember(member string) bool {
    s.l.RLock()
    defer s.l.RUnlock()

    _, found := s.d[member]
    return found
}

// Members method to retrieve all members of the StringSet.
func (s *StringSet) Members() []string {
    s.l.RLock()
    defer s.l.RUnlock()

    keys := make([]string, 0)
    for k := range s.d {
        keys = append(keys, k)
    }
    return keys
}

// Size method to get the cardinality of the StringSet.
func (s *StringSet) Size() int {
    s.l.RLock()
    defer s.l.RUnlock()

    return len(s.d)
}

// Clear method to remove all members from the StringSet.
func (s *StringSet) Clear() {
    s.l.Lock()
    defer s.l.Unlock()

    s.d = make(map[string]struct{})
}

We can use this set data structure like this:

s := ds.New()
s.Add("hello")
s.Add("world")
isMember := s.IsMember("hello") // returns true
members := s.Members() // returns all members of the set.

We can add more methods to this set implementation, like union, intersection etc. But this should be sufficient for our use case here.

Implement a cache store

Now that we have our set data structure, let’s see how we can use this in our cache library. So, let’s initialise our Ristretto cache store.

// file: ./cache/cache.go

package cache

import (
    "log"
    "sync"
    "time"

    "github.com/abvarun226/ristretto-cache/ds"
    "github.com/dgraph-io/ristretto"
)

// New initialises a new cache store.
func New() *Store {
    c, errNew := ristretto.NewCache(&ristretto.Config{
        MaxCost:     1 << 30, // 1GB
        NumCounters: 1e7, // 10M
        BufferItems: 64,
    })
    if errNew != nil {
        log.Fatalf("failed to initialise cache: %v", errNew)
    }
    return &Store{store: c}
}

// Store is our struct for cache store.
type Store struct {
    l     sync.RWMutex
    store *ristretto.Cache
}

The above code snippet is part of our cache library and it initialises the Ristretto cache and returns our local cache store. Based on the settings used, it can store upto 1GB of data in the cache. We’ll now add methods that will help us interact with this local cache store.

// file: ./cache/cache.go

// SetByTags method to set cache by given tags.
func (s *Store) SetByTags(key, value string, expiry time.Duration, tags []string) {
    s.l.Lock()
    defer s.l.Unlock()

    for _, tag := range tags {
        set := ds.New()
        if v, found := s.store.Get(tag); found {
            set = v.(*ds.StringSet)
        }
        set.Add(key)
        s.store.Set(tag, set, 1)
    }

    s.store.SetWithTTL(key, value, 1, expiry)
}

SetByTags method will cache the object and also tag it with the tag names provided as a list of strings. You can use any of these strings to invalidate the cache object. We’ll use the locks to ensure that two or more requests don’t update the same set at the same time.

Now let’s look at the code to invalidate cache using the tags.

// file: ./cache/cache.go

// Invalidate method to invalidate cache with given tags.
func (s *Store) Invalidate(tags []string) {
    s.l.Lock()
    defer s.l.Unlock()

    keys := make([]string, 0)
    for _, tag := range tags {
        set := ds.New()
        if v, found := s.store.Get(tag); found {
            set = v.(*ds.StringSet)
        }
        keys = append(keys, set.Members()...)
        keys = append(keys, tag)
    }

    for _, k := range keys {
        s.store.Del(k)
    }
}

Again, here we use locks to synchronise any changes to the set with the given tag name. This method will get all the cache keys associated with the tag name and delete the keys from the cache store, along with the set.

As for other methods to interact with the cache store, it is pretty straight forward.

// file: ./cache/cache.go

// Get method to retrieve the value of a key. If not present, returns false.
func (s *Store) Get(key string) (string, bool) {
    var value string
    val, found := s.store.Get(key)
    if found {
        value = val.(string)
    }
    return value, found
}

// Delete method to delete a key from cache.
func (s *Store) Delete(key string) {
    s.store.Del(key)
}

// Close method clear and then close the cache store.
func (s *Store) Close() {
    s.store.Clear()
    s.store.Close()
}

Here is how you can use the cache store.

// file: ./example.go

// create a new cache.
c := cache.New()

key1 := "key1"
val1 := "value1"
tags := []string{"tag1", "tag2"}

// cache value and set the tags along with expiry.
c.SetByTags(key1, val1, expiry, tags)

// Do some work here.
time.Sleep(5 * time.Millisecond)

// check if key exists in cache.
if res, found := c.Get(key1); found {
    fmt.Printf("found the key `%s`: `%s`", key1, res)
} else {
    fmt.Printf("key `%s` not found", key1)
}

// Invalidate cache using a tag.
c.Invalidate(tags[0])

// now, check if key exists in cache.
if res, found := c.Get(key1); found {
    fmt.Printf("found the key `%s`: `%s`", key1, res)
} else {
    fmt.Printf("key `%s` not found", key1)
}

The output of the above snippet should look like this:

found the key `key1`: `value1`
key `key1` not found

Final Thoughts

Ristretto is quite performant, probably faster than using Redis, since it avoids a network call to a standalone Redis server. It is pretty easy to implement a caching strategy using Ristretto, but if you want the same cache objects to be available across different processes or instances, then you’d need to use a Redis server or probably a distributed caching library like Groupcache. I’ll write an article about how we can use Groupcache in a future post.

You can access the code here: https://github.com/abvarun226/ristretto-cache

I hope this was helpful and if you find any errors or issues in this post, please do let me know in the comments section below.

In case you would like to get notified about more articles like this, please subscribe.

Subscribe now

Cache Invalidation

When frequently accessed data which is stored in a cache is updated, it is necesssary for the application to evict the stale data from the cache. So when subsequent requests for the same data is received by the application, it will retrieve fresh data from the database and store it in the cache. This way, users will never receive stale data from the application. The process of cache invalidation can be made easier by tagging data with specific tags when storing in the cache. This way, when the data needs to be invalidated, we can use the tag names to evict them from the cache.

Here is an implementation of caching records by tags. We will use Redis and github.com/go-redis/redis package.

package cache

import (
    "github.com/go-redis/redis/v7"
)

type cache struct {
    client redis.Client
}

// SetByTags to cache objects and then tag them.
func (c *cache) SetByTags(ctx context.Context, key string, val interface{}, tags []string, expiry time.Duration) error {
	pipe := c.client.TxPipeline()
	for _, tag := range tags {
		pipe.SAdd(tag, key)
		pipe.Expire(tag, expiry)
	}

	pipe.Set(key, b, expiry)

	_, errExec := pipe.Exec()
	return errExec
}

// Invalidate to invalidate cached objects with given tags.
func (c *cache) Invalidate(ctx context.Context, tags []string) {
	keys := make([]string, 0)
	for _, tag := range tags {
		k, _ := c.client.SMembers(tag).Result()
		keys = append(keys, tag)
		keys = append(keys, k...)
	}
	c.client.Del(keys...)
}

In SetByTags method, we maintain a set data structure for each tag provided. We then add the given cache key into these sets. When invalidating cache by tags, we get all the members, i.e. cache keys, for the given tags and delete them from Redis.

Example usage:

tags := []string{"post1", "post2"}
value := "blog data"

key1 := "blog:one:post1"
key2 := "blog:one:post2"

// Set cache by tags here:
cache.SetByTags(ctx, key1, value, tags, 24 * time.Hour)
cache.SetByTags(ctx, key2, value, tags, 24 * time.Hour)


// Invalidate cache by tag "post1" here.
// Both key1 and key2 will be evicted, since both were tagged with "post1".
cache.Invalidate(ctx, []string{"post1"})

References

• https://redis.io

• https://github.com/go-redis/redis.git

Subscribe now to immediately get notified when more such articles are published.

Subscribe now

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!