Good evening!

Go is a relatively simple language with less than a hundred pages of specification.

And there are many resources out there that give you the basic knowledge, such as the official Tour and Go by Example.

But it still has many things that are not mentioned in the basic courses, or features that are easily forgotten, or simply things that are not well-documented and can be understood only via experimentation.

This article is simply a compilation of various features of Go language that are either obscure or easily forgettable. I used some talks, lectures (such as Mail.ru Technostream (Russian)) and applied a lot of tinkering. Consider it an almost-advanced but not really expert guide to Go langauge. Hope you find it useful.

Strings and bytes

In Go, a string is an immutable slice of bytes. That’s it. It does not have to have any encoding or whatever. It may contain any arbitrary bytes, the only difference from []byte is that you cannot modify its elements. That’s why it’s so easy to convert one into another via []byte(myString) or string(myByteSlice).

Writing bytes in strings

If you know Python, you are probably aware of a way of writing certain bytes in bytestrings using the \x format. For example, b'\x50\x51\x52', which is equivalent to b'PQR'.

You can do the same thing in Go directly in strings and characters (see next section), such escape sequences are supported.

Difference between “, ` and ‘

Consider the following example

package main

import (
	"fmt"
)

func main() {
	fmt.Println('z')
	fmt.Printf("%T\n", 'z')
	fmt.Printf("%T\n", "a")
	fmt.Println('я')

	fmt.Println()//simply to break lines to make the output more readable

	fmt.Println("a\x50\n\x51")

	fmt.Println()//simply to break lines to make the output more readable

	fmt.Println(`a\n
		b\x52
		c`)

}

The output is this:

122
int32
string
1103

aP
Q

a\n
		b\x52
		c

What is happening here? The first block shows a difference between a character (or a rune, denoted by ') and a string (denoted by "). Additionally, the character (rune) is represented by a 32-bit integer. That is because it can be a Unicode character, and those can be bigger than one byte. More info here and here. By the way, if you call fmt.Println('a'+'b') (notice the '), it will return 195, because they are int32 under the hood (kinda reminds me of C language).

The next block simply shows escape sequences in action. \x50 and \x51 are converted to characters representing them (P and Q) and the newline is inserted.

Finally, there is a block denoted by `. It represents a raw string in Go, it is shown as-is, even tabs are preserved, and escape sequences are not processed.

What if you wanted to insert a ` at the end of the second line, for example? I don’t know any elegant answers, but my best bet would be

fmt.Println(`a\n
	b\x52`+ "`" +`
	c`)

I simply concatenated a string containing a `, because you cannot write this symbol into a raw string directly without terminating it.

Iterating over strings

Even though strings are simply immutable byte structures, when we iterate over them with a range, they try to be clever. It may be useful, but we should be aware of that.

Consider a string "Я㛼Z". The first element is a Cyrillic letter that is encoded with two bytes. The second is a Chinese character that is encoded with 3 bytes. And English Z requires only one byte. Let’s iterate over this string

s := "Я㛼Z"

for i, ch := range s {
	fmt.Println("Position", i, "symbol", ch)
}

The output

Position 0 symbol 1071
Position 2 symbol 14076
Position 5 symbol 90

It really iterates over characters (runes), not bytes. If we need bytes, we can convert it to byte slice explicitly.

for i, ch := range []byte(s) {
	fmt.Println("Position", i, "symbol", ch)
}

The output of this is

Position 0 symbol 208
Position 1 symbol 175
Position 2 symbol 227
Position 3 symbol 155
Position 4 symbol 188
Position 5 symbol 90

One tiny note though. According to this blogpost, a copy is created in memory when we make a conversion between a string and a byte slice; therefore the string is unaffected is unaffected by the modifications of a source slice (and the string itself is immutable, so there’s no need to talk about the opposite). So keep it in mind when deciding whether it is acceptable by your efficiency requirements or not.

Accessing individual elements

At the same time, if you try to access an element of a string by an index, it will return a byte. So s[0] returns 208.

So, another way of iterating over the bytes of a string is

	s := "Я㛼Z"
	for i:=0;i<len(s);i++ {
		fmt.Println("Position", i, "byte", s[i])
	}

since len returns the byte length, not a rune length.

You can get the rune length either via RuneCountInString or by casting it explicitly via len([]rune(string)). The latter is said to be pretty optimized as of Go1.11. There are more tricks and benchmarks here.

Integer to string

If you come from Python, you may be used to converting integers to their text representations in a string with str(14076). But in Go, if you do fmt.Println(string(14076)), it’ll print out . It’s a literal that is represented by this number in symbols table (UTF probably?). To convert it to a string, you need to import the strconv module and invoke strconv.Itoa(14076). You may also take a look at FormatInt and FormatUint if you need broader options.

To convert the other way around, use Atoi, ParseInt and ParseUint.

Peculiar enumerations

Constants are a quite powerful tool in Go, you can create curious enumerations using them and an identifier iota.

Let’s look at a very straightforward way of writing enumerations, without any magic.

const (
	zero  = 0
	one   = 1
	two   = 2
	three = 3
	four  = 4
	five  = 5
)

Understandable? Yes, but maybe too verbose. And here’s an example that yields exactly the same result.

const (
	zero  = iota
	one
	two
	three
	four
	five
)

The iota becomes 0 on const and increments itself with every new entry.

If you feel like skipping an entry, just put a _ there like this

const (
	zero  = iota
	one
	two
	three
	_
	five
)

four is gone, but five still equals 5. If you remove the _, five will be equal to 4.

Expression preserving

Here’s an example with days of week

type Weekday uint8

const (
	Sunday Weekday = iota + 1
	Monday
	Tuesday
	Wednesday
	Thursday
	Friday
	Saturday
)

It shows an example of starting from 1 instead of 0. So Monday will be 2, Tuesday will be 3, etc. Notice that you shouldn’t specify the type for everything beyond the first element, or it will throw an error.

Another example is denoting magnitudes, like this

type ByteSize uint64

const (
	_           = iota // we don't need a zero
	B
	KB ByteSize = 1 << (10 * iota)  //1024
	MB  //1048576
	GB
	TB
	PB
	//Anything above this will overflow
)

We have dropped a zero, so B is 1. At KB, we made a new expression, so it is 1024. For everything that’s left, it will utilize this same expression. Being able to redefine the expression on our way is a pretty powerful tool.

Another recipe is making bit flags.

type flagg uint8

const (
	flagOne   flagg = 1 << iota //00000001
	flagTwo                     //00000010
	flagThree                   //00000100
	flagFour                    //00001000
	flagFive                    //00010000
	flagSix                     //00100000
	flagSeven                   //01000000
	flagEight                   //10000000
)

Arrays

Just quick note on arrays. Some believe that you won’t be using them because they are too low-level, preferring slices instead. But there are some situations when you might want a fixed-size array.

Slices are specified like this a := []int{1, 2, 3}, while arrays are specified either this way a := [3]int{1, 2, 3} or this way a := [...]int{1, 2, 3}. Functions can take arrays of a given size.

func foo(a [3]int) {
	fmt.Println(a[2], a[1], a[0])
}

func main() {
	//if we initialize all the elements of the array, no need to specify its length explicitly
	a := [...]int{1, 2, 3}
	foo(a)
}

In this example foo will accept only integer arrays of size 3; slices or arrays of other sizes won’t work.

Is there a way to convert an array to a slice? Yes, but it’s a bit hacky and requires tricking Go into thinking that the array is a slice and copy into it

//Converting slice to an array is tricky
b := []int{10, 20, 30, 40, 50} //slice
//We need to create an actual array
var c [3]int //array
//and trick the copy function into thinking that it is a slice by specifying [:].
//It is the way to convert array into a slice, by the way.
copy(c[:], b[:3]) //copying 10, 20 and 30 into array
foo(c)

Arrays can be useful if you have some kind of a fixed-sized data and your function manipulates certain elements of it; you don’t have to check the size of the array, it is always the same.

Arrays and slices in assignments

An unobvious difference between slices and arrays is when you assign the existing entity to another variable. Take a look at this example.

////////
//arrays
////////
a := [...]uint8{1, 2, 3}
b := a

b[1] = 42

fmt.Println("a", a, "b", b)

////////
//slices
////////
c := []uint8{1, 2, 3}
d := c

d[1] = 42

fmt.Println("c", c, "d", d)

The output is

a [1 2 3] b [1 42 3]
c [1 42 3] d [1 42 3]

It seems that when you assign an array to a variable, it is copied (passed by value), while a slice is passed by reference. So, a remains the same when we change an element in b because it’s a copy, while d and c are pointers to the same underlying array and therefore they both look changed (because in reality it is the same structure).

Label breaks

Just a couple of quick examples on breaking from loops. Both have the same loops, one inside another.

for i := 0; i < 5; i++ {
	for j := 0; j < 3; j++ {
		fmt.Println(i, j)
		if i == 1 && j == 0 {
			break
		}
	}
}

The output is

0 0
0 1
0 2
1 0
2 0
2 1
2 2
3 0
3 1
3 2
4 0
4 1
4 2

Sounds reasonable, it breaks from the inner loop once, when i is 1 and j is 0, but the other iterations of the outer loop run the inner loop fully.

Now let’s add just a little element, a label and tie it to that break

outer:
for i := 0; i < 5; i++ {
	for j := 0; j < 3; j++ {
		fmt.Println(i, j)
		if i == 1 && j == 0 {
			break outer
		}
	}
}

The output

0 0
0 1
0 2
1 0

This is basically the way to break the outer loop. Something that we miss in Python.

Panics and threads

…or goroutines, rather. Consider the following example.

package main

import (
	"fmt"
	"time"
)

type float float64

func foo(a float) {
	fmt.Println("Testing", a)
	defer fmt.Println("foo defer", a)
	if a == 42 {
		panic("You've found The Answer!")
	}
}

func bar(a float) {
	defer fmt.Println("bar defer", a)
	foo(a)
}

func main() {
	defer fmt.Println("Deferred function")

	for i := 0; i < 50; i++ {
		go bar(float(i))
	}

	time.Sleep(5 * time.Second)
	fmt.Println("Finished successfully!")
}

If you think of goroutines as threads you might think that the 42nd goroutine would crash, while others would finish their operation normally and both Finished successfully! and Deferred function would be printed as the main function would end successfully as well. But here’s what we see instead

Testing 0
foo defer 0
bar defer 0
Testing 1
foo defer 1
bar defer 1
Testing 2
foo defer 2
bar defer 2
//...skipped some outputs for brevity
Testing 41
foo defer 41
bar defer 41
Testing 42
foo defer 42
bar defer 42
panic: You've found The Answer!

goroutine 47 [running]:
main.foo(0x0, 0x40450000)
	/tmp/sandbox264863691/prog.go:13 +0x120
created by main.main
	/tmp/sandbox264863691/prog.go:21 +0xc0

Program exited: status 2.

They all instantly crash. Including the main, and the deferred function in main() will not be executed! That is because goroutines are not really threads, but rather a convoluted mix of concurrent (or asynchronous) tasks split among threads that are created as needed. And they are still considered the same process. If you know Python, goroutines are like a more efficient multicore threading in terms of execution (since they still have race conditions), but are more like asyncio in terms of crashing (since async tasks are executed within the same thread, so it all crashes if one task throws an unhandled exception or, in Go terms, panics).

One thing to note though is that defers in foo() and bar() are executed on panic. Why are they executed but the one in main() is not? When a program panics (more info), it stops, executes deferred functions and gives control to its caller. That caller executes its deferred functions. And so on until the highest function finishes, then the entire program terminates immediately. The thing is, main() is not a direct caller of bar(), it only creates a goroutine. So when panic makes it go up the stack, the highest function is bar(), not main(), and therefore its defers are not executed. Very confusing, I know; takes more experience to internalize than you can get from simply reading this blog.

Recover

Panics can be recovered using deferred functions, and it might look something like exception handling. Here’s an example

package main

import (
	"fmt"
	"time"
)

func foo() (err error) {

	defer func() {
		if r := recover(); r != nil {
			fmt.Println("Recovered in f", r)
			err = fmt.Errorf("There was a panic: %v", r)
		}
	}()

	panic([]byte{1, 2, 3})

	return
}

func main() {

	fmt.Println(foo())

	time.Sleep(time.Second * 5)
}

This outputs

Recovered in f [1 2 3]
There was a panic: [1 2 3]

In a nutshell, recover() will return nil in any case except when called from a deferred function on panic. It is said to be quite an expensive operation and should be used only as a last-resort handling. Program logic should not rely heavily on panic/recover; instead, returning error from functions is an idiomatic way of exception handling.

It is discussed in this official article in detail.

Execution order on recover

Execution order in the example above is pretty confusing already, but if you’ve managed to internalize it, let’s make it even more convoluted. We’ll add recover to the equation. Let’s modify it a bit.

package main

import (
	"fmt"
	"time"
)

type float float64

func foo(a float) {
	fmt.Println("Testing", a)
	defer fmt.Println("foo defer", a)
	if a == 42 {
		panic("You've found The Answer!")
	}
}

func bar(a float) {
	defer fmt.Println("bar defer", a)
	
	defer func() {
		if r := recover(); r != nil {
			fmt.Println("panic recovered", a)
		}
	}()

	foo(a)
	fmt.Println("bar done", a)
}

func main() {
	defer fmt.Println("Deferred function")

	for i := 0; i < 50; i++ {
		go bar(float(i))
	}

	time.Sleep(5 * time.Second)
	fmt.Println("Finished successfully!")
}

I added fmt.Println("bar done", a) and the recover function in bar(). The output

...deleted for brevity...
Testing 41
foo defer 41
bar done 41
bar defer 41
Testing 42
foo defer 42
panic recovered 42
bar defer 42
Testing 43
foo defer 43
bar done 43
bar defer 43
...deleted for brevity...

When a == 42, foo() panics. First, it executes its deferred function (so we see foo defer 42). Then it gives control to bar() and it executs its deferred functions in the reverse order (as it should be).

It gets interesting here. First, the recovery function is run, we see panic recovered 42. Then, the next deferred function is executed , bar defer 42 is printed. So yes, all deferred functions that were set before foo() are executed.

But we don’t see bar done 42 here. That means, whatever is after the panicking foo() is not executed; upon copleting the execution of defers, bar() instantly returns. Whoa, my head is spinning already! But wait, there is more!

WHAT exactly does it return?

The bar() function is a goroutine in the above example, therefore we don’t care about its return value. But what if we did? Let’s modify our example a bit further. We will make foo() run the recovery; along with that, we will make it return something that we will print in bar().

package main

import (
	"fmt"
	"time"
)

type float float64

func foo(a float) int {
	fmt.Println("Testing", a)
	defer fmt.Println("foo defer", a)

	defer func() {
		if r := recover(); r != nil {
			fmt.Println("panic recovered", a)
		}
	}()

	if a == 42 {
		panic("You've found The Answer!")
	}

	return 100
}

func bar(a float) {
	defer fmt.Println("bar defer", a)

	fmt.Println(foo(a))
	fmt.Println("bar done", a)
}

func main() {
	defer fmt.Println("Deferred function")

	for i := 0; i < 50; i++ {
		go bar(float(i))
	}

	time.Sleep(5 * time.Second)
	fmt.Println("Finished successfully!")
}

Brace yourselves! Here’s the output!

...deleted for brevity...
Testing 41
foo defer 41
100
bar done 41
bar defer 41
Testing 42
panic recovered 42
foo defer 42
0
bar done 42
bar defer 42
Testing 43
foo defer 43
100
bar done 43
bar defer 43
...deleted for brevity...

So, foo() panics and recovers (panic recovered 42), then runs its defer (foo defer 42) and then it returns. bar() prints it out for us and we see that the result is 0. So the panicking function that returns immediately after recovery and running its deferred function outputs the default value for the given return types. If there were error it would be nil; a string would be an empty string, etc.

And then, bar() continues its execution as if no panics happened, because foo() has already taken care of it.

Internalizing it all is tricky yet possible. But still, I find this behaviour to be quite complex and easy to forget. Another reason to refrain from using it heavily in production code.

Functions

Default return variable

There is a somewhat uncommon sugar when it comes to return values of functions in Go. Consider this example

package main

import (
	"fmt"
)

func foo(a int) (res int) {
	res = 42 * a
	return
}

func main() {
	fmt.Println(foo(100))
}

There is a res of type int specifies in return values. This way, you basically create a local variable res that you can manipulate within the function. And when you return, you don’t have to specify res, it is simply returned by default. So the output will be 4200.

Can you return multiple defaults? Sure!

package main

import (
	"fmt"
	"errors"
)

func foo(a int) (res int, err error) {
	res = 42 * a
	if a == 42 {
		err = errors.New("The Answer")
	}
	return
}

func main() {
	fmt.Println(foo(100))
	fmt.Println(foo(42))
}

The output

4200 <nil>
1764 The Answer

If you want to return something else instead, you can specify it in the return value and it’ll work just fine. The following code prints 0 (and for some reason does not consider res to be unused).

package main

import (
	"fmt"
)

func foo(a int) (res int) {
	res = 42 * a
	lol := 0
	return lol
}

func main() {
	fmt.Println(foo(100))
}

init() function

The init() function is special. If you create it, it will be called on program startup (you don’t have to call it explicitly), in every file where you put it (in no particular order so don’t rely on it).

It is useful, for example, to initialize a package-level map variable.

package main

import (
	"fmt"
)

var myMap map[int]string

func init() {
	myMap = map[int]string{}
}

func main() {
	myMap[42] = "The Answer"
	fmt.Println(myMap[42])
}

I don’t have to call init() explicitly, it is invoked automatically and initializes the map so this program runs smoothly. If you remove it, you’ll get panic: assignment to entry in nil map.

Some useful read here. There is a diagram showing the order of execution. A curious takeaway is that init() is executed only once, no matter how many times a package was imported in the current program. Other answers there are useful as well so take a look. For example, it’s important to remember that if you initialize a package level variable via a function call (putting var WhatIsThe = AnswerToLife() outside functions in a file), that function will be executed before init().

A remark on chained defers

Here’s an example

package main

import (
	"fmt"
)

type temmie struct {
	says string
}

func foo() temmie {
	fmt.Println("Creating temmie")
	return temmie{"h0i!"}
}

func (tem temmie) greeting() {
	fmt.Println(tem.says)
}

func main() {
	defer foo().greeting()
	fmt.Println("Nothing to say")
}

The output

Creating temmie
Nothing to say
h0i!

That’s a bit strange, because we expected that foo() would be deferred, so Nothing to say should be the first and Creating temmie should be the second. Well, not exactly. In Go, defer takes only the last function in a chain into account (only greeting() in this case). Everything before it is executed immediately.

Structures

Simulating OOP

Golang is not an object-oriented language per se, but there are some ways to simulate inheritance and polymorphism (and incapsulation? Not really, but never mind). Here’s a big example.

package main

import (
	"fmt"
)

type Person struct {
	name string
	ID   uint64
}

func (p *Person) getName() string {
	return p.name
}

func (p *Person) getPersonalID() string {
	return fmt.Sprintf("%020d", p.ID)
}

type Weapon struct {
	name        string
	weapon_type string
}

//If I call this `getName` and try to call getName on Mercenary,
//it will say `ambiguous selector m.getName` and won't compile.
//Reminds me of multiple inheritance, which is generally a dangerous thing.
func (p *Weapon) getWeaponName() string {
	return p.name
}

func (p *Weapon) getWeaponType() string {
	return p.weapon_type
}

type Mercenary struct {
	Person
	Weapon
	employer string
	ID       string
}

func (m *Mercenary) getPersonalID() string {
	return "CLASSIFIED"
}

func (m *Mercenary) getMercenaryID() string {
	return fmt.Sprintf("%010s", m.ID)
}

func (m *Mercenary) getWeapon() string {
	return fmt.Sprintf("%s (type: %s)", m.Weapon.name, m.Weapon.weapon_type)
}

func main() {
	m := Mercenary{Person: Person{name: "Mister Smith", ID: 10},
		Weapon: Weapon{name: "Colt", weapon_type: "Firearm"}, employer: "KillMe Inc.", ID: "DEADBEEF"}

	fmt.Println("Merc's name:", m.getName())
	fmt.Println("Personal ID number:", m.getPersonalID())
	fmt.Println("Mercenary ID:", m.getMercenaryID())
	fmt.Println("Weapon of choice:", m.getWeapon())
}

The output is

Merc's name: Mister Smith
Personal ID number: CLASSIFIED
Mercenary ID: 00DEADBEEF
Weapon of choice: Colt (type: Firearm)

There is so much going on in here!

First of all, we have Mercenary structure that has nameless fields of Person and Weapon types in it. This looks as if Mercenary inherited Person and Weapon. And the population of this structure is somewhat convoluted, as you can see in the first line of main(). We had to specify structures within structure.

Can we access, say, weapon_type directly? Yes, via m.Weapon.weapon_type. We have accessed it in getWeapon() this way. But the cool thing is that we can create the getWeaponType() method and get it with less dots via m.getWeaponType(). That’s what we did with the name in the first print statement, same idea.

Another thing, we can have fields with the same names in different structures. Both Person and Weapon have name. Both Person and Mercenary have ID fields. I have created different methods to get them (getPersonalID() and getMercenaryID() respectively). Originally, the second output line was supposed to be Personal ID number: 00000000000000000010

…but I have added the second getPersonalID() method, this time for Mercenary, which simply returns CLASSIFIED. And yes, you can create functions with the same name, if they are tied to different structures. So, this is an example of polymorphism simulation. A common person has an openly available personal ID, but a mercecnary has it classified, and that is guaranteed by the new getPersonalID(). Although, nothing really prevents you from getting it via m.Person.ID, that’s why I say there is no incapsulation.

Finally, there is a reason why I called it getWeaponName() and not getName(). In that case, we would have two structures (Person and Weapon) belonging to Mercenary and having methods with the same name tied to them. If I invoked m.getName(), it would not be able to determine which of the two to run. Such a program would not even compile, we’d get ambiguous selector m.getName error. This reminds me of an icky thing called multiple inheritance, which Go disallows when it comes to equal names of methods. Actually, you can name them equally, but then you would have to call them as m.Weapon.getName() or m.Person.getName() but not m.getName(). If we are pursuing brevity when we create methods like that, what’s the point of doing it this way?

Inheriting… interfaces?

Strangely enough, you can put interfaces interfaces into structures too. It’s even more confusing, but consider this

package main

import (
	"fmt"
)

type coolThing uint8

func (ct coolThing) DoThings() string {
	fmt.Println("Cool thing is", ct)
	return "temmie"
}

type iThing interface {
	DoThings() string
}

type thing struct {
	iThing
	data string
}

func (t thing) foo() {
	fmt.Println(t.DoThings())
}

func main() {
	a := coolThing(42)
	
	ta := thing{a, "lol"}
	ta.DoThings()
	
	ta.foo()
}

It is a completely valid code that returns

Cool thing is 42
Cool thing is 42
temmie

So what’s going on? We have created a new type coolThing which represents an integer, made a DoThings() method for it and created iThing interface which coolThing is compatible with. Simple enough so far.

Then we created the thing structure and foo() method tied to it. But wait, there is iThing interface in it. Plus, foo() calls DoThings() which thing does not implement. How is it supposed to work? Let’s see what’s going on in main().

First, we create an instance of coolThing and then we pass it as the first element to thing structure. That first element can be anything that implements DoThings and therefore complies with iThing interface. Then we call DoThings() and it calls the coolThing’s method. Very similar to situation with calling Person’s getName() directly from Mercenary instance.

When we call ta.foo(), it calls the same DoThings(), appends its own data to it and prints.

Everything’s pretty much the same as before. May take some time to understand, since interfaces by themselves can be confusing.

Decorators, sorta…

I haven’t understood this clearly yet myself, but this seems to be a bizarre way that you can use to implement something similar to a Python decorator in Go. Given the complexity, not sure if it is viable in most cases. I will update this section once it becomes clearer.

package main

import (
	"fmt"
	"strconv"
)

//The required boilerplate

type Client interface {
	Process(*int) (string, error)
}

type ClientFunc func(*int) (string, error)

func (f ClientFunc) Process(a *int) (string, error) {
	return f(a)
}

type Decorator func(Client) Client

func Decorate(c Client, decorators ...Decorator) Client {
	decorated := c
	for _, decorator := range decorators {
		decorated = decorator(decorated)
	}
	return decorated
}

//Decorators

func postIncrementer(inc int) Decorator {
	return func(c Client) Client {
		return ClientFunc(func(i *int) (string, error) {
			result, _ := c.Process(i)
			fmt.Println("Incrementing by", inc)
			*i += inc
			return result, nil
		})
	}
}

//Client implementations

type HeavyClient struct {
	alpha uint8
}

func (c HeavyClient) Process(i *int) (string, error) {
	fmt.Println("Processing")
	return strconv.Itoa(int(*i)) + "Pootis" + strconv.Itoa(int(c.alpha)), nil
}

func main() {
	var a int = 42
	client := HeavyClient{100}

	fullProcessor := Decorate(client,
		postIncrementer(5),
	)

	fmt.Println(fullProcessor.Process(&a))
	fmt.Println("a", a)

}

Output:

Processing
Incrementing by 5
42Pootis100 <nil>
a 47

Goroutines and channels

Closures. Nasty with goroutines.

Closures exist in many languages. Passing a variable into a function defined within another function is common in pretty much any language that allows inline functions. It should be done with care though. Here’s an example where we create a slice containing functions that have i enclosed in them, and then call them one by one.

package main

import (
	"fmt"
)

func main() {
	foos := make([]func(), 5)

	for i := 0; i < 5; i++ {
		foos[i] = func() {
			fmt.Println(i)
		} //don't forget these braces, because you need to call this function
	}

	for i := 0; i < 5; i++ {
		foos[i]()
	}
}

If you’re familiar with other languages that allow closures, such as Python, you will probably guess that it will output five fives, not 1 2 3 4 5 each on new line. That is because these functions actually reference the same variable, they don’t get a copy of it upon creation.

Let’s see another example.

package main

import (
	"fmt"
	"time"
)

func main() {
	for i := 0; i < 5; i++ {
		func() {
			fmt.Println(i)
		}() //don't forget these braces, because you need to call this function
	}
	time.Sleep(time.Second)
}

Here we create a function and call it immediately in the same iteration. i hasn’t changed when we call it, so it will be 1 2 3 4 5 as expected.

What does it have to do with goroutines?

Now add just one tiny go to the above code, and everything changes drastically!

package main

import (
	"fmt"
	"time"
)

func main() {
	for i := 0; i < 5; i++ {
		go func() {
			fmt.Println(i)
		}() //don't forget these braces, because you need to call this function
	}
	time.Sleep(time.Second)
}

First of all, we get a warning from go vet: ./prog.go:11:16: loop variable i captured by func literal. This does not prevent the program from compiling, but it still warns us that about possible unexpected behaviour.

What do we get as an output? Five fives, again. So what happens here?

It creates separate goroutines and passes i to them, by reference, just like in the first example in this section. It is very likely that the thread context swith will not happen until the main reaches time.Sleep(time.Second), so these goroutines will not start execution until then. Once they do, they all refer to the same variable and print 5. There is a tiny probability that it will be something like 4 5 5 5 5, if one of the goroutines gets the context before the loop finishes (when it is at its fourth iteration); but I’m not sure if it ever happens, might depend on the implementation, version, OS and CPU. But, if you put time.Sleep inside the loop at the end, even if it is as miniscule as time.Nanosecond, you will get consecutive numbers in the output. It seems that Go’s logic makes the context switch even on the tiniest of Sleeps. Curious; but still, a reminder: you can never be sure in what order the goroutines will be executed. So use channels, mutexes and other tools to ensure the order if you require that.

Here’s an interesting answer regarding that. It gives us a hint: if you want i to be in the function, simply pass it as an argument. It will be passed by value, as a copy.

package main

import (
	"fmt"
	"time"
)

func main() {
	for i := 0; i < 5; i++ {
		go func(i int) {
			fmt.Println(i)
		}(i) //don't forget these braces, because you need to call this function
	}
	time.Sleep(time.Second)
}

Another good read on closures in Go.

A note on closing channels.

I find it easy to forget. When you close a channel, you can still read from it.

A convenient way is via range.

package main

import (
	"fmt"
	"time"
)

func main() {
	ch := make(chan int, 2)

	ch <- 42
	ch <- 42

	go func(cha <-chan int) {
		for i := range cha {
			fmt.Println(i)
		}
		fmt.Println("end")
	}(ch)

	close(ch)

	time.Sleep(time.Second * 5)
	fmt.Println("done")
}

But if you’re reading them directly into variables, you can still track whether they are closed or not via a second variable a channel returns.

package main

import (
	"fmt"
	"time"
)

func main() {
	ch := make(chan int, 2)

	ch <- 42
	ch <- 42

	go func(cha <-chan int) {
		for {
			c, more := <-cha
			time.Sleep(time.Second * 1)
			fmt.Println(c, more)
		}
	}(ch)

	close(ch)

	time.Sleep(time.Second * 5)
	fmt.Println("done")
}

This will print:

42 true
42 true
0 false
0 false
0 false
done

The more is true when there are still values and false when the channel is closed and no more values are left to read. Keep in mind, if I omitted that close, that goroutine would just block on reading the channel and would be terminated when the main gets to the end. So the output would be:

42 true
42 true
done

Generators in Go? Maybe…

Generators are familiar to Pythonistas. It is a way to receive a sequence of values without creating a data structure to store them; they are created on-the-fly. Here’s a simple example of a generator-like approach in Go; it generates random squared numbers for as long as we need them.

package main

import (
	"fmt"
	"math"
	"math/rand"
)

func randomSquaresKindaGenerator() <-chan uint64 {
	ch := make(chan uint64)

	go func() {
		for {
			res := uint64(math.Pow(float64(rand.Int()%100), 2))
			ch <- res
		}
	}()

	return ch
}

func main() {
	gen := randomSquaresKindaGenerator()

	for i := 0; i < 5; i++ {
		fmt.Println(<-gen)
	}
}

We simply create a goroutine and return a channel from that function. Since it loops forever, we will never encounter a block on that channel. One drawback is that there is no obvious way to destroy the generator (unless you pass another channel that makes the loop break when something is sent to it). So be careful, don’t call it multiple times or it’ll clutter the memory with useless routines. You could even leverage Once, which will be discussed later in this article.

Regulating number of threads and scheduling

This example is most likely obsolete and you won’t need it in most cases, but it’s still curious.

package main

import (
	"fmt"
	"runtime"
)

func main() {
	for i := 0; i < 10; i++ {
		go func(i int) {
			fmt.Println("goroutine", i)
		}(i)
	}

	for {
		//infinite loop
	}
}

It will output ten goroutine <number> lines, as you might expect. The program launches with several threads already provided by the runtime, at least it does in the version 1.12 that I’m using. But back in the day, only one thread launched by default, and developers had to put runtime.GOMAXPROCS(runtime.NumCPU()) in the beginning of their programs. Now it is all done automatically. But let’s try recreating that behaviour by putting runtime.GOMAXPROCS(1) in the beginning of main().

What happens? We see no outputs from goroutines. We have only one therad, so there is no true parallelism, only concurrency. That means, the runtime should give control to another task at some point. But there is nothing in that loop to stop at, so it just runs forever and the goroutines are never executed.

Is there a way to make it give control? Yes, and there are two way I can tell right now. One is time.Sleep(time.Nanosecond) inside that infinite loop (We have already used it for this purpose here). Or you can be more explicit and put runtime.Gosched() which yields the processor for other tasks.

But again, it is just an experiment. It is unlikely that you will need these tricks in practice.

Wait Group

Just a quick remark on WaitGroup. It is a convenient way to launch several parallel tasks and wait until they all finish execution.

package main

import (
	"fmt"
	"math/rand"
	"sync"
	"time"
)

func main() {
	var wg sync.WaitGroup

	const nRoutines = 5

	wg.Add(nRoutines) //Sets the counter
	for i := 0; i < nRoutines; i++ {

		go func(i int) {
			defer wg.Done() //Decrements the counter
			time.Sleep(time.Millisecond * 500 * time.Duration(rand.Int()%10))
			fmt.Println("done", i)
		}(i)
	}

	wg.Wait() //Unblocks when the counter is zero
	fmt.Println("All done!")
}

We created the WaitGroup object and assigned the number of tasks to its internal counter via Add(). We could put wg.Add(1) inside the loop in the beginning instead, it would have the same effect.

Then we create goroutines. It’s okay to pass it not as argument, because we need to pass wg by reference anyway. The Done() method decrements the counter in the WaitGroup. Note that it has no arguments, it always decrements by one, unlike Add(n) which lets you increase the value of the counter by any number.

Finally, the main task gets to Wait() which simply blocks until the counter in wg is zero. Similar to join in many other languages, except you don’t need to run it in a loop here.

The output is

done 3
done 4
done 2
done 0
done 1
All done!

RWMutex

If the number of reading operations of your shared data structure significantly outweighs the amount of writes, consider using RWMutex instead of Mutex.

Let’s look at the example where we have 100000 goroutines. Each tries to read a shared datastructure (there is no actual data structure here, but we pretend we have it), except for one, that tries to write into it. Reading operation takes 10 milliseconds, writing - 1 second.

package main

import (
	"fmt"

	"math/rand"
	"sync"
	"time"
)

var wg sync.WaitGroup

var Lock = sync.RWMutex{}

func writing() {
	fmt.Println("writing")
	time.Sleep(time.Second)
}

func reading() {
	//fmt.Println("reading")
	time.Sleep(time.Millisecond * 10)
}

func foo(write bool) {
	defer wg.Done()
	//some operations
	time.Sleep(time.Millisecond * time.Duration(rand.Int63n(10)))

	if write {
		Lock.Lock()
		defer Lock.Unlock()
		writing()
	} else {
		Lock.RLock()
		defer Lock.RUnlock()
		reading()
	}

}

func main() {
	start := time.Now()
	defer func() {
		fmt.Println(time.Since(start))
	}()

	const nRoutines = 100000

	writerIndex := rand.Intn(nRoutines)

	wg.Add(nRoutines)
	for i := 0; i < nRoutines; i++ {
		writer := (i == writerIndex)
		go foo(writer)
	}

	wg.Wait() //Unblocks when the counter is zero
}

So, what is happening here? Goroutines all invoke reading() in parallel, but since they use RLock, they don’t block each other’s access; so they execute it as if there were no lock at all. But when a writing goroutine comes into play, it waits until all the readers finish their jobs, takes the lock and starts writing. At this point, all the readers that remain pause at their RLock() and wait until the writer calls Unlock(). Then they start reading, again, in parallel without blocking each other. The execution time of this program was about 1.3-1.5 seconds.

What if I used Mutex{} with Lock() and Unlock() at all times? I tried, but I didn’t even have patience to sit to the end of the program’s execution, it took forever.

Sharing by communication

…not communication by sharing. Probably said by Dave Cheney, but I’m not sure.

This is a curious principle that lets us modify a shared data structure without even accessing it directly; instead, we can use a goroutine that gets data from other routines via channels and performs actions with the data structure directly. We can eliminate race conditions this way, since it is the only routine that accesses it.

Let’s make a simple banking account and 10 users that have access to it and perform random operations.

package main

import (
	"fmt"
	"math/rand"
)

type insufficientFundsError struct {
	amount float64
	by     float64
}

func (e insufficientFundsError) Error() string {
	return fmt.Sprintf("insufficient funds, cannot withdraw %f, you need %f more to perform this opration", e.amount, e.by)
}

type Account struct {
	balance        float64
	deltaChannel   chan float64
	balanceChannel chan float64
	errChannel     chan error
}

//Private methods

func (a *Account) applyChanges(amount float64) error {
	action := "depositing"
	if amount < 0 {
		action = "withdrawing"
	}
	fmt.Println(action, amount)

	newBalance := a.balance + amount
	if newBalance < 0 {
		return insufficientFundsError{by: newBalance, amount: amount}
	}
	a.balance = newBalance
	return nil
}

func (a *Account) run() {
	var delta float64
	for {
		select {
		case delta = <-a.deltaChannel:
			a.errChannel <- a.applyChanges(delta)
		case a.balanceChannel <- a.balance:
		}
	}
}

//Public methods

//Constructor
func NewAccount(balance float64) (a *Account) {
	a = &Account{
		balance:        balance,
		deltaChannel:   make(chan float64),
		balanceChannel: make(chan float64),
		errChannel:     make(chan error, 1), //I'm not sure if buffering will help anything, but it may. Just don't make the buffer bigger or else there can be a desync in error messages.
	}

	//running the service routine
	go a.run()
	return
}

func (a *Account) Balance() float64 {
	return <-a.balanceChannel
}

func (a *Account) Deposit(amount float64) error {
	a.deltaChannel <- amount
	return <-a.errChannel
}

func (a *Account) Withdraw(amount float64) error {
	a.deltaChannel <- -amount
	return <-a.errChannel
}

func main() {

	acc := NewAccount(20)

	for i := 0; i < 10; i++ {
		go func(i int) {
			var err error
			for k := 0; k < 100; k++ {

				op := rand.Intn(2)

				if op == 1 {
					err = acc.Withdraw(50)
				} else {
					err = acc.Deposit(50)
				}

				if err != nil {
					fmt.Println("Error in routine", i, ":", err.Error())
				}

			}
		}(i)
		fmt.Println("Balance at the end of ", i, " execution:", acc.Balance())
	}
}

We have declared not only the balance, but also the channels that will be used for access. There is a constructor NewAccount() which runs the service goroutine in addition to initializing the fields. Even though Golang does not enforce encapsulation (that’s why I don’t consider Golang a language following OOP. Nor Python.), we will use names starting with small letter for the methods we should not access directly (such as applyChanges() and run()) and capital letter for the interfaces (not Go interface, mind you) that will be used to access and manipulate data (Withdraw(), Deposit() and Balance()).

Changes to the balance are processed synchronously thanks to channels. I heard that with every new version of Go channels get a performance boost, unlike mutexes. And it is simply a… Go-ic (idiomatic) way of doing stuff like that.

Execute only Once

Once is a very simple object that has only one method Do(). It ensures that no matter how many times a function call happens and from how many goroutines and how many different functions are called via this object, a function that is the first to be called will be executed, and only once. Useful for one-time initialization of global objects or configuration reading or whatever.

package main

import (
	"fmt"
	"sync"
)

func foo() {
	fmt.Println("Does this happen once?")

}

func bar() {
	fmt.Println("That too?") //Nope!
}

func main() {
	once := sync.Once{}
	for i := 0; i < 5; i++ {
		once.Do(foo)
		once.Do(bar)
	}
}

Does this happen once? will be printed; bar() will not be executed. One thing to remember is that it takes only functions with no arguments (niladic, I have learnt a new word today!) and no return values. And if foo() panics, Once will consider it executed and will not run it again if you, say, recover.

A recipe for a pool of workers

Worker pool is not implemented in Go just yet. Don’t confuse it with Pool (which is a completely different thing, maybe I will write about it in another article). Creation of goroutines is quite expensive, so if you have millions of tasks, using just several to process them instead of creating a goroutine for each task might be justifiable.

package main

import (
	"fmt"
	"sync"
	"time"
)

type Task interface {
	Execute()
}

type WorkerPool struct {
	mu    sync.Mutex
	size  int
	tasks chan Task
	kill  chan struct{} //empty structure weighs nothing, so it is useful for signalling
	wg    sync.WaitGroup
}

//Private methods

func (p *WorkerPool) worker() {
	defer p.wg.Done()
	for {
		select {
		case task, ok := <-p.tasks:
			if !ok {
				//if task channel is closed and exhausted, exit
				return
			}
			task.Execute()
		case <-p.kill:
			//got termination signal. On Resize(), for example
			return
		}
	}
}

//Public methods

//Constructor
func NewWorkerPool(size int) *WorkerPool {
	pool := &WorkerPool{
		//giving it a big buffer so the main program wouldn't block when assigning tasks
		tasks: make(chan Task, 256),
		kill:  make(chan struct{}),
	}

	pool.Resize(size)
	return pool
}

func (p *WorkerPool) Resize(size int) {
	p.mu.Lock()
	defer p.mu.Unlock()
	for p.size < size {
		//adding workers
		p.size++
		p.wg.Add(1)
		go p.worker()
	}
	for p.size > size {
		//killing workers
		p.size--
		//killing synchronously, while they are waiting
		p.kill <- struct{}{}
	}

}

func (p *WorkerPool) Close() {
	//closing the task channel, this will cause all workers to... go home 8)
	close(p.tasks)
}

func (p *WorkerPool) Wait() {
	p.wg.Wait()
}

func (p *WorkerPool) Exec(task Task) {
	p.tasks <- task
}

//Client code

type MyTask struct {
	param int
}

func (t *MyTask) Execute() {
	fmt.Println("executing", t.param)
	time.Sleep(time.Second)
	fmt.Println("finished executing", t.param)
}

func main() {
	pool := NewWorkerPool(5)

	for i := 0; i < 100; i++ {
		pool.Exec(&MyTask{i})
	}

	//if I used pool.Wait(), it would cause a deadlock
	time.Sleep(time.Second * 10)

	//boosting!
	pool.Resize(20)

	for i := 100; i < 200; i++ {
		pool.Exec(&MyTask{i})
	}

	//closing even though the execution hasn't finished yet.
	//we will not put any new tasks, and workers can read from a closed channel.
	//Workers will all stop once the channel is empty.
	pool.Close()

	//Waiting while they finish the remaining tasks
	pool.Wait()
}

A task is anything that implements Execute(). We have created a simple structure here.

We create the pool, assign some tasks (which take a second each) and look at them running. Too slow, so we increase the amount of workers to 20. It runs way faster now. We assign some more tasks and close the task channel. We have discussed previously that a closed channel can still be read from (but not written to). We have assigned all the tasks, nothing left to write. The worker() logic will make it terminate once the task cahnnel is closed and exhausted. We just have to wait for them to finish processing tasks.

This approach makes the client code very simple and minimalistic. Just be careful with pool.Wait(); if you call it without closing the task channel, the workers will not terminate execution in the end, and it will result in a deadlock.

In conclusion

This is only the tip of the gopher iceberg. There will be more “textbook” articles like that in the future so stay tuned.

Thanks for reading!