Problem of the sample
1. First, before we start, a little bit about it.
In Golang, there are many data structure manipulations that are not thread-safe, such as well-known maps, such as the Container/List package. Thread-safe, which refers to variables instantiated based on such data structures, can operate concurrently, that is, multiple Goroutines can operate simultaneously.
Also, as you may know, Golang supports concurrent contention detection at compile time. Go Build — Race, a lot of Gopher is actually familiar. One thing to note here is that — Race is not only supported for build time, but also for single test time, aka Go Test — Race.
Ok, combining the two points above, let’s look at an example (file xx_test.go) (code example 1) :
package main
import (
"sync"
"testing"
)
func TestAppend(t *testing.T) {
x := []string{"start"}
wg := sync.WaitGroup{}
wg.Add(2)
go func() {
defer wg.Done()
y := append(x, "hello", "world")
t.Log(cap(y), len(y))
} ()
go func() {
defer wg.Done()
z := append(x, "goodbye", "bob")
t.Log(cap(z), len(z))
} ()
wg.Wait()
}
Copy the codeThe code is very simple, initialize a slice, start two coroutines, and operate the slice concurrently.
We do a single test and a competitive test:
$ go test --race
PASS
Ok to test/test12 1.017 s
Copy the codeAccording to the result, the competition test result is passed.
2. Let’s change the above code a bit. Line 9 of the above code, the slice initialization looks like this:
x := []string{"start"}
Copy the codeLet’s make a change and give it a size
x := make([]string, 0, 6)
Copy the codeThat’s it, nothing changes, then we look at the complete code again and do the race check again (code example 2).
package main
import (
"sync"
"testing"
)
func TestAppend(t *testing.T) {
x := make([]string, 0, 6)
wg := sync.WaitGroup{}
wg.Add(2)
go func() {
defer wg.Done()
y := append(x, "hello", "world")
t.Log(cap(y), len(y))
} ()
go func() {
defer wg.Done()
z := append(x, "goodbye", "bob")
t.Log(cap(z), len(z))
} ()
wg.Wait()
}
Copy the codeLet’s take the test again and look at the results:
$ go test --race
= = = = = = = = = = = = = = = = = =
WARNING: DATA RACE
Write at 0x00c0000a4120 by goroutine 8:
test/test12.TestAppend.func2()
/Users/shuai/go/src/test/test12/aaa_test.go:20 +0xbe
Previous write at 0x00c0000a4120 by goroutine 7:
// ...
- FAIL: TestAppend (0.00 s)
aaa_test.go:16: 6 2
aaa_test.go:21: 6 2
testing.go:809: race detected during execution of test
FAIL
exit status 1
FAIL the test/test12 0.012 s
Copy the codeResults:
Very straightforward, Golang directly tells us that there is a data race and the data race test fails. We only changed the initialization of Slice.
Why did the test fail
To understand why this failed, we need to look at the memory changes in our slices in our two examples.
Sliced memory layout for code example 1
Let’s review the way x is initialized in code example 1 (the first code example) in which the race test passes:
x := []string{"start"}
Copy the code
In this slice, the name is X, the length is 1, and the capacity is also 1.
However, it should be noted that in the code example, one or two different coroutines need to add elements to X: “hello”, “world” and “Goodbye “,” Bob “respectively. Therefore, Golang needs to create a new memory space, and the memory change of slice X is shown in the figure:
There are a few key points to this diagram:
-
The original slice is X, and its length and capacity are both 1.
-
Coroutine 1 is slice X, adds elements, and assigns the result to the new variable Y. It is equivalent to opening up memory space Y directly and doing the operation of adding elements.
-
Coroutine 2 is slice X, adds elements, and assigns the result to the new variable Z. It is equivalent to opening up the memory space Z directly and doing the operation of adding elements.
-
When multiple threads read memory X, there is no data contention because the underlying layer of x never changes. The competition test was passed.
Sliced memory layout for code example 2
In the later example, code example 2, the code changes, so let’s review:
x := make([]string, 0, 6)
Copy the code
As can be seen from the figure, the memory layout of slice X is changed, with a length of 0 but a capacity of 6. In code example 2, there are two coroutines, adding two elements each to x. The problem is, in this slice x, there’s enough room for six new elements. So coroutine 1 and coroutine 2 both add new elements to the memory space of slice X.
A race occurs because both goroutines are trying to write to the same area of memory. Hence the data race. Golang Test — Race failed.
The schematic diagram of the competing data on slice X is as follows:
As shown in the figure, coroutine 1 and coroutine 2 compete to operate on the same slice X. In the end, we don’t know who won.
Conclusion:
In Golang’s slicing operation, each call to Append does not force a new memory allocation. So, in the case above, this is a feature of Golang itself, not a bug.
How to avoid the above problems
Solution 1: Pre-allocate target variable memory
The easiest way to do this is if you want to append new data, don’t start with a variable that has a shared state as the first variable to append.
For example, create a new slice with the total capacity you need and use the new slice as the first variable to append.
Here is a code example:
package main
import (
"sync"
"testing"
)
func TestAppend(t *testing.T) {
// Raw slice x
x := make([]string, 0, 6)
x = append(x, "start")
wg := sync.WaitGroup{}
wg.Add(2)
go func() {
defer wg.Done()
// initialize the slice for variable y.
y := make([]string, 0, len(x)+2)
// Put all elements of x into Y, and then append.
y = append(y, x...)
y = append(y, "hello", "world")
t.Log(cap(y), len(y), y[0])
} ()
go func() {
defer wg.Done()
z := make([]string, 0, len(x)+2)
z = append(z, x...)
z = append(z, "goodbye", "bob")
t.Log(cap(z), len(z), z[0])
} ()
wg.Wait()
}
Copy the codeIn general (take the operation of coroutine 1 as an example) :
-
Before append, create a new variable y.
-
Add the original data of X to Y.
-
Implement the new element you need to append.
This operation is a bit cumbersome, not elegant, and somewhat wasteful of memory efficiency.
Solution 2: Add a lock
package main
import (
"github.com/k0kubun/pp"
"sync"
"testing"
)
func TestAppend(t *testing.T) {
x := make([]string, 0, 6)
// instantiate a lock
lock := sync.Mutex{}
wg := sync.WaitGroup{}
wg.Add(2)
go func() {
// Coroutine 1 is locked before append
lock.Lock()
defer lock.Unlock()
defer wg.Done()
y := append(x, "hello", "world")
t.Log(cap(y), len(y))
} ()
go func() {
// Coroutine 2 is locked before append
lock.Lock()
defer lock.Unlock()
defer wg.Done()
z := append(x, "goodbye", "bob")
t.Log(cap(z), len(z))
} ()
wg.Wait()
pp.Println(x)
}
Copy the codeOf course, if you have a better solution, welcome to correct.
Welcome to pay attention to the “South of cape” public account for updates
