Unsafe. Pointer (unsafe.Pointer); unsafe.Pointer (unsafe.Pointer); unsafe.Pointer (unsafe.Pointer);

Unsafe.Pointer is located in the unsafe package. In this article, we delve into the unsafe package. To be clear, this article is not as long as it used to be, so you can read it easily, which is not too often.

Last time I published an article, including more than 5W words of code, the experience of background editor was very bad, which made me doubt my life. As I said before, you can probably read less than 1 percent of a long article like map. Like the evaluation of the following several students, not many.

Personally, learning itself is not a pleasant thing, edutainment is a good wish. To understand deeply requires an invisible effort. Learning is never an easy thing, boring is normal. Be patient and delve into a problem. Read a book, read an article, write a blog.

Pointer types

Before introducing the Unsafe package, we need to focus on pointer types in the Go language.

When I started programming as an undergraduate, MY first language was C. After that, I have learned C++, Java and Python successively. These languages are quite powerful, but without C language, they are so “simple”. Until I got into Go and found that feeling again. Ken Thompson, one of the authors of Go, is also the author of C. Therefore, Go can be regarded as a C language, and it has many features similar to C, including Pointers.

However, Pointers to the Go language have many limitations compared to Pointers to C. This is for safety, of course, given that modern languages like Java/Python don’t have any Pointers (in this case explicit Pointers) in case programmers make mistakes. Not to mention things like C/C++ that require programmers to clean up the “junk” themselves. So for Go, having a pointer is pretty good, despite its limitations.

Why do we need pointer types? Reference go101.org gives an example:

package main

import "fmt"

func double(x int) {
	x += x
}

func main(a) {
	var a = 3
	double(a)
	fmt.Println(a) / / 3
}
Copy the code

Very simple. I want to double a in double, but the function in this example can’t do that. Why is that? This is because function arguments in Go are passed by value. X in double is just a copy of argument A. Operations on x inside the function cannot be fed back to argument A.

If at this point, a pointer will solve the problem! This is also a common “trick”.

package main

import "fmt"

func double(x *int) {
	*x += *x
	x = nil
}

func main(a) {
	var a = 3
	double(&a)
	fmt.Println(a) / / 6
	
	p := &a
	double(p)
	fmt.Println(a, p == nil) // 12 false
}
Copy the code

It’s a routine operation, no explanation required. The only thing that might be confusing is this:

x = nil
Copy the code

It takes a little thought to conclude that this line of code doesn’t matter at all. X is just a copy of & A because it’s passed by value.

*x += *x
Copy the code

This sentence doubles the value x points to (that is, &a points to the variable a). But operating on x itself (a pointer) doesn’t affect the outer a, so x = nil doesn’t cause any fuss.

Here’s a self-vindication:

However, compared to the flexibility of Pointers in C, Go Pointers are more limited. But that’s part of Go’s success: you can enjoy the convenience of Pointers without their danger.

Limitation 1: The pointer to Go cannot perform mathematical operations.

Here’s a simple example:

a := 5
p := &a

p++
p = &a + 3
Copy the code

The above code will fail to compile and will report an compile error: invalid operation, which means it cannot perform mathematical operations on the pointer.

Limitation 2: Pointers of different types cannot be converted to each other.

Take this short example:

func main(a) {
	a := int(100)
	var f *float64
	
	f = &a
}
Copy the code

A compilation error is also reported:

cannot use &a (type *int) as type *float64 in assignment
Copy the code

I don’t want to go into details about whether two Pointers can be converted to each other in the Go 101 article in Resources. I don’t think there’s any point in remembering this, so if you’re a perfectionist, you can read the original text. Of course I have perfectionism too, but I sometimes hold back.

Restriction 3: Different types of Pointers cannot use == or! = more.

You can compare two Pointers only if they are of the same type or can be converted to each other. In addition, Pointers can be passed by == and! = directly compared to nil.

Limitation 4: Pointer variables of different types cannot be assigned to each other.

This is the same as restriction 3.

What is unsafe

The pointer mentioned above is type-safe, but it has many limitations. Go also has non-type-safe Pointers, which are provided by the unsafe package, unsafe.Pointer. In some cases, it makes code more efficient and, of course, more dangerous.

The Unsafe package is used by the Go compiler at compile time. As the name suggests, it is unsafe and officially not recommended. Unsafe bags make me feel uncomfortable, which is probably what the language designers intended.

But how could a higher-level Gopher not use an unsafe package? It can bypass the Go language’s type system and manipulate memory directly. For example, normally you can’t manipulate unexported members of a structure, but with the unsafe package, you can. The Unsafe package lets me read and write directly to memory, and also cares what you export or unexport.

Why unsafe

The Go language type system is designed for security and efficiency, and sometimes security leads to inefficiency. With the Unsafe package, advanced programmers can use it to bypass the inefficiencies of the type system. As such, it makes sense, and reading the Go source code, you’ll find numerous examples of using the Unsafe package.

Unsafe implementation principle

Let’s look at the source code:

type ArbitraryType int

type Pointer *ArbitraryType
Copy the code

It is Arbitrary that Pointer can point to any type, but it is actually similar to void* in C.

The unsafe package also has three other functions:

func Sizeof(x ArbitraryType) uintptr
func Offsetof(x ArbitraryType) uintptr
func Alignof(x ArbitraryType) uintptr
Copy the code

Sizeof returns the number of bytes occupied by type x, but does not contain the Sizeof what x points to. For example, for a pointer, the size returned by the function is 8 bytes (on 64-bit machines) and the size of a slice is the size of the Slice header.

Offsetof returns the number of bytes between the location of a structure member in memory and the start of the structure. The passed argument must be a structure member.

Alignof returns m, which means that when a type is memory-aligned, its allocated memory address is divisible by m.

Note that all three of these functions return a uintptr type, which is interchangeable with unsafe.Pointer. All three functions are executed at compile time, and their results can be assigned directly to const variables. In addition, because the results of the three functions are operating system – and compiler-dependent, they are not portable.

To sum up, the Unsafe package offers two important capabilities:

Pointers of any type and unsafe.Pointer can be converted to each other.

The Uintptr type and unbroadening.Pointer can be converted to each other.

Pointer cannot be mathematically operated on directly, but can be converted to uintptr, mathematically operated on the Uintptr type, and then converted to pointer.

// Uintptr is an integer type that is large enough to store
type uintptr uintptr
Copy the code

One other thing to note is that the Uintptr doesn’t have pointer semantics, meaning that the objects that the Uintptr points to are ruthlessly recycled by gc. Unsafe.Pointer, on the other hand, has Pointer semantics that protect the object it points to from being garbage collected if it is “useful.”

The unsafe package has several functions that are executed at compile time. After all, the compiler already knows how to allocate memory. In/usr/local/go/SRC/CMD/compile/internal/gc/unsafe. Go path, you can see the compile time go to unsafe package processing function.

The deeper principles require studying the source code of the compiler, which will not be delved into here. Let’s focus on its usage and move on.

How to use Unsafe

Get slice length

From the previous slice article, we learned about the slice header structure definition:

// runtime/slice.go
type slice struct {
    array unsafe.Pointer // Element pointer
    len   int / / the length
    cap   int / / capacity
}
Copy the code

Makeslice (makeslice); makeslice (makeslice); makeslice (makeslice); makeslice (makeslice); makeslice (makeslice)

func makeslice(et *_type, len.cap int) slice
Copy the code

Therefore, we can convert from unsafe.Pointer to uintptr to get the field value of slice.

func main(a) {
	s := make([]int.9.20)
	var Len = *(*int)(unsafe.Pointer(uintptr(unsafe.Pointer(&s)) + uintptr(8)))
	fmt.Println(Len, len(s)) / / 9 9

	var Cap = *(*int)(unsafe.Pointer(uintptr(unsafe.Pointer(&s)) + uintptr(16)))
	fmt.Println(Cap, cap(s)) / / 20 20
}
Copy the code

Len, the conversion process of CAP is as follows:

Len: &s => pointer => uintptr => pointer => *int= >int
Cap: &s => pointer => uintptr => pointer => *int= >int
Copy the code

Get map length

Take a look at the map we talked about in the previous article:

type hmap struct {
	count     int
	flags     uint8
	B         uint8
	noverflow uint16
	hash0     uint32

	buckets    unsafe.Pointer
	oldbuckets unsafe.Pointer
	nevacuate  uintptr

	extra *mapextra
}
Copy the code

Unlike slice, the makemap function returns a pointer to hmap:

func makemap(t *maptype, hint int64, h *hmap, bucket unsafe.Pointer) *hmap
Copy the code

We can still convert to the hamp field via unsafe.Pointer and uintptr, but now count is a second-level Pointer:

func main(a) {
	mp := make(map[string]int)
	mp["qcrao"] = 100
	mp["stefno"] = 18

	count := **(**int)(unsafe.Pointer(&mp))
	fmt.Println(count, len(mp)) 2 / / 2
}
Copy the code

Count conversion process:

&mp => pointer => **int= >int
Copy the code

Map source code application

In map source code, mapAccess1, mapAssign, and mapDelete functions need to locate the location of the key, and hash the key first.

Such as:

b := (*bmap)(unsafe.Pointer(uintptr(h.buckets) + (hash&m)*uintptr(t.bucketsize)))
Copy the code

H. buckkets is an unsafe.Pointer, convert it to uintptr, then add (hash&m)*uintptr(t.bucksize), and the result is again converted to uintptr, and finally, Convert to a Bmap pointer to get the location of the bucket that the key fell into. If you are not familiar with this formula, you can read the previous article, which makes it easy to understand.

The above example is relatively simple, but let’s look at a more difficult example of assignment:

// store new key/value at insert position
if t.indirectkey {
	kmem := newobject(t.key)
	*(*unsafe.Pointer)(insertk) = kmem
	insertk = kmem
}
if t.indirectvalue {
	vmem := newobject(t.elem)
	*(*unsafe.Pointer)(val) = vmem
}

typedmemmove(t.key, insertk, key)
Copy the code

This code does the “assignment” operation after finding where the key is to be inserted. Insertk and val represent the address to be “placed” by the key and value, respectively. If t.indirectKey is true, the bucket stores a pointer to the key, so we need to treat inserTK as a pointer to the pointer, so that the value of the corresponding position in the bucket can be set to the address value of the real key, that is, the key stores a pointer.

The following diagram shows the entire operation of setting a key:

Obj is where the real key is stored. Figure 4, obj indicates that typedmemmove function is successfully assigned after execution.

Offsetof gets the Offsetof a member

For a structure, the offset function can be used to obtain the offset of the structure member, and then obtain the address of the member, read and write the address of the memory, can achieve the purpose of changing the value of the member.

Here’s a memory allocation related fact: a structure is allocated a contiguous chunk of memory, and the address of the structure also represents the address of the first member.

Let’s look at an example:

package main

import (
	"fmt"
	"unsafe"
)

type Programmer struct {
	name string
	language string
}

func main(a) {
	p := Programmer{"stefno"."go"}
	fmt.Println(p)
	
	name := (*string)(unsafe.Pointer(&p))
	*name = "qcrao"

	lang := (*string)(unsafe.Pointer(uintptr(unsafe.Pointer(&p)) + unsafe.Offsetof(p.language)))
	*lang = "Golang"

	fmt.Println(p)
}
Copy the code

Run the code, output:

{stefno go}
{qcrao Golang}
Copy the code

Name is the first member of the structure, so you can parse &p directly to *string. The same principle was used earlier to get the count member of the map.

For a private member of a structure, there is now a way to change its value via unbroadening.Pointer.

I updated the Programmer structure to add a field:

type Programmer struct {
	name string
	age int
	language string
}
Copy the code

And in other packages, so that in main, its three fields are private member variables that cannot be changed directly. However, I can use the safe.sizeof () function to get the Sizeof the member and then calculate the address of the member, modifying the memory directly.

func main(a) {
	p := Programmer{"stefno".18."go"}
	fmt.Println(p)

	lang := (*string)(unsafe.Pointer(uintptr(unsafe.Pointer(&p)) + unsafe.Sizeof(int(0)) + unsafe.Sizeof(string(""))))
	*lang = "Golang"

	fmt.Println(p)
}
Copy the code

Output:

{stefno 18 go}
{stefno 18 Golang}
Copy the code

String and slice conversions

This is a very classic example. Implement conversion between string and bytes slices, requiring zero-copy. If you think about it, the usual way to do this is to iterate through strings or bytes slices and assign values one by one.

To do this, we need to understand the underlying data structure of slice and string:

type StringHeader struct {
	Data uintptr
	Len  int
}

type SliceHeader struct {
	Data uintptr
	Len  int
	Cap  int
}
Copy the code

Above is the structure under the reflection package, SRC /reflect/value.go. Zero-copy can be achieved by simply sharing the underlying []byte array.

func string2bytes(s string) []byte {
	stringHeader := (*reflect.StringHeader)(unsafe.Pointer(&s))

	bh := reflect.SliceHeader{
		Data: stringHeader.Data,
		Len:  stringHeader.Len,
		Cap:  stringHeader.Len,
	}

	return* (* []byte)(unsafe.Pointer(&bh))
}

func bytes2string(b []byte) string{
	sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer(&b))

	sh := reflect.StringHeader{
		Data: sliceHeader.Data,
		Len:  sliceHeader.Len,
	}

	return* (*string)(unsafe.Pointer(&sh))
}
Copy the code

The code is relatively simple and will not be explained in detail. Convert string to Byte slice by constructing slice header and String header.

conclusion

The Unsafe package bypasses Go’s type system in order to manipulate memory directly, making it risky to use. However, in some cases, using the functions provided by the Unsafe package can make code more efficient, and the Go source code uses the unsafe package extensively.

The unsafe package defines Pointer and three functions:

type ArbitraryType int
type Pointer *ArbitraryType

func Sizeof(x ArbitraryType) uintptr
func Offsetof(x ArbitraryType) uintptr
func Alignof(x ArbitraryType) uintptr
Copy the code

There are three functions to get information about the size, offset, alignment, and so on.

The Uintptr can convert to and from unsafe.Pointer, and the Uintptr can do math. In this way, the combination of uintptr and unbroadening.Pointer resolves the constraint that the Go Pointer cannot perform mathematical operations.

Using the unsafe function, you can obtain the addresses of private members of structures and perform read and write operations on them, breaking the type-safety restrictions of Go. With the unsafe package, we focus more on its usage.

Unsafe-looking, by the way, even its name seemed unsightly after its broadening address. On the contrary, it feels cool to use something that is not officially advocated. That’s what it feels like to be a rebel.

Finally, click to read the original article and you will participate in the growth of a thousand stars project. You deserve it!

The resources

【 blowing merciless blog www.flysnow.org/2017/07/06/…

Unaddressed, gocn. VIP /question/37…

[Official document] golang.org/pkg/unsafe/

【 example 】www.opscoder.info/golang_unsa…

Fish bosses blog segmentfault.com/a/119000001 】…

The language bible 】【 www.kancloud.cn/wizardforce…

The pointer and system calls 】 blog.gopheracademy.com/advent-2017…

The pointer and uintptr 】 my.oschina.net/xinxingegey…

The unsafe. Pointer go101.org/article/uns 】…

【go pointer type 】go101.org/article/poi…

[code hole fast learning Go language unsafe] juejin.cn/post/684490…

[Official document] golang.org/pkg/unsafe/

[Jasper’s Nest] www.opscoder.info/golang_unsa…

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Unscrambling the Go language in depth

Pointer types

What is unsafe

Why unsafe

Unsafe implementation principle

How to use Unsafe

Get slice length

Get map length

Map source code application

Offsetof gets the Offsetof a member

String and slice conversions

conclusion

The resources

Unscrambling the Go language in depth

Pointer types

What is unsafe

Why unsafe

Unsafe implementation principle

How to use Unsafe

Get slice length

Get map length

Map source code application

Offsetof gets the Offsetof a member

String and slice conversions

conclusion

The resources

Related Posts

Here are nine hidden changes coming to Windows 11

Simple code to implement the basic functions of DispatcherServlet in springMVC

In data science, Should you choose Python or R?