Arslan. IO / 2017/09/14 /…

By Fatih Arslan

Translator: oopsguy.com

I wrote a tool called GomodifyTags that made my job a lot easier. It automatically populates structure label fields based on field names. Let me show you what it does:

Multiple fields of a structure can be easily managed using such a tool. The tool can also add and remove labels, manage label options (such as omitEmpty), define transformation rules (snake_case, camelCase, and so on), and so on. But how does the tool work? What Go package does it use internally? There are a lot of questions to answer.

This is a very long blog post that explains how to write such a tool and the details of each build. It contains a lot of unique details, tricks, and unknown Go knowledge.

Grab a cup of coffee ☕️, let’s dive in!

First, let me list the things the tool needs to do:

  1. It needs to read the source file, understand it, and be able to parse the Go file
  2. It needs to find the relevant structures
  3. Once the structure is found, it needs to get the field name
  4. It needs to update the structure tag according to the field name (according to the transformation rules, e.gsnake_case)
  5. It needs to be able to update these changes to a file or to output the changed results in a consumable way

Let’s start by understanding what a struct tag is, and from there we can learn everything and how to put it all together and use it, from which you can build tools like this.

The tag value of a structure (content such as JSON: “foo”) is not part of the official specification, but the Reflect package defines an unofficial specification of the format standard, which is also used by stdlib packages such as Encoding /json. It is defined by reflect.structtag:

This definition is a little long and not very easy to understand. Let’s try to break it down:

  • A structure tag is a string literal (because it has a string type)
  • The key part is an unquoted string literal
  • The value part is a quoted string literal
  • Keys and values are separated by a colon (:). Values separated by colons are called key-value pairs
  • Structure tags can optionally contain multiple key-value pairs. Key-value pairs are separated by Spaces.
  • The part that is not defined is the option setting. likeencoding/jsonSuch packages are read as a comma-separated list of values. Everything after the first comma is the options section, for examplefoo,omitempty,string. It has a name calledfooThe value of and [omitempty.string] options
  • Because the structure tag is a string literal, it needs to be surrounded by double or backquotes. Because values must be quoted, we always use backquotes to process the entire tag.

In summary:

Now that we know what a structure tag is, we can easily modify it as needed. Now the question is, how do we parse it so that we can easily modify it? Fortunately, reflect.StructTag contains a method that allows us to parse and return the value of the specified key. Here’s an example:

package main

import (
	"fmt"
	"reflect"
)

func main(a) {
	tag := reflect.StructTag(`species:"gopher" color:"blue"`)
	fmt.Println(tag.Get("color"), tag.Get("species"))}Copy the code

Results:

blue gopher
Copy the code

If the key does not exist, an empty string is returned.

This is very useful, but there are some disadvantages that make it not suitable for us, as we need more flexibility:

  • It cannot detect whether the label is ill-formed (for example, the key part is enclosed in quotes, the value part is not quoted, and so on).
  • It has no way of knowing the semantics of the options.
  • It has no way to iterate over existing tags or return them. We have to know which labels to modify. What if I don’t know the name?
  • It is not possible to modify existing labels.
  • We cannot build new structure tags from scratch.

To improve on this, I wrote a custom Go package that addresses all of the issues mentioned above and provides an API that makes it easy to change aspects of structure tags.

The package is called structtag and can be downloaded from github.com/fatih/struc… To obtain. This package allows us to parse and modify labels in a concise manner. Here is a complete example that you can copy/paste and try for yourself:

package main

import (
	"fmt"

	"github.com/fatih/structtag"
)

func main(a) {
	tag := `json:"foo,omitempty,string" xml:"foo"`

	// parse the tag
	tags, err := structtag.Parse(string(tag))
	iferr ! =nil {
		panic(err)
	}

	// iterate over all tags
	for _, t := range tags.Tags() {
		fmt.Printf("tag: %+v\n", t)
	}

	// get a single tag
	jsonTag, err := tags.Get("json")
	iferr ! =nil {
		panic(err)
	}

	// change existing tag
	jsonTag.Name = "foo_bar"
	jsonTag.Options = nil
	tags.Set(jsonTag)

	// add new tag
	tags.Set(&structtag.Tag{
		Key:     "hcl",
		Name:    "foo",
		Options: []string{"squash"}})// print the tags
	fmt.Println(tags) // Output: json:"foo_bar" xml:"foo" hcl:"foo,squash"
}
Copy the code

Now that we know how to parse, modify, or create structure tags, it’s time to try modifying a Go source file. In the example above, the tag already exists, but how do you get the tag from the existing Go structure?

The answer is through the AST. The AST (Abstract Syntax Tree) allows us to retrieve each identifier (node) from source code. Below you can see an AST (simplified version) of a struct type:

In this tree, we can retrieve and manipulate every identifier, every string, every parenthesis, and so on. These are represented by AST nodes. For example, we can change the field name from Foo to Bar by replacing the node that represents it. The same logic applies to structure tags.

To get a Go AST, we need to parse the source file and convert it into an AST. In fact, both are handled through the same process.

To do this, we’ll use the Go/Parser package to parse the file for the AST (the entire file), and then use the Go/AST package to process the entire tree (we can do this manually, but that’s a topic for another blog post). You can see a complete example below:

package main

import (
	"fmt"
	"go/ast"
	"go/parser"
	"go/token"
)

func main(a) {
	src := `package main type Example struct { Foo string` + " `json:\"foo\"` }"

	fset := token.NewFileSet()
	file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments)
	iferr ! =nil {
		panic(err)
	}

	ast.Inspect(file, func(x ast.Node) bool {
		s, ok := x.(*ast.StructType)
		if! ok {return true
		}

		for _, field := range s.Fields.List {
			fmt.Printf("Field: %s\n", field.Names[0].Name)
			fmt.Printf("Tag: %s\n", field.Tag.Value)
		}
		return false})}Copy the code

Output result:

Field: Foo
Tag:   `json:"foo"`
Copy the code

The code does the following:

  • We defined an example of a Go package using a single structure
  • We use thego/parserPackage to parse the string.parserPackages can also read files (or entire packages) from disk.
  • After parsing, we process the node (assigned to the variable file) and look for theast.StructTypeThe AST node defined (see AST diagram). throughast.Inspect()The function completes the tree processing. It iterates through all nodes until it receives a false value. This is very convenient because it does not need to know every node.
  • We print the structure’s field name and structure label.

We can now do two important things. First, we know how to parse a Go source file and retrieve structure tags (via Go/Parser). Second, we learned how to parse the Go structure tag and modify it as needed (via github.com/fatih/struc…). .

With that in mind, we can now start building our tool (named GomodifyTags) by using these two points of knowledge. The tool should do the following in order

  • Gets a configuration that tells us which structure to modify
  • Find and modify structures based on configuration
  • The output

Since GomodifyTags will primarily apply to the editor, we will pass in the configuration through the CLI flag. The second step involves several steps, such as parsing the file, finding the correct structure, and then modifying the structure (by modifying the AST). Finally, we output the results, whether in the format of the original Go source file or some custom protocol (such as JSON, more on that later).

Here are the main features of the simplified GomodifyTags:

Let’s explain each step in more detail. For the sake of simplicity, I will try to explain the important parts in a general form. The principle is the same, once you’ve finished reading this blog post, you’ll be able to read the entire source code without any guidance (all resources are included at the end of the guide)

Let’s start with the first step to understand how to get the configuration. Here is our configuration, with all the necessary information

type config struct {
	// first section - input & output
	file     string
	modified io.Reader
	output   string
	write    bool

	// second section - struct selection
	offset     int
	structName string
	line       string
	start, end int

	// third section - struct modification
	remove    []string
	add       []string
	override  bool
	transform string
	sort      bool
	clear     bool
	addOpts    []string
	removeOpts []string
	clearOpt   bool
}
Copy the code

It is divided into three main parts:

The first part contains Settings on how and which files to read. This can be the filename of the local file system, or it can come directly from stdin (mainly used in the editor). It also sets how to output the results (go source file or JSON) and whether the file should be overwritten rather than output to STdout.

The second part defines how to select a structure and its fields. There are several ways to do this. We can define it by its offset (cursor position), structure name, single line (select fields only), or series of lines. Finally, we get the starting/ending row anyway. For example, in the following example, you can see that we use its name to select the structure, and then extract the start and end lines to select the correct field:

If used for an editor, it is best to use byte offsets. For example, here you can see that our cursor is just after the port field name, from where we can easily get the start/end line:

The third part of the configuration is actually a one-to-one mapping to the structTag package. It basically allows us to pass the configuration to the StructTag package after reading the field. As you know, the StructTag package allows us to parse a structure tag and modify its parts. But it does not overwrite or update structure fields.

How do we get the configuration? We simply use the Flag package, then create a flag for each field in the configuration, and then assign them. Here’s an example:

flagFile := flag.String("file".""."Filename to be parsed")
cfg := &config{
	file: *flagFile,
}
Copy the code

We do the same for each field in the configuration. For the complete content, see the gomodifyTag tag definition for the current master branch

Once we have the configuration, we can do some basic verification:

func main(a) {
	cfg := config{ ... }

	err := cfg.validate()
	iferr ! =nil {
		log.Fatalln(err)
	}

	// continue parsing
}

// validate validates whether the config is valid or not
func (c *config) validate(a) error {
	if c.file == "" {
		return errors.New("no file is passed")}if c.line == "" && c.offset == 0 && c.structName == "" {
		return errors.New("-line, -offset or -struct is not passed")}ifc.line ! =""&& c.offset ! =0|| c.line ! =""&& c.structName ! =""|| c.offset ! =0&& c.structName ! ="" {
		return errors.New("-line, -offset or -struct cannot be used together. pick one")}if (c.add == nil || len(c.add) == 0) &&
		(c.addOptions == nil || len(c.addOptions) == 0) &&! c.clear && ! c.clearOption && (c.removeOptions ==nil || len(c.removeOptions) == 0) &&
		(c.remove == nil || len(c.remove) == 0) {
		return errors.New("one of " +
			"[-add-tags, -add-options, -remove-tags, -remove-options, -clear-tags, -clear-options]" +
			" should be defined")}return nil
}
Copy the code

Place the validation section in a separate function for testing. Now that we know how to get the configuration and validate it, we continue parsing the file:

We are already discussing how to parse the file. The resolution here is a method of the Config structure. In fact, all methods are part of the config structure:

func main(a) {
	cfg := config{}

	node, err := cfg.parse()
	iferr ! =nil {
		return err
	}

	// continue find struct selection ...
}

func (c *config) parse(a) (ast.Node, error) {
	c.fset = token.NewFileSet()
	var contents interface{}
	ifc.modified ! =nil {
		archive, err := buildutil.ParseOverlayArchive(c.modified)
		iferr ! =nil {
			return nil, fmt.Errorf("failed to parse -modified archive: %v", err)
		}
		fc, ok := archive[c.file]
		if! ok {return nil, fmt.Errorf("couldn't find %s in archive", c.file)
		}
		contents = fc
	}

	return parser.ParseFile(c.fset, c.file, contents, parser.ParseComments)
}
Copy the code

The parse function does only one thing: parse the source code and return an ast.node. It would be very simple if we passed in a file, in which case we use the parser.parsefile () function. Note the token.newfileset (), which creates a * token.fileset type. We store it in c.set and also pass it to the parser.parsefile () function. Why is that?

Fileset is used to store location information for each node independently for each file. This can be useful later to get the exact location of ast.node (note that ast.node uses a compressed location information token.pos. To get more information, it needs to get a token.position via the token.fileset.position () function, which contains more information)

Let’s move on. This is even more interesting if the source file is passed through stdin. The config.modified field is an easy-to-test IO.Reader, but we’re actually passing stdin. How do we detect if we need to read from stdin?

We asked users if they wanted to deliver content via stdin. In this case, the tool user needs to pass the Modified flag (which is a Boolean flag). If the user passes it, we just assign stdin to C. modified:

flagModified = flag.Bool("modified".false."read an archive of modified files from standard input")

if *flagModified {
	cfg.modified = os.Stdin
}
Copy the code

If you look again at the config.parse() function above, you will see that we check to see if the.Modified field has been assigned. Because STDIN is an arbitrary data stream, we need to be able to parse it according to the given protocol. In this case, we assume that the archive contains the following:

  • File name, followed by a new line
  • File size (decimal), followed by a new line
  • Contents of the document

Because we know the size of the file, we can parse the contents of the file without any problems. Anything beyond a given file size, we simply stop parsing.

This method is also used by several other tools (guru, GogetDoc, etc.) and is useful for editors. This allows the editor to pass the modified contents of the file without saving them to the file system. Hence the name modified.

Now that we have our node, let’s proceed to the “search struct” step:

In the main function, we will call findSelection() using the ast.node obtained from the previous parsing:

func main(a) {
	// ... parse file and get ast.Node

	start, end, err := cfg.findSelection(node)
	iferr ! =nil {
		return err
	}

	// continue rewriting the node with the start&end position
}
Copy the code

The cfg.findSelection() function returns the start and end positions of the structure based on the configuration to tell us how to select a structure. It iterates over the given node and returns the start/end position (as described in the configuration section above) :

But how? Remember there are three patterns. These are row selection, offset, and structure name:

// findSelection returns the start and end position of the fields that are
// suspect to change. It depends on the line, struct or offset selection.
func (c *config) findSelection(node ast.Node) (int.int, error) {
	ifc.line ! ="" {
		return c.lineSelection(node)
	} else ifc.offset ! =0 {
		return c.offsetSelection(node)
	} else ifc.structName ! ="" {
		return c.structSelection(node)
	} else {
		return 0.0, errors.New("-line, -offset or -struct is not passed")}}Copy the code

Row selection is the easy part. Here we just return the flag value itself. So if the user passes the –line 3,50 flag, the function returns (3, 50, nil). All it does is split the flag value and convert it to an integer (again performing validation) :

func (c *config) lineSelection(file ast.Node) (int.int, error) {
	var err error
	splitted := strings.Split(c.line, ",")

	start, err := strconv.Atoi(splitted[0])
	iferr ! =nil {
		return 0.0, err
	}

	end := start
	if len(splitted) == 2 {
		end, err = strconv.Atoi(splitted[1])
		iferr ! =nil {
			return 0.0, err
		}
	}

	if start > end {
		return 0.0, errors.New("wrong range. start line cannot be larger than end line")}return start, end, nil
}
Copy the code

The editor uses this mode when you select a set of rows and highlight them.

The offset and structure name selection requires more work. For these, we first need to collect all given structures so that we can compute offset positions or search for structure names. To do this, we first have a function that collects all structures:

// collectStructs collects and maps structType nodes to their positions
func collectStructs(node ast.Node) map[token.Pos] *structType {
	structs := make(map[token.Pos]*structType, 0)
	collectStructs := func(n ast.Node) bool {
		t, ok := n.(*ast.TypeSpec)
		if! ok {return true
		}

		if t.Type == nil {
			return true
		}

		structName := t.Name.Name

		x, ok := t.Type.(*ast.StructType)
		if! ok {return true
		}

		structs[x.Pos()] = &structType{
			name: structName,
			node: x,
		}
		return true
	}
	ast.Inspect(node, collectStructs)
	return structs
}
Copy the code

We use the ast.inspect () function to step through the AST and search for structures. We first search for * ast.typespec so that we can get the structure name. A search for *ast.StructType is given the structure itself, not its name. That’s why we have a custom structType that holds the name and structure node itself. It’s easy to get around. Because the location of each structure is unique, and there cannot be two different structures in the same location, we use the location as the map’s key.

Now that we have all the structures, at the end we can return a structure with the offset of the start and end positions and the structure name pattern. For the offset position, we check if the offset is between the given structures:

func (c *config) offsetSelection(file ast.Node) (int.int, error) {
	structs := collectStructs(file)

	var encStruct *ast.StructType
	for _, st := range structs {
		structBegin := c.fset.Position(st.node.Pos()).Offset
		structEnd := c.fset.Position(st.node.End()).Offset

		if structBegin <= c.offset && c.offset <= structEnd {
			encStruct = st.node
			break}}if encStruct == nil {
		return 0.0, errors.New("offset is not inside a struct")}// offset mode selects all fields
	start := c.fset.Position(encStruct.Pos()).Line
	end := c.fset.Position(encStruct.End()).Line

	return start, end, nil
}
Copy the code

We use collectStructs() to collect all the structures and then iterate over them. Remember that we stored the original token.fileset for parsing files?

We can now use it to get the Offset information for each structure node (we decode it into a token.Position, which provides us with the.offset field). All we do is a simple check and iteration until we find the structure (named encStruct here) :

for _, st := range structs {
	structBegin := c.fset.Position(st.node.Pos()).Offset
	structEnd := c.fset.Position(st.node.End()).Offset

	if structBegin <= c.offset && c.offset <= structEnd {
		encStruct = st.node
		break}}Copy the code

With this information, we can extract the starting and ending positions of the found structures:

start := c.fset.Position(encStruct.Pos()).Line
end := c.fset.Position(encStruct.End()).Line
Copy the code

The same logic applies to structure name selection. All we do is try to check the structure name until we find a structure that matches the given name, rather than check if the offset is within the given structure:

func (c *config) structSelection(file ast.Node) (int.int, error) {
	// ...

	for _, st := range structs {
		if st.name == c.structName {
			encStruct = st.node
		}
	}

	// ...
}
Copy the code

Now that we have the start and end locations, we can finally move on to the third step: modifying the structure fields.

In the main function, we’ll call the cfg.rewrite() function using the node parsed from the previous step:

func main(a) {
	// ... find start and end position of the struct to be modified


	rewrittenNode, errs := cfg.rewrite(node, start, end)
	iferrs ! =nil {
		if_, ok := errs.(*rewriteErrors); ! ok {return errs
		}
	}


	// continue outputting the rewritten node
}
Copy the code

This is the heart of the tool. In the rewrite function, we’ll rewrite all the structure fields between the start and end positions. Before diving in, we can look at the general contents of this function:

// rewrite rewrites the node for structs between the start and end
// positions and returns the rewritten node
func (c *config) rewrite(node ast.Node, start, end int) (ast.Node, error) {
	errs := &rewriteErrors{errs: make([]error, 0)}

	rewriteFunc := func(n ast.Node) bool {
		// rewrite the node ...
	}

	if len(errs.errs) == 0 {
		return node, nil
	}

	ast.Inspect(node, rewriteFunc)
	return node, errs
}
Copy the code

As you can see, we use ast.inspect () again to step through the tree for a given node. We overwrite the label for each field in the rewriteFunc function (more on that later).

Because the function passed to ast.inspect () does not return an error, we will create an error map (defined using the errs variable) and collect the error later as we step through the tree and process each individual field. Now let’s talk about the inner workings of rewriteFunc:

rewriteFunc := func(n ast.Node) bool {
	x, ok := n.(*ast.StructType)
	if! ok {return true
	}

	for _, f := range x.Fields.List {
		line := c.fset.Position(f.Pos()).Line

		if! (start <= line && line <= end) {continue
		}

		if f.Tag == nil {
			f.Tag = &ast.BasicLit{}
		}

		fieldName := ""
		if len(f.Names) ! =0 {
			fieldName = f.Names[0].Name
		}

		// anonymous field
		if f.Names == nil {
			ident, ok := f.Type.(*ast.Ident)
			if! ok {continue
			}

			fieldName = ident.Name
		}

		res, err := c.process(fieldName, f.Tag.Value)
		iferr ! =nil {
			errs.Append(fmt.Errorf("%s:%d:%d:%s",
				c.fset.Position(f.Pos()).Filename,
				c.fset.Position(f.Pos()).Line,
				c.fset.Position(f.Pos()).Column,
				err))
			continue
		}

		f.Tag.Value = res
	}

	return true
}
Copy the code

Remember that this function is called by every node in the AST tree. Therefore, we only look for nodes of type * ast.structType. Once we have it, we can start iterating through the structure fields.

Here we use the start and end variables. This defines whether we want to modify the field. If the field position is between start-end, we continue, otherwise we ignore:

if! (start <= line && line <= end) {continue // skip processing the field
}
Copy the code

Next, we check for the presence of labels. If the label field is empty (that is, nil), the label field is initialized. This helps the later cfg.process() function avoid panic:

if f.Tag == nil {
	f.Tag = &ast.BasicLit{}
}
Copy the code

Now let me explain an interesting point, and then I’ll move on. Gomodifytags attempts to get the field name of the field and process it. However, what if it is an anonymous field? :

type Bar string

type Foo struct {
	Bar //this is an anonymous field
}
Copy the code

In this case, since there is no field name, we try to get the field name from the type name:

// if there is a field name use it
fieldName := ""
if len(f.Names) ! =0 {
	fieldName = f.Names[0].Name
}

// if there is no field name, get it from type's name
if f.Names == nil {
	ident, ok := f.Type.(*ast.Ident)
	if! ok {continue
	}

	fieldName = ident.Name
}
Copy the code

Once we have the field name and label value, we can start working on the field. The cfg.process() function is responsible for processing fields with field names and label values (if any). After it returns the processing result (struct tag format in our case), we use it to overwrite the existing tag value:

res, err := c.process(fieldName, f.Tag.Value)
iferr ! =nil {
	errs.Append(fmt.Errorf("%s:%d:%d:%s",
		c.fset.Position(f.Pos()).Filename,
		c.fset.Position(f.Pos()).Line,
		c.fset.Position(f.Pos()).Column,
		err))
	continue
}

// rewrite the field with the new result,i.e: json:"foo"
f.Tag.Value = res
Copy the code

In fact, if you remember structTag, it returns a String() representation of the tag instance. Before we return to the final representation of the tag, we modify the structure as needed using various methods of the StructTag package. Here is a simple illustration:

For example, we want to extend the removeTags() function in process(). This feature creates an array of labels (key names) to delete using the following configuration:

flagRemoveTags = flag.String("remove-tags".""."Remove tags for the comma separated list of keys")

if*flagRemoveTags ! ="" {
	cfg.remove = strings.Split(*flagRemoveTags, ",")}Copy the code

In removeTags(), we check to see if we use –remove-tags. If so, we’ll use structTag’s tag.delete () method to remove the tag:

func (c *config) removeTags(tags *structtag.Tags) *structtag.Tags {
	if c.remove == nil || len(c.remove) == 0 {
		return tags
	}

	tags.Delete(c.remove...)
	return tags
}
Copy the code

The same logic applies to all functions in cfg.process ().

Now that we have a rewrite node, let’s talk about the last topic. Output and format results:

In the main function, we will call the cfg.format() function using the node overridden in the previous step:

func main(a) {
	// ... rewrite the node

	out, err := cfg.format(rewrittenNode, errs)
	iferr ! =nil {
		return err
	}

	fmt.Println(out)
}
Copy the code

One thing you need to notice is that we output to stdout. This model has many advantages. First, you can see the results simply by running the tool, and it doesn’t change anything, just for the tool user to see the results immediately. Second, stdout is composable, can be redirected anywhere, and can even be used to override the original tool.

Now let’s look at the format() function:

func (c *config) format(file ast.Node, rwErrs error) (string, error) {
	switch c.output {
	case "source":
		// return Go source code
	case "json":
		// return a custom JSON output
	default:
		return "", fmt.Errorf("unknown output mode: %s", c.output)
	}
}
Copy the code

We have two output modes.

The first (source) prints ast.node in Go format. This is the default option, and is perfect for you if you use it at the command line or just want to see changes in files.

The second option (JSON) is more advanced and is designed for other environments (especially editors). It encodes the output according to the following structure:

type output struct {
	Start  int      `json:"start"`
	End    int      `json:"end"`
	Lines  []string `json:"lines"`
	Errors []string `json:"errors,omitempty"`
}
Copy the code

The input to the tool and the final output (without any errors) are roughly as follows:

Go back to the format() function. As mentioned earlier, there are two modes. The Source pattern uses the GO /format package to format the AST as go source. The package is also used by many other official tools such as Gofmt. Here is how the Source pattern is implemented:

var buf bytes.Buffer
err := format.Node(&buf, c.fset, file)
iferr ! =nil {
	return "", err
}

if c.write {
	err = ioutil.WriteFile(c.file, buf.Bytes(), 0)
	iferr ! =nil {
		return "", err
	}
}

return buf.String(), nil
Copy the code

The format package takes IO.Writer and formats it. This is why we create an intermediate Buffer (var buf bytes.buffer) that we can use to overwrite the file when the user passes in a -write flag. After formatting, we return a string representation of the buffer containing the formatted Go source code.

The JSON schema is more interesting. Since we are returning a piece of source code, we need to render it exactly as it was originally formatted, which also means including comments. The problem is that when printing individual constructs using form.node (), if they are lossy, the Go annotation cannot be printed.

What is a Lossy comment? Consider this example:

type example struct {
	foo int 

	// this is a lossy comment

	bar int 
}
Copy the code

Each Field is of type * ast.field. This structure has an *ast.Field.Comment field that contains comments for a field.

But, in the example above, who does it belong to? Foo or bar?

Because it is impossible to determine, these comments are called lossy comments. If you now print the above structure using the form.node () function, you will have a problem. When you print it, you may get (play.golang.org/p/peHsswF4J…). :

type example struct {
	foo int

	bar int
}
Copy the code

The problem is that lossy comments are part of the * ast.file, which is separate from the tree. Print only if you print the entire file. So the solution is to print the entire file and then delete the specified line we want to return in the JSON output:

var buf bytes.Buffer
err := format.Node(&buf, c.fset, file)
iferr ! =nil {
	return "", err
}

var lines []string
scanner := bufio.NewScanner(bytes.NewBufferString(buf.String()))
for scanner.Scan() {
	lines = append(lines, scanner.Text())
}

if c.start > len(lines) {
	return "", errors.New("line selection is invalid")
}

out := &output{
	Start: c.start,
	End:   c.end,
	Lines: lines[c.start- 1 : c.end], // cut out lines
}

o, err := json.MarshalIndent(out, ""."")
iferr ! =nil {
	return "", err
}

return string(o), nil
Copy the code

This ensures that we can print all comments.

That’s all!

We successfully completed our tool and here is the complete step diagram we implemented throughout the guide:

To review what we did:

  • We retrieve the configuration through the CLI flag
  • We’re throughgo/parserPackage parsing file to get oneast.Node.
  • After parsing the file, we search for the appropriate structure to get the start and end positions so we know which fields need to be modified
  • Once we have the start position and the end position, we iterate againast.NodeOverride each field between the start and end positions (by usingstructtagPackage)
  • After that, we’ll format the overwritten node and output either the Go source code or custom JSON for the editor

Since creating the tool, I’ve received a lot of friendly comments about how the tool simplifies their daily tasks. As you can see, although it looks easy to make, we’ve made it special for many special cases throughout the guide.

Gomodifytags has been successfully used in the following editors and plug-ins for several months, enabling thousands of developers to improve their productivity:

  • vim-go
  • atom
  • vscode
  • acme

If you are interested in the original source code, it can be found here:

  • Github.com/fatih/gomod…

I also gave a talk at Gophercon 2017. If you’re interested, check out the YouTube address below:

www.youtube.com/embed/T4AIQ…

Thank you for reading this article. Hopefully this guide will inspire you to create a new Go tool from scratch.