This introduction to THE JS Abstract syntax tree (AST) ushered me into a new world

Github/TSCCSS, star🌟 I blog address: github/blog, if this article is helpful to you, reward a star🌟, thank you master!

An opportunity to

Recently, when I was building an open source project environment, I needed to type an ES module package, so that developers could install and use it directly through NPM. However, the project was destined to have style, and I wanted to type the package in the same directory as my development directory, so Rollup seemed to be a good choice. But I (masochistically) chose TSC, the compiler that comes with Typescript, and I’m on my way to filling in the holes

TSC encountered pits

There are three basic pits for me so far when compiling my code using TSC, and I’ll cover them briefly below, before looking at the directory structure that will be compiled.

|-- src
  |-- assets
    |-- test.png
  |-- util
    |-- classnames.ts
  |-- index.tsx
  |-- index.scss
Copy the code

Simplify the reference path problem

First I wrote the simplified reference path configuration in tsconfig.json. For example, for the above directory, I did this:

{
  "compilerOptions": {
    "baseUrl": ". /"."paths": {
      "@Src/*": ["src/*"]."@Utils/*": ["src/utils/*"]."@Assets/*": ["src/assets/*"]}}}Copy the code

No matter how deep I go, it’s especially handy if I want to refer to util or assets, such as in index.tsx:

Compile the front:

import classNames from "@Utils/classnames";
import testPNG from "@Assets/test.png";
Copy the code

After compilation (expected 😢) :

import classNames from "./util/classnames";
import testPNG from "./assets/test.png";
Copy the code

However, the result of the actual compilation made me disappointed, since TSC does not even support translation!! So I went to the official website and found that there was no related configuration item, so I went to the external website and found that someone had the same problem as me. It provided a solution: use the plugin tscPaths and add a NPM command after the compilation:

"scripts": {
  "build": "tsc -p tsconfig.json && tscpaths -p tsconfig.json -s src -o dist,
},
Copy the code

When executing this command:

tscpaths -p tsconfig.json -s src -o dist
Copy the code

The plugin will iterate through every.js file we’ve compiled from TSC, converting our simplified reference paths to relative paths, and we’re done

Static resources are not packaged

As shown above, if I add an image to assets in the index. TSX file:

import testPNG from "@Assets/test.png";
Copy the code

After compiling TSC and using our command line tools, we reference the correct path, but when we look at the package directory, we do not see assets in the folder. This is normal because TSC is just a Typescript compiler. To implement other packaging features, do it yourself!

The solution is to use the CopyFiles command line tool, which, like the plugins we introduced above, does some extra work after TSC is compiled to do what we want.

As its name suggests, it is used to copy files. We add this to the build command under NPM scripts:

copyfiles -f src/assets/* dist/assets
Copy the code

This will copy the resource folder to the packaged file directory.

Introduce style file name extensions

When we do a project, it is inevitable to use sass or less. For this project, WE chose Sass. I introduced the style file in index. TSX as follows:

import "./index.scss";
Copy the code

But after compiling TSC to a.js file, open index.js and find that the style suffix introduced is still.scss. As a package for other developers, it must have a.css file format, you can’t be sure that everyone else is using Sass, so I went to the Internet to find a solution, and found very few people to mention this problem, and there are no plug-ins to use.

When I was at a loss, I suddenly thought, oh my God, this is a tool similar to tSCPaths, which also does string substitution in files. So I quickly download its source code, look at the next is probably using node to read the tsconfig.json bathUrl and Paths configuration, as well as user defined entry path, exit path to find.js file, analysis into a relative path after the regular match to the corresponding reference path to replace!

When you’re ready to practice, you suddenly realize the limitations of global re matching. For example, if you write the same code as the reference in the developer code (this situation is almost impossible, but still should be considered), you will change the logic code of others. For example:

import React from "react";
import "./index.scss";

const Tool = () = > {
  return (
    <div>
      <p>You should import style file like this:</p>
      <p>import './index.scss'</p>
    </div>
  );
};
Copy the code

How to do, you do global replacement, will replace someone else’s logic source code. Of course, you could write a better lookup algorithm (or regex) to make the exact substitution, but there are a lot more cases to consider implicitly; Do we have a better way to do that? This is when I thought about abstract Syntax trees (AST).

Note ⚠️ : TSC also does not compile.scss files. It requires Node-sass to compile each.scss file into the appropriate package directory. After TSC is compiled, execute the following command:

"build-css": "node-sass -r src -o dist".Copy the code

What is AST?

If you know or use tools like ESLint, Babel and Webpack, you already have a good idea of the power of AST. How does ESLint fix your code? Take a look at the less rigorous picture below:

A loose language description would be that esLint parses the current JS code into an abstract syntax tree and makes some modifications to the tree, such as cutting off a branch to remove the extra space in the code; Like trimming a tree branch, var is converted to const, etc. After finishing the conversion to our JS code!

Each “branch” in the tree represents a description object for a field in JS code, such as the following simple code:

const a = 1;
Copy the code

Let’s start by customizing a simple set of rules for converting to AST syntax, which can represent the above line of code:

{
  "type": "VariableDeclaration"."kind": "const"."declarations": [{"type": "VariableDeclarator"."id": {
        "type": "Identifier"."name": "a"
      },
      "init": {
        "type": "Literal"."value": 1."raw": "1"}}}]Copy the code

If we had a way to take a node representing the value 1, change it to init.value 2, and convert the syntax tree to JS source code, we would get:

const a = 2;
Copy the code

With the development of JavaScript language, ESTree, a project created by some big names to update AST rules, has become a community standard. Then other projects in the community like ESlint and Babel would use ESTree or make modifications to it, and then spawn their own set of rules, and create conversion tools to expose apis for developers to use.

With tools

Because the generated AST structure can seem cumbersome, it can be confusing to learn or write code without good tools or documentation, so I’ll introduce you to three powerful tools.

Online debugging toolAST Explorer

This is a great site, just type in your current JS code to see the transformed AST structure.

With this site you can see in real time what the parsed AST looks like and what their types are, which is especially useful when writing code to modify the AST later! Because you know exactly what you want to change.

For example, in the figure above, we want to change 1 to 2. We use a tool to find the node whose type is Literal in the AST, set its value to 2, and then convert it into JS code to achieve this requirement.

There are plenty of alternatives, but we’ll use Facebook’s official open source tool: Jscodeshift

AST conversion tooljscodeshift

Jscodeshift is a library based on the RECast wrapper, which encapsulates and expose apis that are friendlier to JS developers than Recast’s unfriendly API design, making it easier to modify the AST.

I suggest you to know this tool first, the specific API use I will pick a few typical to tell you, have a specific impression on the line, to tell the truth, this library documentation is not good, also not suitable for beginners to read, especially English is not good. It’s not too late to read it when you’ve used some of its apis and have a feel for it

AST Types@babel/types

This is an AST type dictionary. If we want to generate some new code, that is, to generate some new nodes, the syntax dictates that you pass in the type of node you want to add as specified. For example, const is type: VariableDeclaration, of course, type is just an attribute of a node, and there are other things you can check in here.

@babel/types:

Type the name	The Chinese translation	describe
Program	application	The body of the entire code
VariableDeclaration	Variable declarations	Declare variables, such as let const var
FunctionDeclaration	Function declaration	Declare functions, such as function
ExpressionStatement	Expression statement	Usually by calling a function, such as console.log(1)
BlockStatement	Block statements	Statements wrapped inside {}, such as if (true) {console.log(1)}
BreakStatement	Break statement	Usually refers to break
ContinueStatement	Last statement	Usually refers to the continue
ReturnStatement	Return statement	Usually refers to the return
SwitchStatement	A Switch statement	Usually refers to the switch
IfStatement	If control flow statement	If (true) {} else {}
Identifier	identifier	Identifies, for example, a in const a = 1 in a declaration variable statement
ArrayExpression	Array expression	Usually an array, such as [1, 2, 3]
StringLiteral	Character literals	Usually a literal of type string, such as the ‘1’ of const a = ‘1’
NumericLiteral	Numeric literals	Usually a literal of an exponential type, such as const a = 1 in 1
ImportDeclaration	Introduce statement	Declaration imports, such as import

Add, delete, modify, and query AST nodes

Jscodeshift is a simple development environment. You can either add, delete, change, or query a tree from your site.

The development environment

Step 1: Create a project folder

mkdir ast-demo
cd ast-demo
Copy the code

Step 2: Project initialization

npm init -y
Copy the code

Step 3: Install jscodeshift

npm install jscodeshift --save
Copy the code

Step 4: Create 4 JS files to add or delete the query.

touch create.js delete.js update.js find.js
Copy the code

Step 5: In the following example, open the AST Explorer and copy the values you want to convert to see the tree structure for better understanding.

Find nodes

The find. Js:

const jf = require("jscodeshift");

const value = `
import React from 'react';
import { Button } from 'antd';
`;

const root = jf(value);
root
  .find(jf.ImportDeclaration, { source: { value: "antd" } })
  .forEach((path) = > {
    console.log(path.node.source.value);
  });
Copy the code

Execute the following command on the console:

node find.js
Copy the code

Then you can see that the console prints ANTD.

Just to clarify, the value string defined in the above code is the text content that we are manipulating. In practice, we usually read the file and do the processing.

In the.find function above, the first argument is the type to look for and the second argument is the query condition. If you copy the value above into the AST Explorer, you can see why the query condition is structured this way.

Modify the node

Update. Js:

const jf = require("jscodeshift");

const value = `
import React from 'react';
import { Button, Input } from 'antd';
`;

const root = jf(value);
root
  .find(jf.ImportDeclaration, { source: { value: "antd" } })
  .forEach((path) = > {
    const { specifiers } = path.node;
    specifiers.forEach((spec) = > {
      if (spec.imported.name === "Button") {
        spec.imported.name = "Select"; }}); });console.log(root.toSource());
Copy the code

The purpose of the above code is to change the imported Button from ANTD to Input. In order to locate the row accurately, we first use the ImportDeclaration and condition parameters to find the Input node. After finding the Button node, we can make changes after a simple judgment.

You can see that on the last line we execute toSource(), which simply turns the AST back to our source code. The console prints the following:

import React from "react";
import { Select, Input } from "antd"; // You can see that Button has been accurately replaced with Select
Copy the code

Add node

The create. Js:

const jf = require("jscodeshift");

const value = `
import React from 'react';
import { Button, Input } from 'antd';
`;

const root = jf(value);
root
  .find(jf.ImportDeclaration, { source: { value: "antd" } })
  .forEach((path) = > {
    const { specifiers } = path.node;
    specifiers.push(jf.importSpecifier(jf.identifier("Select")));
  });

console.log(root.toSource());
Copy the code

The above code first finds the antD line again, and then adds a new node to the last bit of the speciFIERS array, which is represented in the converted JS code by the introduction of a Select:

import React from "react";
import { Button, Input, Select } from "antd"; // You can see that Select is introduced
Copy the code

Remove nodes

Delete. Js:

const jf = require("jscodeshift");

const value = `
import React from 'react';
import { Button, Input } from 'antd';
`;

const root = jf(value);
root
  .find(jf.ImportDeclaration, { source: { value: "antd" } })
  .forEach((path) = > {
    jf(path).replaceWith("");
  });

console.log(root.toSource());
Copy the code

Delete the entire line introducing ANTD, that’s it.

More API

Add, delete, modify, and check all of these are just one of many ways to implement them. As long as you are familiar with the API and have a good imagination, there is no way to stop it. I just want to say that you can go to the official Collection and Extensions to find out what apis there are. It will always get what you want.

Actual combat parsing

Technology serves demand.

Clear requirements

With a basic understanding of jscodeshift, we’ll do a command line tool to solve the “introducing style file suffixes” I mentioned above, and then simply use Commander, which makes the NodeJS command line interface much simpler

Let me clarify my current requirements again: for TSC compiled directories, such as dist, I want to convert all js files generated from the introduction of style files, such as import ‘./style.scss’, to.css suffixes.

The command line tool IS called TSCCSS.

Set up the environment

As above, we initialize the project first, so we don’t use Typescript for demonstration purposes. Instead, we write native NodeJS as native modules. If the project is more specific, we can add ESLint, Prettier, etc. Go to the command line tool TSCCSS that I’ve already written on Github for a reference.

Ok, now let’s do it in one go. Here are the steps:

Create project directory
mkdir tsccss
cd tsccss

# initialization
npm init -y

Install dependencies
npm i commander globby jscodeshift --save

Create an entry file
mkdir src
cd src
touch index.js
Copy the code

The current directory is as follows:

|-- node_modules
|-- src
  |-- index.js
|-- package.json
Copy the code

Next add the following code somewhere in package.json:

{
  "main": "src/index.js"."bin": {
    "tsccss": "src/index.js"
  },
  "files": ["src"]}Copy the code

SRC /index.js SRC /index.js SRC /index.js SRC /index.js SRC /index.js SRC /index.js

#! /usr/bin/env node
Copy the code

This code solves the problem of different user node paths, allowing the system to dynamically find node to execute your script files.

Using the commander

Add the following code directly to index.js:

const { program } = require("commander");

program.version("0.0.1").option("-o, --out <path>"."output root path");

program.on("--help".() = > {
  console.log(` You can add the following commands to npm scripts: ------------------------------------------------------ "compile": "tsccss -o dist" ------------------------------------------------------ `);
});

program.parse(process.argv);

const { out } = program.opts();
console.log(out);

if(! out) {throw new Error("--out must be specified");
}
Copy the code

Next, in the project root directory, execute the following console command:

node src/index.js -o dist
Copy the code

You’ll notice that the console prints dist. Yeah, that’s what -o dist does. A little bit about Version and options.

• version

Function: Defines the version number of a command program. Usage examples:.version(‘0.0.1’, ‘-v, –version’); Parameter analysis:

1. The first argument, the version number < must >;
2. The second argument, the custom flag < omitted >, defaults to -v and –version.

• option

Function: Defines command options. Usage examples:.option(‘-n, –name ‘, ‘edit your name’, ‘vortesnail’); Parameter analysis:

1. The first parameter, the custom flag < must >, is divided into long and short identifiers separated by commas, vertical lines, or Spaces. (Flags can be followed by parameters, which can be modified with <> or [], the former meaning required parameter, the latter meaning optional parameter)
2. The second argument, the option description < omission without error >, displays the flag description when using the –help command;
3. The third parameter, option parameter default value, optional.

So you can also try these two commands:

node src/index.js --version
node src/index.js --help
Copy the code

Read js file under dist

Dist directory is the root of the file where we assume we are going to do style file suffixes. Now we need to use the globby tool to automatically read all js file paths in this directory. We need to introduce two functions at the top:

const { resolve } = require("path");
const { sync } = require("globby");
Copy the code

Then append the code below:

const outRoot = resolve(process.cwd(), out);

console.log(`tsccss --out ${outRoot}`);

// Read output files
const files = sync(`${outRoot}/ * * /! (*.d).{ts,tsx,js,jsx}`, {
  dot: true,
}).map((x) = > resolve(x));
console.log(files);
Copy the code

Js -o dist to see if the console correctly prints out the absolute path of these files.

Write alternative methods

Because of the foreshadowing of the previous increase, deletion, change and check, in fact, now this step is very simple, the idea is:

• Find all types forImportDeclarationThe node;
• Using the re to determine the nodesource.valueWhether or not to.scss  或 .lessAt the end.
• If the re is matched, we use some usage of re to replace the suffix with.css 。

That’s it, we’ll go straight to jscodeshift:

const jscodeshift = require("jscodeshift");
Copy the code

Then append the following code:

function transToCSS(str) {
  const jf = jscodeshift;
  const root = jf(str);
  root.find(jf.ImportDeclaration).forEach((path) = > {
    let value = "";
    if (path && path.node && path.node.source) {
      value = path.node.source.value;
    }
    const regex = /(scss|less)('|"|`)? $/i;
    if (value && regex.test(value.toString())) {
      path.node.source.value = value
        .toString()
        .replace(regex, (_res, _$1, $2) = > ($2 ? `cssThe ${$2}` : "css")); }});return root.toSource();
}
Copy the code

As you can see, this method returns the converted JS code directly, which can be written directly to the source file.

Read and write files

After getting the file path files, we need the node native module FS to help us read and write the file. This part of the code is very simple, the idea is: read the JS file, convert the file content to AST for node value replacement, then convert the JS code, and finally write back to the file, and then it is OK.

const { readFileSync, writeFileSync } = require("fs");

// ...

const filesLen = files.length;
for (let i = 0; i < filesLen; i += 1) {
  const file = files[i];
  const content = readFileSync(file, "utf-8");
  const resContent = transToCSS(content);
  writeFileSync(file, resContent, "utf8");
}
Copy the code

Now go to the dist index1.js and index2.js files and type the following to see the effect:

import "style.scss";
import "style.less";
import "style.css";
Copy the code

And then do our bidding one last time:

node src/index.js -o dist
Copy the code

Index1. js and index2.js are all replaced correctly:

import "style.css";
import "style.css";
import "style.css";
Copy the code

Comfortable ~ 😊

The above code can be optimized in many places, such as you can also write some additional code to replace the position of the statistics, number, number of file modification, etc., all of these can be printed in the console, when people use also can get good feedback – or even replace the regular method also can be modified to do, see you!

Last thing I want to say

Although the actual combat a AST usage is very simple, but the main role of this article is to take everybody introduction, use this kind of thinking to solve some problems in the work or study, in my opinion, there are on a method of cognition, the way you solve the problem of aeriform in more than a. In fact, technology is not the most important to some extent, what is important is the cognition of technology.

After all, if you don’t know something, the idea of using it won’t happen, but you know that no matter how difficult the technology is, it can always be overcome!

Finally, thank you for reading this carefully. If there are any mistakes in the article, welcome to discuss them.

Github/TSCCSS, star🌟 I blog address: github/blog, if this article is helpful to you, reward a star🌟, thank you master!

Commander Play between hift and jscodeshift