background

Recently, I am working on a CLI tool for Proto file processing. Before I used Proto file, it is generally divided into two ways:

  1. Direct reference to the Proto file, using runtime dynamic generation of JS code
  2. Use the Protoc tool to generate the corresponding JS file and reference it in your project

The latter is better because the compilation process takes place before the program is run, so the latter is usually used.

Phenomenon of the problem

Protoc is a generic tool, so the proto file is also dynamic. It simply emulates a possible scene in the local environment, and then executes the command protoc:

# grpc_tools_node_protoc encapsulates grpc_tools_node_protoc for the protoc Node.js version --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto

Findings that everything is working well, I write the corresponding code into the script, replace part of the path as a variable, submit the code, package, install locally, verify.

The result is this:

Could not make proto path relative: ./protos/**/*.proto: No such file or directory /usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc.js:43 throw error; ^ Error: Command failed: /usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto Could not  make proto path relative: ./protos/**/*.proto: No such file or directory at ChildProcess.exithandler (child_process.js:303:12) at ChildProcess.emit (events.js:315:20) at maybeClose (internal/child_process.js:1021:16) at Socket.<anonymous> (internal/child_process.js:443:11) at Socket.emit (events.js:315:20) at Pipe.<anonymous> (net.js:674:12) { killed: false, code: 1, signal: null, cmd: '/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto' }

Shockingly, and even more surprising, when I copied the CMD into the terminal and ran it again, everything was fine.

Shocked, I rechecked my code implementation.

Troubleshoot problems

Because GRPC_TOOLS_NODE_PROTOC is also a encapsulated Node.js module, so I look at its source code and find that execFile is used. Then check the Node.js documentation to see if there is any difference between the two. Because the previous error message is “No such file or directory”, first of all, I wonder if the path is wrong because CLI is a global installation. Therefore, after a targeted look at the definition of current working directory of the two APIs, I found a few differences:

Current working directory of the child process. Default: Current working directory of the child process. Default: Process.cwd ()., and execFile’s CWD parameter is described as Current working directory of the child process..

It seems that there is no default value for the latter, so is it because the working directory is not correct? So we added the CWD parameter in the code and restarted the validation process.

As a result, there is no difference, it is still an error.

Exec calls execFile from within the Node.js API to see if it is CWD. Execfile calls execFile from within the Node.js API. There is no difference between the default value of CWD parameter and the default value of CWD parameter. At the same time, the DEBUG message output is added to the source code to check that CWD is indeed the directory where we expect to run at the moment.

Since the problem is not here, we need to analyze it from other places. Since we are confident in our own code (which is really not a few lines), we took a closer look at the implementation of GRPC-Tools and found that the code looks like this:

var protoc = path.resolve(__dirname, 'protoc' + exe_ext); var plugin = path.resolve(__dirname, 'grpc_node_plugin' + exe_ext); var args = ['--plugin=protoc-gen-grpc=' + plugin].concat(process.argv.slice(2)); var child_process = execFile(protoc, args, function(error, stdout, stderr) { if (error) { throw error; }});

The above program error output CMD parameter is actually the result of the args parameter here. Out of curiosity, we added a DEBUG log to the source code, and discovered a curious situation.

When we run it through Node.js exec, the output looks like this:

[
  '/usr/local/bin/node',
  '/usr/local/bin/grpc_tools_node_protoc',
  '--js_out=import_style=commonjs,binary:./src/main/proto',
  '--grpc_out=grpc_js:./src/main/proto',
  './protos/**/*.proto'
]

If we execute the command directly from the terminal, the output looks like this:

[
  '--plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin',
  '--js_out=import_style=commonjs,binary:./src/main/proto',
  '--grpc_out=grpc_js:/./src/main/proto',
  './protos/examples/example-base-protos/kuaishou/base/base_message.proto'
]

The last parameter of the two turns out to be different.

Protoc does not support ** for wildcard file input. If the proto file path is not supported by **, the proto file path is not supported. A new question arises, then, as to why two different modes of operation would cause the parameters passed in to change.

Since the executables of Node.js modules are registered via Package Bin, it is reasonable to wonder if NPM is doing something small, so I wrote a shell file with a simple output:

Echo $* # outputs all parameters

Json, then it is almost certain that NPM is responsible for this.

The output result is as follows:

package-lock.json package.json proto.json

By doing this through a terminal, you can get a full file path, indicating at least some operations that are not NPM.

It was possible to type bash and run the same command sh test.sh **/*. Json, and sure enough we got **/*.

Think of oneself of the terminal is used ZSH, so through the corresponding documents, sure enough to find the corresponding description: https://zsh.sourceforge.io/Do… “, [turning to 14.8.6 Recursive Globbing] When I first realized what the problem was, a line of oh my f**king ZSH floated through my mind.

Zsh will recursively match the path and then expand it in the execution parameters, so the ultimate cause is also identified as a convenience feature of Zsh that I mistakenly thought was a feature of Protoc and ended up exposing the problem in a non-Zsh environment.

conclusion

The problem encountered this time is very strange phenomenon, but the reason is very helpless, fortunately in the process of screening or more harvest, forced to read some module source code, a deeper understanding of the entire compilation process of proto files. After I got used to using Zsh, some of the capabilities it provided made me think they were provided by the program, and I didn’t think about them during the whole troubleshooting process, and I didn’t know if the “good” tool would surprise me in other scenarios.