background

Recently in doing a PROto file processing CLI tool, before using proto file, generally divided into two ways:

  1. Direct reference to proto file, using the runtime dynamic generation OF JS code
  2. The protoc tool generates the corresponding JS file and references it in the project

The latter is usually used for higher performance because the compilation process is before the program runs.

Phenomenon of the problem

The proto file is also dynamic because it is a general-purpose tool. It simply simulates the possible scenarios in the local environment, and then the terminal executes the protoc command:

#Grpc_tools_node_protoc is the encapsulation of the protoc node. js version
grpc_tools_node_protoc --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto
Copy the code

Found that everything is running normally, then write the corresponding code into the script, replace part of the path as variables, submit the code, send the package, local installation, verification.

The result is this:

Could not make proto path relative: ./protos/**/*.proto: No such file or directory /usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc.js:43 throw error; ^ Error: Command failed: /usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto Could not  make proto path relative: ./protos/**/*.proto: No such file or directory at ChildProcess.exithandler (child_process.js:303:12) at ChildProcess.emit (events.js:315:20) at maybeClose (internal/child_process.js:1021:16) at Socket.<anonymous> (internal/child_process.js:443:11) at Socket.emit (events.js:315:20) at Pipe.<anonymous> (net.js:674:12) { killed: false, code: 1, signal: null, cmd: '/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin --js_out=import_style=commonjs,binary:./src/main/proto --grpc_out=grpc_js:./src/main/proto ./protos/**/*.proto' }Copy the code

Shockingly, and even more surprisingly, when I copied the contents of CMD into the terminal and ran it again, everything was fine.

Shocked, I rechecked my code implementation.

Troubleshoot problems

Grpc_tools_node_protoc is also a wrapped Node.js module, so I took a look at its source code and found that the source code is execFile. Error: “No such file or directory” : No such file or directory “: No such file or directory” : No such file or directory “: No such file or directory” : No such file or directory “: No such file or directory” : No such file or directory “: No such file or directory” : No such file or directory” Therefore, I took a look at the definition of current working directory between the two apis and found a few differences:

Current working directory of the child process. Default: Process.cwd ()., and the CWD parameter for execFile is described as Current working directory of the child process.

It doesn’t look like the latter has a default value, so maybe it’s the wrong working directory, so we add the CWD parameter to the code and re-validate the process.

As a result, there is no difference, still an error.

So I looked at the difference between the implementation of exec and execFile API in Node.js to see if it is CWD, and found that exec calls execFile internally. We can basically confirm that there is no difference in the CWD parameter default value processing, and also add DEBUG message output in the source code to check that CWD is actually running in the directory we expect.

Since the problem is not here, we need to analyze it elsewhere. Since we are more confident in our code (which is not a few lines), we took a closer look at the implementation of grPC-Tools and found that the code looks like this:

var protoc = path.resolve(__dirname, 'protoc' + exe_ext);

var plugin = path.resolve(__dirname, 'grpc_node_plugin' + exe_ext);

var args = ['--plugin=protoc-gen-grpc=' + plugin].concat(process.argv.slice(2));

var child_process = execFile(protoc, args, function(error, stdout, stderr) {
  if (error) {
    throwerror; }});Copy the code

The CMD argument is the result of the args argument. Out of curiosity, we added a DEBUG log to the source code and found an amazing situation.

When we run through Node.js exec, the output looks like this:

[
  '/usr/local/bin/node'.'/usr/local/bin/grpc_tools_node_protoc'.'--js_out=import_style=commonjs,binary:./src/main/proto'.'--grpc_out=grpc_js:./src/main/proto'.'./protos/**/*.proto'
]
Copy the code

When we run the command directly from the terminal, the output looks like this:

[
  '--plugin=protoc-gen-grpc=/usr/local/lib/node_modules/@infra-node/grpc-tools/bin/grpc_node_plugin'.'--js_out=import_style=commonjs,binary:./src/main/proto'.'--grpc_out=grpc_js:/./src/main/proto'.'./protos/examples/example-base-protos/kuaishou/base/base_message.proto'
]
Copy the code

The last parameter turns out to be different.

Therefore, I tried to put proto’s detailed file path into the command and run it through exec again, and found that everything was normal. Therefore, the problem is the last proto file path, since protoc does not support ** such a wildcard file input. The new question arises as to why the two different modes of operation cause the parameters passed in to change.

Since node.js module executables are registered through package bin, it is reasonable to suspect that NPM has done something slightly wrong, so it writes a shell file with a simple output:

Echo $* # prints all parametersCopy the code

If we can get **/*. Json from sh test.sh **/*. Json, then we are almost sure that NPM is doing something wrong.

Results The output result is:

package-lock.json package.json proto.json
Copy the code

Output through the terminal will already be able to get a complete file path, indicating at least some operations not NPM.

One possibility occurred to us. Type bash and run the same command sh test.sh **/*. Json, and sure enough we got **/*.

Think of oneself of the terminal is used ZSH, so through the corresponding documents, sure enough to find the corresponding description: ZSH. Sourceforge. IO/Doc/Release… When I first realized what the problem was, I thought oh my F **king ZSH.

ZSH recursively matches the path and expands it in the execution parameters, so eventually the cause was also located, because a convenience feature of ZSH caused me to mistake it for a protoc feature and ended up exposing the problem in a non-ZSH environment.

conclusion

This encounter problem phenomenon is very strange, but the reason is very helpless, fortunately in the process of investigation or more harvest, forced to read some module source code, a deeper understanding of the whole compilation process of proto file. After getting used to using ZSH, I mistook some of its capabilities for those provided by the program. I didn’t think that way during the troubleshooting process and wondered if such a “good” tool would surprise me in other scenarios.