The “lead” TensorFlow Serving provides the GRPC interface to efficiently fulfill prediction requests to the model, but it only provides the Python-based API itself, so if you are using another language for GRPC access, you need to manually generate the corresponding GRPC interface file. This article mainly describes how to generate TensorFlow Serving API files using the Protoc tool and provides a complete project example for your reference.
ProtoBuf file compilation
TensorFlow serves GPRC communication based on the Protocol Buffer. The source file with the proto suffix defines a series of data structures (messages) and RPC services. They represent, respectively, the format for data exchange and the interface for performing remote operations. Proto files themselves cannot be used directly in code; they need to be further converted into language-specific code files to compile and run properly.
The Protocol Buffer Compiler (Protoc) Compiler is used to compile proto files and generate language-specific code files. The Protocol Buffer Compiler currently supports code generation for many languages, including Golang, Java, c++, c#, etc. Using protoc tools can greatly reduce our coding effort, allowing us to focus more on specific business implementations rather than having to define various language-specific data structures.
Therefore, when communicating GRPC with TensorFlow Serving in other languages, we need to use the Protoc tool to generate language-specific API files for subsequent use. Note that since some of the Proto files in TensorFlow Serving source are dependent on the proto files in TensorFlow, we need to use both source codes to generate the required API files.
The following is a brief introduction to the installation process of the Protoc tool in Linux:
-
Download the latest protoc binary package from Github of the Protocol Buffer, or run the following command to download the package.
Wget HTTP: / / https://github.com/protocolbuffers/protobuf/releases/download/v3.12.3/protoc-3.12.3-linux-x86_64.zipCopy the code
-
Then decompress the package to the /usr/local/protoc directory.
Unzip protoc - 3.12.3 - Linux - x86_64.zip -d/usr/local/protocCopy the code
-
Add the /usr/local/protoc/bin directory to the PATH environment variable. You can accomplish this by adding the following statement to the /etc/profile file.
export PATH=$PATH:/usr/local/protoc/bin Copy the code
-
Finally, the test installation was successful
protoc --version Copy the code
API file generation and use
In general, TensorFlow Serving API files can be generated for any language supported by the Protocol Buffer and GRPC. In the previous article IN TensorFlow 2.x model Serving I introduced an example of GPRC requests using Python, This article focuses on generating TensorFlow Serving API files and GPRC requests using Golang and Java.
To generate the executable code file, we first need to clone the source code for both TensorFlow and TensorFlow Serving locally.
mkdir tensorflow-serving-api && cd tensorflow-serving-api
git clone https://github.com/tensorflow/tensorflow.git
git clone https://github.com/tensorflow/serving.git
Copy the code
You can then use the proto file in the source code to generate the TensorFlow Serving API file for the corresponding language.
Golang
To generate the Golang-related API code files, we need to install the Golang environment and some Protoc plug-ins to assist us in file generation. The following operations are performed on the Linux operating system.
-
Install Golang as follows.
Wget https://dl.google.com/go/go1.14.4.linux-amd64.tar.gz tar ZXVF go1.14.4. Linux - amd64. Tar. Gz - C/usr/local export PATH=$PATH:/usr/local/go/bin go versionCopy the code
-
Protoc-gen-go plug-in is installed to generate go files.
go get -u google.golang.org/protobuf/cmd/protoc-gen-go # or go get -u github.com/golang/protobuf/protoc-gen-go Copy the code
Protoc -gen-go will be installed in $GOPATH/bin by default, make sure that directory is under PATH so that the Protoc tool can find the plugin.
-
Install the GRPC plug-in to generate GRPC Go files.
go get -u google.golang.org/grpc Copy the code
-
Switches source code to the specified branch or label.
CD tensorflow-serving- API /tensorflow git checkout tags/v2.2.0 CD Tensorflow -serving- API/Serving Git checkout tags/v2.2.0Copy the code
-
Use the Protoc tool to generate go files.
cd tensorflow-serving-api protoc -I=serving -I=tensorflow --go_out=plugins=grpc:golang serving/tensorflow_serving/*/*.proto protoc -I=serving -I=tensorflow --go_out=plugins=grpc:golang serving/tensorflow_serving/sources/storage_path/*.proto protoc -I=serving -I=tensorflow --go_out=plugins=grpc:golang tensorflow/tensorflow/core/framework/*.proto protoc -I=serving -I=tensorflow --go_out=plugins=grpc:golang tensorflow/tensorflow/core/example/*.proto protoc -I=serving -I=tensorflow --go_out=plugins=grpc:golang tensorflow/tensorflow/core/protobuf/*.proto protoc -I=serving -I=tensorflow --go_out=plugins=grpc:golang tensorflow/tensorflow/stream_executor/*.proto Copy the code
-i specifies the path for proto files to search for dependent files. The value can be specified multiple times. –go_out specifies the directory to save the go file (in this case, golang) and the GRPC plug-in to use. The last line of the command specifies, in the form of a wildcard, the input location of the proto file.
As for why the above proTO file is chosen for API generation, it depends on the actual use and the dependency between proTO files. Starting with the source code of Proto and referring to the code implementation of the Python GRPC sample serving, find the proto file of the portal, generate the API code file based on the proTO file itself and the dependent proTO file, and then test the coding to make up for any missing points. Until all code files compile correctly.
-
After executing the above command, two directories are generated in the golang directory, github.com and tensorflow_SERVING. The former contains the GO file generated from the proto file in the Tensorflow source. The latter contains the GO file generated from the Proto file in serving source. Because the proto files in tensorflow source code all contain go_package options such as option go_package = “github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_go_proto”; They specify the output directory of the generated go file, so the generated GO file is in the github.com directory. The Proto file in serving source does not contain this option, so the output directory of the go file defaults to the same directory as the source file.
-
A circular reference error may occur in the go file generated under the tensorFLOW_SERVING directory, as follows:
import cycle not allowed package github.com/alex/tensorflow-serving-api-go imports tensorflow_serving/apis imports tensorflow_serving/core imports tensorflow_serving/apis Copy the code
Go file in tensorflow_serving/core and prediction_log.pb.go file in tensorflow_serving/apis to solve this problem. Deleting the above code file does not affect subsequent GRPC model prediction requests.
-
Suppose I have a model called first_model deployed on TensorFlow Serving, and its metadata information looks like this:
$ curl http://localhost:8501/v1/models/first_model/versions/0/metadata { "model_spec": { "name": "first_model", "signature_name": "", "version": "0" }, "metadata": { "signature_def": { "signature_def": { "serving_default": { "inputs": { "input_1": { "dtype": "DT_INT64", "tensor_shape": { "dim": [ { "size": "-1", "name": "" }, { "size": "31", "name": "" } ], "unknown_rank": false }, "name": "serving_default_input_1:0" } }, "outputs": { "output_1": { "dtype": "DT_FLOAT", "tensor_shape": { "dim": [ { "size": "-1", "name": "" }, { "size": "1", "name": "" } ], "unknown_rank": false }, "name": "StatefulPartitionedCall:0" } }, "method_name": "tensorflow/serving/predict" }, "__saved_model_init_op": { "inputs": {}, "outputs": { "__saved_model_init_op": { "dtype": "DT_INVALID", "tensor_shape": { "dim": [], "unknown_rank": true }, "name": "NoOp" } }, "method_name": "" } } } } } Copy the code
Inputs we need to focus on the inputs above, which define the key value (input_1 in this case), the dimension of the input ((-1, 31)), and the type of the input (DT_INT64) for this model. When GRPC predictive requests are made, the input data specified in the code needs to match the various input information defined in the metadata, otherwise the correct model output cannot be obtained.
-
Create the GO project and send the GRPC prediction request to first_model. For details of the project, please refer to Github’s implementation and instructions. Only the main function code is listed here, as follows:
package main import ( "context" "log" apis "tensorflow_serving/apis" "time" "github.com/golang/protobuf/ptypes/wrappers" "github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_go_proto" "github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_shape_go_proto" "github.com/tensorflow/tensorflow/tensorflow/go/core/framework/types_go_proto" "google.golang.org/grpc" ) var ( // TensorFlow serving grpc address. address = "127.0.0.1:8500" ) func main(a) { // Create a grpc request. request := &apis.PredictRequest{ ModelSpec: &apis.ModelSpec{}, Inputs: make(map[string]*tensor_go_proto.TensorProto), } request.ModelSpec.Name = "first_model" request.ModelSpec.SignatureName = "serving_default" // request.ModelSpec.VersionChoice = &apis.ModelSpec_VersionLabel{VersionLabel: "stable"} request.ModelSpec.VersionChoice = &apis.ModelSpec_Version{Version: &wrappers.Int64Value{Value: 0}} request.Inputs["input_1"] = &tensor_go_proto.TensorProto{ Dtype: types_go_proto.DataType_DT_INT64, TensorShape: &tensor_shape_go_proto.TensorShapeProto{ Dim: []*tensor_shape_go_proto.TensorShapeProto_Dim{ &tensor_shape_go_proto.TensorShapeProto_Dim{ Size: int64(2), }, &tensor_shape_go_proto.TensorShapeProto_Dim{ Size: int64(31), }, }, }, Int64Val: []int64{ 1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1,}}// Create a grpc connection. conn, err := grpc.Dial(address, grpc.WithInsecure(), grpc.WithBlock(), grpc.WithTimeout(10*time.Second)) iferr ! =nil { log.Fatalf("couldn't connect: %s", err.Error()) } defer conn.Close() // Wrap the grpc uri with client. client := apis.NewPredictionServiceClient(conn) ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() // Send the grpc request. response, err := client.Predict(ctx, request) iferr ! =nil { log.Fatalf("couldn't get response: %v", err) } log.Printf("%+v", response) } Copy the code
-
Making the address is: https://github.com/AlexanderJLiu/tensorflow-serving-api/tree/master/golang
Java
To generate java-related API code files, we need to install the Java environment and some Protoc plug-ins to assist us in file generation. The following operations are performed on the Linux operating system.
-
Its installation.
# centos yum-config-manager --enable rhel-7-server-optional-rpms yum install java-11-openjdk-devel # ubuntu apt-get install openjdk-11-jdk # test java -version Copy the code
-
Protoc -gen-grpc-java is installed to generate GRPC Java files.
Wget https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/1.30.2/protoc-gen-grpc-java-1.30.2-linux-x86_64.exe mv Protoc - gen - GRPC - Java - 1.30.2 - Linux - x86_64. Exe/usr/local/protoc/bin/protoc - gen - GRPC - JavaCopy the code
-
Switches source code to the specified branch or label.
CD tensorflow-serving- API /tensorflow git checkout tags/v2.2.0 CD Tensorflow -serving- API/Serving Git checkout tags/v2.2.0Copy the code
-
Generate Java files using the Protoc tool.
cd tensorflow-serving-api protoc -I=serving -I=tensorflow --plugin=/usr/local/protoc/bin/protoc-gen-grpc-java --grpc-java_out=java --java_out=java serving/tensorflow_serving/*/*.proto protoc -I=serving -I=tensorflow --plugin=/usr/local/protoc/bin/protoc-gen-grpc-java --grpc-java_out=java --java_out=java serving/tensorflow_serving/sources/storage_path/*.proto Copy the code
-i specifies the path for proto files to search for dependent files. The value can be specified multiple times. Plugin specifies the path of the GRPC plug-in to use. — grpc-javA_out Specifies the directory for saving GRPC Java files. — javA_out Specifies the directory to save Java files. The last line of the command specifies, in the form of a wildcard, the input location of the proto file.
Since TensorFlow has already generated a Java file for TensorFlow based on the proto file, there is no need to generate a Java file for TensorFlow. Implementation (” org. Tensorflow: proto: 1.15.0 “).
-
Create a Java project and send a GRPC prediction request to first_model. For details of the project, please refer to Github’s implementation and instructions. Only the main function code is listed here, as follows:
package com.github.alex; import java.util.Arrays; import java.util.concurrent.TimeUnit; import com.google.protobuf.Int64Value; import org.tensorflow.framework.DataType; import org.tensorflow.framework.TensorProto; import org.tensorflow.framework.TensorShapeProto; import org.tensorflow.framework.TensorShapeProto.Dim; import io.grpc.ManagedChannel; import io.grpc.ManagedChannelBuilder; import io.grpc.stub.StreamObserver; import tensorflow.serving.Model.ModelSpec; import tensorflow.serving.Predict.PredictRequest; import tensorflow.serving.Predict.PredictResponse; import tensorflow.serving.PredictionServiceGrpc; import tensorflow.serving.PredictionServiceGrpc.PredictionServiceBlockingStub; import tensorflow.serving.PredictionServiceGrpc.PredictionServiceStub; public class App { public String getGreeting(a) { return "Hello world."; } public static void main(String[] args) { PredictRequest.Builder requestBuilder = PredictRequest.newBuilder(); ModelSpec.Builder modelSpecBuilder = ModelSpec.newBuilder(); modelSpecBuilder.setSignatureName("serving_default"); modelSpecBuilder.setName("first_model"); modelSpecBuilder.setVersion(Int64Value.newBuilder().setValue(0L)); requestBuilder.setModelSpec(modelSpecBuilder.build()); TensorProto.Builder tensorProtoBuilder = TensorProto.newBuilder(); tensorProtoBuilder.setDtype(DataType.DT_INT64); Dim[] dim = {Dim.newBuilder().setSize(1).build(), Dim.newBuilder().setSize(31).build()}; tensorProtoBuilder .setTensorShape(TensorShapeProto.newBuilder().addAllDim(Arrays.asList(dim))); // tensorProtoBuilder.setTensorShape(TensorShapeProto.newBuilder() // .addDim(Dim.newBuilder().setSize(1)).addDim(Dim.newBuilder().setSize(31))); Long[] inputs = {1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L.1L}; tensorProtoBuilder.addAllInt64Val(Arrays.asList(inputs)); requestBuilder.putInputs("input_1", tensorProtoBuilder.build()); PredictRequest request = requestBuilder.build(); System.out.println(request); String target = "127.0.0.1:8500"; // Create a communication channel to the server, known as a Channel. Channels are // thread-safe and reusable. It is common to create channels at the beginning of your // application and reuse them until the application shuts down. ManagedChannel channel = ManagedChannelBuilder.forTarget(target) // Channels are secure by default (via SSL/TLS). For the example we disable TLS to // avoid needing certificates. .usePlaintext().build(); PredictionServiceBlockingStub stub = PredictionServiceGrpc.newBlockingStub(channel); try { PredictResponse response = stub.predict(request); System.out.println(response); channel.shutdownNow().awaitTermination(5, TimeUnit.SECONDS); } catch(InterruptedException e) { e.printStackTrace(); }}}Copy the code
-
Making the address is: https://github.com/AlexanderJLiu/tensorflow-serving-api/tree/master/java
Other languages
I created a project on Github called TensorFlow-Serving API, which aims to generate Tensorflow serving API files for all the languages supported by the Protocol Buffer and GRPC, Examples of use are given in the form of complete projects.
The project is progressing and currently implements the Golang, Java, and Python TensorFlow Serving API and project examples. More implementations will be added and you are welcome to contribute.
Project link address: https://github.com/AlexanderJLiu/tensorflow-serving-api
The resources
- Go Installation Instructions
- Protocol Buffer Basics: Go
- ProtoBuf: Go Generated Code
- Go GRPC Examples
- Install OpenJDK on Windows and Linux
- Protocol Buffer Basics: Java
- ProtoBuf: Java Generated Code
- GRPC: Java Generated-code Reference