Protobuf is a serialization and deserialization protocol library developed by Google. We can design the format of the data to be passed by ourselves. Proto files define the format of the data to be passed. For example, the ONNX interchange model commonly used in deep learning is written using.proto. We can read the.onnx model from various front ends (MNN, NCNN, TVM front ends), but first you need to install protobuf.

Onnx has been briefly introduced in previous posts, where onnx.proto represents the basic data structure of the ONNX model. In general, protobuf is often used with Cmake, which has an official modules that can be generated using a few simple commands protobuf_generate_cpp to generate the corresponding.pb.cc and.pb.h.

A simple example:

find_package(Protobuf REQUIRED)
include_directories(${Protobuf_INCLUDE_DIRS})
include_directories(${CMAKE_CURRENT_BINARY_DIR})
protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS foo.proto)
protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS EXPORT_MACRO DLL_EXPORT foo.proto)
protobuf_generate_python(PROTO_PY foo.proto)
add_executable(bar bar.cc ${PROTO_SRCS} ${PROTO_HDRS})
target_link_libraries(bar ${Protobuf_LIBRARIES})
Copy the code

But this example is so simple that if we only have one.proto file or all of them in the same directory, there’s nothing wrong with using this command…

But if this is the case, our file directory is as follows:

├ ─ ─ CMakeLists. TXT ├ ─ ─ the README. Md ├ ─ ─ meta │ └ ─ ─ proto │ ├ ─ ─ CMakeLists. TXT │ └ ─ ─ common │ ├ ─ ─ bar │ │ ├ ─ ─ CMakeLists. TXT │ │ └ ─ ─ bar. Proto │ └ ─ ─ foo │ ├ ─ ─ CMakeLists. TXT │ └ ─ ─ foo proto └ ─ ─ the SRC ├ ─ ─ CMakeLists. TXT ├ ─ ─ c_proto. Cc └ ─ ─ c_proto.hhCopy the code

The foo.proto file is as follows:

message foo_msg 
{
  optional string name = 1;
}
Copy the code

The file for bar.proto is as follows:

import "common/foo/foo.proto";
 
message bar_msg 
{
  optional foo_msg foo = 1;
  optional string name = 2;
}
Copy the code

As mentioned above, the bar file refers to foo and the two are not in the same directory. If protobuf_Generate_cpp is used to generate the bar file, an error will be reported. (This example is taken from one of Yu’s blog posts)

I thought about putting them in the same directory… The import code in bar.proto is then modified, which is fine, but obviously not suitable for large projects.

And that big project is clearly Mediapipe… It took me a long time.

Details about Mediapipe are in another article. Mediapipe uses a lot of ProtoBuf technology to represent graph structures, and instead of using Cmake to build projects, Mediapipe uses Google’s own bazel, which I won’t comment on, but now I need to use Cmake to build it.

This is also the beginning of a nightmare. Mediapipe has many.proto files, and there are many.proto files in the core framework directory, both root and subdirectories.

And there is an order of references between each proto file, the Calculator. proto file in the root directory of the framework:

// mediapipe/framework/calculator.proto
syntax = "proto3";

package mediapipe;

import public "mediapipe/framework/calculator_options.proto";

import "google/protobuf/any.proto";
import "mediapipe/framework/mediapipe_options.proto";
import "mediapipe/framework/packet_factory.proto";
import "mediapipe/framework/packet_generator.proto";
import "mediapipe/framework/status_handler.proto";
import "mediapipe/framework/stream_handler.proto";
Copy the code

Each.proto file imports files from other directories. The import here is similar to include in C++, but the import here can refer to each other. For example, the status_handle. proto reference mediapipe_options.proto.

Using the protobuf_generate_cpp command directly on all of the above.proto files will result in an error because the files are not in the same directory and relative import directories cannot be parsed. In addition, the.cc files in different directories will reference the.pb.h files generated in corresponding directories. We need to generate.pb.cc and.

Pb. cc and.pb.h generated by proto are not in the original directory, but are concentrated in cmake-build-debug(release). In addition, we need to move the.pb.cc and.pb.h files generated therein to the original address (as is the case with Clion).

Modify cmake correctly

In this case, it is appropriate to use command generation directly.

First find all the.proto files that need to be compiled:

file(GLOB protobuf_files
        mediapipe/framework/*.proto
        mediapipe/framework/tool/*.proto
        mediapipe/framework/deps/*.proto
        mediapipe/framework/testdata/*.proto
        mediapipe/framework/formats/*.proto
        mediapipe/framework/formats/annotation/*.proto
        mediapipe/framework/formats/motion/*.proto
        mediapipe/framework/formats/object_detection/*.proto
        mediapipe/framework/stream_handler/*.proto
        mediapipe/util/*.proto
        mediapipe/calculators/internal/*.proto
        )
Copy the code

Next, define the relevant directory address. PROTO_META_BASE_DIR is the directory where files are generated after compilation. PROTO_FLAGS, important, specifies the total path to find when compiling.proto files. The import command in.proto uses this address to connect to other.proto files:

SET(PROTO_META_BASE_DIR ${CMAKE_CURRENT_BINARY_DIR})
LIST(APPEND PROTO_FLAGS -I${CMAKE_CURRENT_SOURCE_DIR})
Copy the code

Once set up, loop through FOREACH through the previous.proto files, compile each file in turn, and then move the generated.pb.cc and.pb.h back to their original directories.

FOREACH(FIL ${protobuf_files})

    GET_FILENAME_COMPONENT(FIL_WE ${FIL} NAME_WE)

    string(REGEX REPLACE ". + / (. +) \ \.. *" 1 \ \ "" FILE_NAME ${FIL})
    string(REGEX REPLACE "(.+)\\${FILE_NAME}.*" 1 \ \ "" FILE_PATH ${FIL})

    string(REGEX MATCH "(/mediapipe/framework.*|/mediapipe/util.*|/mediapipe/calculators/internal/)" OUT_PATH ${FILE_PATH})

    set(PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}${OUT_PATH}${FIL_WE}.pb.cc")
    set(PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}${OUT_PATH}${FIL_WE}.pb.h")

    EXECUTE_PROCESS(
            COMMAND ${PROTOBUF_PROTOC_EXECUTABLE} ${PROTO_FLAGS} --cpp_out=${PROTO_META_BASE_DIR} ${FIL}
    )
    message("Copying " ${PROTO_SRCS} " to " ${FILE_PATH})

    file(COPY ${PROTO_SRCS} DESTINATION ${FILE_PATH})
    file(COPY ${PROTO_HDRS} DESTINATION ${FILE_PATH})

ENDFOREACH(a)Copy the code

Refer to the link

Blog.argcv.com/articles/38… www.v2ex.com/t/602363 stackoverflow.com/questions/2…

Pulled me

  • If you are like-minded with me, Lao Pan is willing to communicate with you;
  • If you like Lao Pan’s content, welcome to follow and support.
  • If you like my article, please like 👍 collect 📁 comment 💬 three times ~

If you want to know how Lao Pan learned to tread pits, and if you want to talk with me about his problems, please follow the public account “Oldpan blog”. Lao Pan will also sort out some of his own private stash, hoping to help you, click on the mysterious portal to obtain.