Thrift golang parsing

RPC

What is the RPC?

Remote Procedure Call (RPC) translates to Remote Procedure Call

The difference between RPC and HTTP

HTTP is a protocol. RPC can be implemented through HTTP or a set of protocols implemented by sockets themselves.

In terms of complexity, an RPC framework is certainly superior to a simple HTTP interface. However, there is no doubt that due to the limitation of HTTP protocol, HTTP interface needs to have HTTP request header, resulting in transmission efficiency or security inferior to RPC

In addition,HTTP has a higher cost of connection and disconnection than TCP. HTTP supports connection pooling reuse (HTTP 1.x)

Thrift architecture

Apache Thrift is a cross-language service framework, essentially RPC, with serialization and deserialization mechanisms. Thrift contains a complete stack structure for building both the client and server sides.

Transport protocol (`TProtocol`)

Thrift allows users to select the difference between the client and server. The transmission protocols are generally classified into text and binary. To save bandwidth and improve transmission efficiency, binary transmission protocols are generally used.

TBinaryProtocol: Data is transmitted in binary encoding format
TCompactProtocol: Efficient, dense binary encoding format for data transmission
TJSONProtocolUse:JSONData encoding protocol for data transmission
TDebugProtocol: Use a readable text format that is easy to understanddebug

Data transmission mode (`TTransport`)

TTransport is the transport layer closely related to the underlying data transport. For each of the supported underlying transport modes, there exists a corresponding TTransport. At this layer, data is processed byte stream by byte stream, that is, the transport layer sees byte after byte and sends and receives these bytes in order. TTransport does not know what type of data it is transmitting, and in fact the transport layer does not care what type of data it is sending and receiving in bytes. Data type parsing is done at the TProtocol layer.

TSocket: Use blockingI/OTransmission is the most common mode
THttpTransport: in this paper,HTTPProtocol for data transmission
TFramedTransPortTo:frameFor unit transmission, used in non-blocking services;
TFileTransPort: Transfers files
TMemoryTransport: The memory is usedI/Otransmission
TZlibTransportUse:zlibCompressed and used in conjunction with other transmission methods
TBufferedTransportTo a certaintransportObject to operate on databuffer, that is, frombufferTo read data for transmission, or write data directly tobuffer

Server network Model (`TServer`)

The main task of the TServer in the Thrift framework is to receive client requests and forward them to a processor for request processing. Thrift provides different TServer models for different access sizes. The server models currently supported by Thrift include:

TSimpleServerThe single-threaded server side uses standard blockingI/O
TTHreaadPoolServerThe multithreaded server side uses standard blockingI/O
TNonblockingServer: Multi-threaded server uses non-blockingI/O
TThreadedServer: Multithreaded network model, using blockingI/OCreate a thread for each request

For ‘Golang’, there is only ‘TSimpleServer’ service mode, and it is non-blocking

TProcesser

The TProcessor operates on the inputProtocol and outputProtocol of a TServer request. That is, the TProcessor reads the request data of the client from the inputProtocol and writes the return value of the user logic to the outputProtocol. TProcessorprocess is a key handler function because all RPC calls from the client are processed and forwarded through it

ThriftClient

Like TProcessor, ThriftClient operates inputProtocol and outputProtocol. The difference is that ThriftClient divides RPC calls into send and receive steps:

sendStep that takes the user’s call parameters as a wholestructwriteTProtocolAnd sent toTServer.
sendAfter the end,thriftClientImmediately enterreceiveState waiting forTServerThe response. forTServerReturn value resolution using the return value resolution class, completerpcThe call.

TSimpleServer Service mode

This is actually not a typical TSimpleServer, because it does not block after accepting a socket. It is more like a TThreadedServer and can handle different connections in different Goroutines. This will work if golang users implement something like conn-pool on the client side.

type TSimpleServer struct {
	quit chan struct{}     // Use blocking channel to judge

	processorFactory       TProcessorFactory
	serverTransport        TServerTransport
	inputTransportFactory  TTransportFactory
	outputTransportFactory TTransportFactory
	inputProtocolFactory   TProtocolFactory
	outputProtocolFactory  TProtocolFactory
}

Copy the code

The following code thrift-idl is used as an example for the following parsing

namespace go echo

struct EchoReq {
    1: string msg;
}

struct EchoRes {
    1: string msg;
}

service Echo {
    EchoRes echo(1: EchoReq req);
}
Copy the code

Server-side Server code

func (p *TSimpleServer) Serve(a) error {
	err := p.Listen()
	iferr ! =nil {
		return err
	}
	p.AcceptLoop()
	return nil
}

func (p *TSimpleServer) AcceptLoop(a) error {
	for {
	    // Accept() blocks and calls listener.accept ()
		client, err := p.serverTransport.Accept()
		iferr ! =nil {
			select {
			case <-p.quit:
				return nil
			default:}return err
		}
		ifclient ! =nil {
			go func(a) {
				iferr := p.processRequests(client); err ! =nil {
					log.Println("error processing request:", err)
				}
			}()
		}
	}
}
Copy the code

Thrift 1.0 does not gracefully restart the server if the server is still processing requests. However, the latest version of Go Thrift uses golang WaitGroup to gracefully restart the server

func (p *TSimpleServer) processRequests(client TTransport) error {
	processor := p.processorFactory.GetProcessor(client)

	inputTransport := p.inputTransportFactory.GetTransport(client)
	outputTransport := p.outputTransportFactory.GetTransport(client)

	inputProtocol := p.inputProtocolFactory.GetProtocol(inputTransport)
	outputProtocol := p.outputProtocolFactory.GetProtocol(outputTransport)
	defer func(a) {
		if e := recover(a); e ! =nil {
			log.Printf("panic in processor: %s: %s", e, debug.Stack())
		}
	}()
	ifinputTransport ! =nil {
		defer inputTransport.Close()
	}
	ifoutputTransport ! =nil {
		defer outputTransport.Close()
	}
	for {
		ok, err := processor.Process(inputProtocol, outputProtocol)

		if err, ok := err.(TTransportException); ok && err.TypeId() == END_OF_FILE {
			return nil
		} else iferr ! =nil {
			log.Printf("error processing request: %s", err)
			return err
		}
		if err, ok := err.(TApplicationException); ok && err.TypeId() == UNKNOWN_METHOD {
			continue
		}
 		if! ok {break}}return nil
}
Copy the code

Process logic

func (p *EchoProcessor) Process(iprot, oprot thrift.TProtocol) (success bool, err thrift.TException) {
	name, _, seqId, err := iprot.ReadMessageBegin()
	iferr ! =nil {
		return false, err
	}
	// Get the name passed, if it exists
	if processor, ok := p.GetProcessorFunction(name); ok {
		return processor.Process(seqId, iprot, oprot)
	}
	// Exception logic
	iprot.Skip(thrift.STRUCT)
	iprot.ReadMessageEnd()
	x3 := thrift.NewTApplicationException(thrift.UNKNOWN_METHOD, "Unknown function "+name)
	oprot.WriteMessageBegin(name, thrift.EXCEPTION, seqId)
	x3.Write(oprot)
	oprot.WriteMessageEnd()
	oprot.Flush()
	return false, x3

}
Copy the code

When the TServer receives the RPC request, it calls TProcessorprocess to process it. TProcessorprocess first call the TTransport readMessageBegin interface, read out the name of the RPC calls and RPC call type.

If the RPC call type is RPC Call, tprocesser.process_fn is called to continue processing, and an exception is thrown for the unknown RPC call type. Process_fn looks for the corresponding RPC handler function in its processMap based on the RPC call name. If a corresponding RPC handler exists, the handler is called to proceed with the request response. If it does not exist, an exception is thrown.

func (p *echoProcessorEcho) Process(seqId int32, iprot, oprot thrift.TProtocol) (success bool, err thrift.TException) {
	args := EchoEchoArgs{}
	// Read the parameters of the input parameter
	iferr = args.Read(iprot); err ! =nil {
		iprot.ReadMessageEnd()
		x := thrift.NewTApplicationException(thrift.PROTOCOL_ERROR, err.Error())
		oprot.WriteMessageBegin("echo", thrift.EXCEPTION, seqId)
		x.Write(oprot)
		oprot.WriteMessageEnd()
		oprot.Flush()
		return false, err
	}

	iprot.ReadMessageEnd()

	result := EchoEchoResult{}
	var retval *EchoRes
	var err2 error
	// The value here is thrift. Why err cannot transmit errors
	ifretval, err2 = p.handler.Echo(args.Req); err2 ! =nil {
		x := thrift.NewTApplicationException(thrift.INTERNAL_ERROR, "Internal error processing echo: "+err2.Error())
		oprot.WriteMessageBegin("echo", thrift.EXCEPTION, seqId)
		x.Write(oprot)
		oprot.WriteMessageEnd()
		oprot.Flush()
		return true, err2
	} else {
		result.Success = retval
	}

	if err2 = oprot.WriteMessageBegin("echo", thrift.REPLY, seqId); err2 ! =nil {
		err = err2
	}

	if err2 = result.Write(oprot); err == nil&& err2 ! =nil {
		err = err2
	}
	if err2 = oprot.WriteMessageEnd(); err == nil&& err2 ! =nil {
		err = err2
	}

	if err2 = oprot.Flush(); err == nil&& err2 ! =nil {
		err = err2
	}
	iferr ! =nil {
		return
	}
	return true, err
}
Copy the code

The service side`stop`code

var once sync.Once
func (p *TSimpleServer) Stop(a) error {
	q := func(a) {
		p.quit <- struct{}{}
		p.serverTransport.Interrupt()
	}
	once.Do(q)
	return nil
}
Copy the code

Stop is a simple function that writes data directly to a blocking queue, and the server stops accepting requests

Client code

Function called by Client

func (p *EchoClient) Echo(req *EchoReq) (r *EchoRes, err error) {

	iferr = p.sendEcho(req); err ! =nil {
		return
	}

	return p.recvEcho()
}
Copy the code

SendEcho () function

func (p *EchoClient) sendEcho(req *EchoReq) (err error) {
	oprot := p.OutputProtocol
	if oprot == nil {
		oprot = p.ProtocolFactory.GetProtocol(p.Transport)
		p.OutputProtocol = oprot
	}
	// seqid + 1
	p.SeqId++

	if err = oprot.WriteMessageBegin("echo", thrift.CALL, p.SeqId); err ! =nil {
		return
	}

	// Build parameters
	args := EchoEchoArgs{
		Req: req,
	}

	iferr = args.Write(oprot); err ! =nil {
		return
	}
	// The notification server has been sent
	iferr = oprot.WriteMessageEnd(); err ! =nil {
		return
	}
	return oprot.Flush()
}
Copy the code

RecvEcho () function

func (p *EchoClient) recvEcho(a) (value *EchoRes, err error) {
	iprot := p.InputProtocol
	if iprot == nil {
		iprot = p.ProtocolFactory.GetProtocol(p.Transport)
		p.InputProtocol = iprot
	}
	//
	method, mTypeId, seqId, err := iprot.ReadMessageBegin()
	iferr ! =nil {
		return
	}
	ifmethod ! ="echo" {
		err = thrift.NewTApplicationException(thrift.WRONG_METHOD_NAME, "echo failed: wrong method name")
		return
	}
	ifp.SeqId ! = seqId { err = thrift.NewTApplicationException(thrift.BAD_SEQUENCE_ID,"echo failed: out of sequence response")
		return
	}
	if mTypeId == thrift.EXCEPTION {
		error0 := thrift.NewTApplicationException(thrift.UNKNOWN_APPLICATION_EXCEPTION, "Unknown Exception")
		var error1 error
		error1, err = error0.Read(iprot)
		iferr ! =nil {
			return
		}
		iferr = iprot.ReadMessageEnd(); err ! =nil {
			return
		}
		err = error1
		return
	}
	ifmTypeId ! = thrift.REPLY { err = thrift.NewTApplicationException(thrift.INVALID_MESSAGE_TYPE_EXCEPTION,"echo failed: invalid message type")
		return
	}
	result := EchoEchoResult{}
	iferr = result.Read(iprot); err ! =nil {
		return
	}
	iferr = iprot.ReadMessageEnd(); err ! =nil {
		return
	}
	value = result.GetSuccess()
	return
}
Copy the code

Thrift installation problem on MAC machines

Question 1:go get git.apache.org/thrift.git/lib/go/thriftfailure
Problem 2: Direct usegithub.comThe supplied version reports unknown errors

Question 2 needs to be based on your thrift To judge which download version – the version of thrift, such as my thrift version 0.10.0 then need to download the thrift address to https://github.com/apache/thrift/archive/0.10.0.zip

Manually create the mkdir -p git.apache.org/thrift.git/lib/go/ directory, and then remove the go after the download file to the directory ~

Reference

www.16boke.com/series/deta…
www.ibm.com/developerwo…
Waylau.com/remote-proc…
www.slideshare.net/dvirsky/int…
www.cnblogs.com/LBSer/p/485…
www.cnblogs.com/winner-0715…
zhuanlan.zhihu.com/p/53685973
zhuanlan.zhihu.com/p/53687302
Segmentfault.com/a/119000001…

RPC

Thrift architecture

Transport protocol (TProtocol)

Data transmission mode (TTransport)

Server network Model (TServer)

TProcesser

ThriftClient

TSimpleServer Service mode

Server-side Server code

The service sidestopcode

Client code

Thrift installation problem on MAC machines

Reference

Related Posts

How do computer networks learn? A wave of angry answer!

Spring Boot practice toss remember (a) : fast, Hello World example

Java Concurrent programming: Thread safety

Transport protocol (`TProtocol`)

Data transmission mode (`TTransport`)

Server network Model (`TServer`)

The service side`stop`code