The series of translated articles have been posted on GitHub: blockchain-tutorial. If there are any updates, they will be posted on GitHub and may not be synchronized here. To run the code directly, clone the tutorial repository on GitHub, go to the SRC directory and execute make.
The introduction
The prototypes we have built so far have all the key features of blockchain: anonymous, secure, randomly generated addresses; Blockchain data storage; Workload proof system; Store transactions reliably. While these features are indispensable, they still fall short. What really makes these features work, and makes cryptocurrencies possible, is the network. What is the use of implementing such a blockchain if it only runs on a single node? What good are these cryptographics-based features if there is only one user? It is the network that makes the whole mechanism work and glow.
You can think of these blockchain features as rules, similar to the rules that humans establish by living together and multiplying, a social arrangement. A blockchain network is a community of programs, where every program plays by the same rules, and it’s by following the same rules that the network survives. Similarly, when people have the same idea, they can put their fists together and build a better life. If someone plays by different rules, they live in a divided community (state, commune, etc.). Similarly, if there are blockchain nodes that follow different rules, then there will be a split network.
The point is: if there is no network, or most nodes don’t follow the same rules, then the rules are useless!
Note: Unfortunately, I didn’t have enough time to implement a real P2P network prototype. In this article, I’ll show you one of the most common scenarios involving different types of nodes. Continuing to improve this scenario and implementing it as a P2P network is a good challenge and practice for you! I can’t guarantee that it will work in any scenario other than the one in this article. I’m sorry!
The code implementation for this article has changed a lot, so click here to see all the code changes.
Blockchain network
Blockchain networks are decentralized, meaning there are no servers, nor do clients need to rely on them to fetch or process data. In a blockchain network, there are nodes, each of which is a full-fledged member of the network. A node is everything: it is both a client and a server. This is something to keep in mind because it’s very different from traditional web applications.
A blockchain network is a PEER-to-peer (peer-to-peer) network, in which nodes are directly connected to other nodes. Its topology is flat because there is no hierarchy in the world of nodes. Here’s how it looks:
Business vector created by Dooder – Freepik.com
Such network nodes are more difficult to implement because they have to perform many operations. Each node must interact with many other nodes, it must request the state of other nodes, compare it with its own state, and update it when the state becomes too old.
The node role
Although nodes have complete and mature properties, they can also play different roles in the network. Such as:
-
Nodes like miners run on powerful or specialized hardware, such as ASics, with the sole goal of digging up new chunks as fast as possible. Miners are the only role in the blockchain that might use proof of work, because mining actually means solving PoW puzzles. In the proof-of-stake PoS blockchain, there is no mining.
-
Full nodes These nodes verify the validity of blocks dug by miners and confirm transactions. To do that, they must have a full copy of the blockchain. At the same time, all nodes perform routing operations to help other nodes discover each other. A very important part of a network is to have enough full nodes. Because it is these nodes that perform the decision function: they determine the validity of a block or a transaction.
-
SPV SPV represents Simplified Payment Verification. These nodes do not store a full copy of the blockchain, but can still validate transactions (though not all transactions, but a subset of them, for example, sent to a specific address). An SPV node relies on a full node to get data, and there may be multiple SPV nodes connected to a full node. SPV makes wallet applications possible: a person does not need to download the entire blockchain, but can still verify his transactions.
Network simplification
In order to implement the network in the current blockchain prototype, we had to simplify things. Because we don’t have enough computers to simulate a multi-node network. Of course, we could use a virtual machine or a Docker to solve this problem, but that would make everything more complicated: you would have to solve the virtual machine or Docker problem that might occur first, and my goal is to focus all of my efforts on blockchain implementation. So, we want to run multiple blockchain nodes on one machine, and we want them to have different addresses. To achieve this, we will use port numbers as node identifiers instead of IP addresses, such as nodes that will have addresses like 127.0.0.1:3000, 127.0.0.1:3001, 127.0.0.1:3002, and so on. We call these port node ids and set them using the environment variable NODE_ID. Therefore, you can open multiple terminal Windows and set different NODE_ID to run different nodes.
This method also requires different blockchain and wallet files. They must now be named depending on the node ID, such as Blockchain_3000.db, Blockchain_30001.db and Wallet_3000.db, Wallet_30001.db and so on.
implementation
So what happens when you download Bitcoin Core and run it for the first time? It must connect to a node to download the latest state of the blockchain. Given that your computer isn’t aware of all or some of the Bitcoin nodes, what exactly is the “node” that you’re connected to?
Hard-coding an address in Bitcoin Core has proven to be a mistake: new nodes cannot join the network because nodes can be attacked or shut down. In Bitcoin Core, DNS seeds are hard-coded. Although these are not nodes, the DNS server knows the addresses of some nodes. When you launch a brand new Bitcoin Core, it connects to a seed node, fetches a full list of nodes, and then downloads the blockchain from those nodes.
However, in our current implementation, we cannot achieve complete decentralization because of the characteristics of decentralization. We will have three nodes:
-
A central node. All other nodes will connect to this node, and this node will send data between other nodes.
-
A miner node. This node stores new transactions in the memory pool, and when there are enough transactions, it packs up and mines a new block.
-
A wallet node. This node is used to send coins between wallets. But unlike an SPV node, it stores a full copy of the blockchain.
scenario
The goal of this article is to achieve the following scenarios:
- The central node creates a blockchain.
- One other (wallet) node connects to the central node and downloads the blockchain.
- Another (miner) node connects to the central node and downloads the blockchain.
- The wallet node creates a transaction.
- The miner node receives the transaction and saves it to the memory pool.
- When there are enough transactions in the memory pool, the miner starts digging a new block.
- When a new block is dug up, it is sent to the central node.
- The wallet node synchronizes with the central node.
- Users of the wallet node check whether their payment was successful.
This is the general process in Bitcoin. Although we will not implement a real P2P network, we will implement a real and most important user scenario for Bitcoin.
version
Nodes communicate via messages. When a new node starts running, it fetches several nodes from a DNS seed and sends them a version message that looks something like this in our implementation:
type version struct {
Version int
BestHeight int
AddrFrom string
}Copy the codeSince we only have one Version of the blockchain, the Version field actually stores no important information. BestHeight stores the height of the nodes in the blockchain. AddFrom stores the address of the sender.
What should the node that receives the Version message do? It responds to its own Version message. This is a handshake 🤝 : there can be no other communication without first greeting each other. However, this is not being polite: Version is used to find a longer blockchain. When a node receives a version message, it checks to see if the value of the node’s blockchain is larger than the value of BestHeight. If not, the node requests and downloads the missing block.
To receive messages, we need a server:
var nodeAddress string
var knownNodes = []string{"localhost:3000"}
func StartServer(nodeID, minerAddress string) {
nodeAddress = fmt.Sprintf("localhost:%s", nodeID)
miningAddress = minerAddress
ln, err := net.Listen(protocol, nodeAddress)
defer ln.Close()
bc := NewBlockchain(nodeID)
ifnodeAddress ! = knownNodes[0] {
sendVersion(knownNodes[0], bc)
}
for {
conn, err := ln.Accept()
go handleConnection(conn, bc)
}
}Copy the codeFirst, we hardcode the address of the central node: each node must know where to start initialization. The minerAddress parameter specifies the address to receive mining rewards. Code snippet:
ifnodeAddress ! = knownNodes[0] {
sendVersion(knownNodes[0], bc)
}Copy the codeThis means that if the current node is not the central node, it must send a version message to the central node to see if its own blockchain is obsolete.
func sendVersion(addr string, bc *Blockchain) {
bestHeight := bc.GetBestHeight()
payload := gobEncode(version{nodeVersion, bestHeight, nodeAddress})
request := append(commandToBytes("version"), payload...)
sendData(addr, request)
}Copy the codeOur message, at the bottom, is a sequence of bytes. The first 12 bytes specify the command name (such as version here), and the following bytes contain the goB encoded message structure, commandToBytes looks like this:
func commandToBytes(command string) []byte {
var bytes [commandLength]byte
for i, c := range command {
bytes[i] = byte(c)
}
return bytes[:]
}Copy the codeIt creates a 12-byte buffer, populates it with command names, and leaves the remaining bytes empty. Here’s an inverse function:
func bytesToCommand(bytes []byte) string {
var command []byte
for _, b := range bytes {
ifb ! =0x0 {
command = append(command, b)
}
}
return fmt.Sprintf("%s", command)
}Copy the codeWhen a node receives a command, it will run bytesToCommand to extract the command name, and select the correct processing command processor main body:
func handleConnection(conn net.Conn, bc *Blockchain) {
request, err := ioutil.ReadAll(conn)
command := bytesToCommand(request[:commandLength])
fmt.Printf("Received %s command\n", command)
switch command {
...
case "version":
handleVersion(request, bc)
default:
fmt.Println("Unknown command!")
}
conn.Close()
}Copy the codeHere is the version command handler:
func handleVersion(request []byte, bc *Blockchain) {
var buff bytes.Buffer
var payload verzion
buff.Write(request[commandLength:])
dec := gob.NewDecoder(&buff)
err := dec.Decode(&payload)
myBestHeight := bc.GetBestHeight()
foreignerBestHeight := payload.BestHeight
if myBestHeight < foreignerBestHeight {
sendGetBlocks(payload.AddrFrom)
} else if myBestHeight > foreignerBestHeight {
sendVersion(payload.AddrFrom, bc)
}
if! nodeIsKnown(payload.AddrFrom) { knownNodes =append(knownNodes, payload.AddrFrom)
}
}Copy the codeFirst, we need to decode the request to extract valid information. All processors are similar in this part, so we’ll omit it in the code snippet below.
The node then compares the BestHeight extracted from the message to itself. If its own node’s blockchain is longer, it replies with a version message. Otherwise, it sends a getBlocks message.
getblocks
type getblocks struct {
AddrFrom string
}Copy the codeGetblocks means “show me what blocks you have” (in bitcoin, this is more complicated). Notice that it doesn’t say “give me all your blocks”, but rather asks for a list of block hashes. This is to lighten the load on the network because blocks can be downloaded from different nodes and we don’t want to download tens of GIGABytes of data from a single node.
Processing commands is simple:
func handleGetBlocks(request []byte, bc *Blockchain){... blocks := bc.GetBlockHashes() sendInv(payload.AddrFrom,"block", blocks)
}Copy the codeIn our simplified implementation, it returns all block hashes.
inv
type inv struct {
AddrFrom string
Type string
Items [][]byte
}Copy the codeBitcoin uses INV to show other nodes what blocks and transactions are in the current node. Again, it does not contain the full blockchain and transactions, just hashes. The Type field indicates whether this is a block or a transaction.
Handling INV is slightly more complicated:
func handleInv(request []byte, bc *Blockchain){... fmt.Printf("Recevied inventory with %d %s\n".len(payload.Items), payload.Type)
if payload.Type == "block" {
blocksInTransit = payload.Items
blockHash := payload.Items[0]
sendGetData(payload.AddrFrom, "block", blockHash)
newInTransit := [][]byte{}
for _, b := range blocksInTransit {
ifbytes.Compare(b, blockHash) ! =0 {
newInTransit = append(newInTransit, b)
}
}
blocksInTransit = newInTransit
}
if payload.Type == "tx" {
txID := payload.Items[0]
if mempool[hex.EncodeToString(txID)].ID == nil {
sendGetData(payload.AddrFrom, "tx", txID)
}
}
}Copy the codeIf we receive block hashes, we want to save them in the blocksInTransit variable to keep track of the downloaded blocks. This allows us to download blocks from different nodes. When placing the block in transit state, we send the getData command to the sender of the INV message and update blocksInTransit. In a real P2P network, we would want to transfer blocks from different nodes.
In our implementation, we will never send an INV with multiple hashes. That’s why when payload.Type == “tx”, you only get the first hash. We then check to see if we already have the hash in the memory pool, and if not, send a getData message.
getdata
type getdata struct {
AddrFrom string
Type string
ID []byte
}Copy the codeGetdata is used for a request for a block or transaction, which can contain only the ID of a block or transaction.
func handleGetData(request []byte, bc *Blockchain){...if payload.Type == "block" {
block, err := bc.GetBlock([]byte(payload.ID))
sendBlock(payload.AddrFrom, &block)
}
if payload.Type == "tx" {
txID := hex.EncodeToString(payload.ID)
tx := mempool[txID]
sendTx(payload.AddrFrom, &tx)
}
}Copy the codeThis handler is relatively straightforward: if they request a block, they return it; If they request a transaction, the transaction is returned. Note that we do not check whether the block or transaction actually exists. This is a bug 🙂
Block and tx
type block struct {
AddrFrom string
Block []byte
}
type tx struct {
AddFrom string
Transaction []byte
}Copy the codeIt is these messages that actually do the data transfer.
Processing block messages is simple:
func handleBlock(request []byte, bc *Blockchain){... blockData := payload.Block block := DeserializeBlock(blockData) fmt.Println("Recevied a new block!")
bc.AddBlock(block)
fmt.Printf("Added block %x\n", block.Hash)
if len(blocksInTransit) > 0 {
blockHash := blocksInTransit[0]
sendGetData(payload.AddrFrom, "block", blockHash)
blocksInTransit = blocksInTransit[1:]}else {
UTXOSet := UTXOSet{bc}
UTXOSet.Reindex()
}
}Copy the codeWhen a new block is received, we place it in the blockchain. If there are more blocks to download, we continue the request from the node where the last block was downloaded. When all the blocks are finally downloaded, the UTXO set is re-indexed.
TODO: Instead of unconditional trust, we should verify each block before adding it to the blockchain.
TODO: Instead of running utxoset.reindex (), you should use utxoset.update (block) because if the blockchain is large, it will take a lot of time to Reindex the entire UTXO set.
Handling tx messages is the hardest part:
func handleTx(request []byte, bc *Blockchain){... txData := payload.Transaction tx := DeserializeTransaction(txData) mempool[hex.EncodeToString(tx.ID)] = txif nodeAddress == knownNodes[0] {
for _, node := range knownNodes {
ifnode ! = nodeAddress && node ! = payload.AddFrom { sendInv(node,"tx"And [] []byte{tx.ID})
}
}
} else {
if len(mempool) >= 2 && len(miningAddress) > 0 {
MineTransactions:
var txs []*Transaction
for id := range mempool {
tx := mempool[id]
if bc.VerifyTransaction(&tx) {
txs = append(txs, &tx)
}
}
if len(txs) == 0 {
fmt.Println("All transactions are invalid! Waiting for new ones...")
return
}
cbTx := NewCoinbaseTX(miningAddress, "")
txs = append(txs, cbTx)
newBlock := bc.MineBlock(txs)
UTXOSet := UTXOSet{bc}
UTXOSet.Reindex()
fmt.Println("New block is mined!")
for _, tx := range txs {
txID := hex.EncodeToString(tx.ID)
delete(mempool, txID)
}
for _, node := range knownNodes {
ifnode ! = nodeAddress { sendInv(node,"block"And [] []byte{newBlock.Hash})
}
}
if len(mempool) > 0 {
goto MineTransactions
}
}
}
}Copy the codeThe first thing to do is to put the new transaction into the memory pool (again, it is necessary to validate the transaction before putting it into the memory pool). Next clip:
if nodeAddress == knownNodes[0] {
for _, node := range knownNodes {
ifnode ! = nodeAddress && node ! = payload.AddFrom { sendInv(node,"tx"And [] []byte{tx.ID})
}
}
}Copy the codeCheck whether the current node is a central node. In our implementation, the central node does not mine. It simply pushes new transactions to other nodes in the network.
The next large code snippet is “exclusive” to the miner node. Let’s break it down:
if len(mempool) >= 2 && len(miningAddress) > 0 {Copy the codeMiningAddress will only be set on the miner node. If there are two or more transactions in the memory pool of the current node (miner), start mining:
for id := range mempool {
tx := mempool[id]
if bc.VerifyTransaction(&tx) {
txs = append(txs, &tx)
}
}
if len(txs) == 0 {
fmt.Println("All transactions are invalid! Waiting for new ones...")
return
}Copy the codeFirst, all transactions in the memory pool are validated. Invalid transactions are ignored, and if there are no valid transactions, mining is interrupted.
cbTx := NewCoinbaseTX(miningAddress, "")
txs = append(txs, cbTx)
newBlock := bc.MineBlock(txs)
UTXOSet := UTXOSet{bc}
UTXOSet.Reindex()
fmt.Println("New block is mined!")Copy the codeVerified transactions are placed in a block, along with coinbase transactions with rewards. When the block is dug out, the UTXO assembly is re-indexed.
TODO: A reminder that UTXOSet.Update should be used instead of UTXOSet.Reindex.
for _, tx := range txs {
txID := hex.EncodeToString(tx.ID)
delete(mempool, txID)
}
for _, node := range knownNodes {
ifnode ! = nodeAddress { sendInv(node,"block"And [] []byte{newBlock.Hash})
}
}
if len(mempool) > 0 {
goto MineTransactions
}Copy the codeWhen a transaction is mined, it is removed from the memory pool. All other nodes to which the current node is connected receive the INV message with the new block hash. After processing the message, they can make requests to the block.
The results of
Let’s review the scenarios defined above.
First, set NODE_ID to 3000 (export NODE_ID=3000) in the first terminal window. To let you know which nodes do what, I’ll use identifiers like NODE 3000 or NODE 3001.
NODE 3000
Create a wallet and a new blockchain:
$ blockchain_go createblockchain -address CENTREAL_NODECopy the code(FOR brevity, I’ll use a fake address.)
A blockchain is then generated that contains only the Genesis block. We need to save the block and use it on other nodes. Genesis blocks assume the role of a chain identifier (in Bitcoin Core, genesis blocks are hard-coded)
$ cp blockchain_3000.db blockchain_genesis.dbCopy the codeNODE 3001
Next, open a new terminal window and set the Node ID to 3001. This will act as a wallet node. Blockchain_go createWallet generates addresses called WALLET_1, WALLET_2, WALLET_3.
NODE 3000
Send some coins to the wallet address:
$ blockchain_go send -from CENTREAL_NODE -to WALLET_1 -amount 10 -mine
$ blockchain_go send -from CENTREAL_NODE -to WALLET_2 -amount 10 -mineCopy the codeThe -mine flag indicates that the block is immediately mined by the same node. We must have this flag because initially there are no miner nodes in the network.
Start node:
$ blockchain_go startnodeCopy the codeThis node continues to run until the end of the scenario defined in this article.
NODE 3001
Activate the blockchain above which holds the Genesis Block node:
$ cp blockchain_genesis.db blockchain_3001.dbCopy the codeRunning node:
$ blockchain_go startnodeCopy the codeIt downloads all blocks from the central node. To check that everything is ok, pause the node and check the balance:
$ blockchain_go getbalance -address WALLET_1
Balance of 'WALLET_1': 10
$ blockchain_go getbalance -address WALLET_2
Balance of 'WALLET_2': 10Copy the codeYou can also check the balance of the CENTRAL_NODE address, since Node 3001 now has its own blockchain:
$ blockchain_go getbalance -address CENTRAL_NODE
Balance of 'CENTRAL_NODE': 10Copy the codeNODE 3002
Open a new terminal window, set its ID to 3002, and generate a wallet. This will be a miner node. Initializing the blockchain:
$ cp blockchain_genesis.db blockchain_3002.dbCopy the codeStart node:
$ blockchain_go startnode -miner MINER_WALLETCopy the codeNODE 3001
Send some coins:
$ blockchain_go send -from WALLET_1 -to WALLET_3 -amount 1
$ blockchain_go send -from WALLET_2 -to WALLET_4 -amount 1Copy the codeNODE 3002
Quickly switch to the miner node and you’ll see a new block dug up! Also, check the output of the central node.
NODE 3001
Switch to wallet node and start:
$ blockchain_go startnodeCopy the codeIt downloads the most recently dug block!
Pause the node and check the balance:
$ blockchain_go getbalance -address WALLET_1
Balance of 'WALLET_1': 9
$ blockchain_go getbalance -address WALLET_2
Balance of 'WALLET_2': 9
$ blockchain_go getbalance -address WALLET_3
Balance of 'WALLET_3': 1
$ blockchain_go getbalance -address WALLET_4
Balance of 'WALLET_4': 1
$ blockchain_go getbalance -address MINER_WALLET
Balance of 'MINER_WALLET': 10Copy the codeThat’s it!
conclusion
This is the last article in this series. I could have gone on to implement a real P2P network prototype, but I really didn’t have that much time. I hope this article has answered some of the questions about bitcoin technology and posed some questions for the reader that you can find answers to on your own. There are a lot of interesting things hidden in bitcoin technology! Good luck!
Postscript: You can start improving the network by implementing ADDR messages, as described in the Bitcoin Network protocol (a link can be found below). This is a very important message because it allows nodes to discover each other. I’ve started implementing it, but I’m not done yet!
Links:
- Source codes
- Bitcoin protocol documentation
- Bitcoin network
译 文 : Building Blockchain in Go. Part 7: Network