Author: freewind

Biyuan Project warehouse:

Making address: https://github.com/Bytom/bytom

Gitee address: https://gitee.com/BytomBlockchain/bytom

In the previous articles, we have been looking at how to establish a connection with a node and request block data from it. However, I soon hit a wall.

Because when I processed the block data I got, I found that I had touched the core of the original chain, that is, the data structure of the block chain and the processing of bifurcation. Without a complete understanding of this piece, there is no way to properly process block data. However, it involves too much content, in a short time to understand it thoroughly is a very difficult thing.

It was as if I wanted to get to know a city, and I followed a road from the periphery to the center. The front has been very smooth, but when I got to the city center, I found that there were many people and roads, and I got a little lost. In this case, I felt I should stop working on the core and start from a different path, from the outside in. Because along the way, I can slowly accumulate more knowledge and put myself in the learning zone instead of the panic zone. The end of the road will also be to get to the core, but not go deep. In this way, after I have walked a few more roads, I will not feel confused when I have accumulated enough knowledge to study the core.

So this article was originally intended to study how we should deal with block data sent to us by other nodes, but now it is to study how the original Dashboard is made. Why did you choose this one? Because it shows us all kinds of information and functionality in a very intuitive way. In this article, we won’t talk too much about what it does, but rather focus on how to implement such a Dashboard at the code level. The functions above it will be studied slowly in the future.

Our question today is “How the original Dashboard was made”, but it’s a bit bigger and less specific, so we’ll break it down as before:

  1. How do we enable Dashboard functionality in the original?
  2. What information and functionality does Dashboard provide?
  3. How did biyuan implement the HTTP server?
  4. What front-end framework does Dashboard use?
  5. In what way is the data on the Dashboard retrieved from the background?

Let’s start with one by one.

How do we enable Dashboard functionality in the original?

When we start bytomd Node, it automatically enables Dashboard functionality without any configuration and opens the page in the browser, which is very convenient.

If this is the first run and no account has been created, it will prompt us to create an account and its associated private key:

We can create this by filling in the account alias, key alias, and corresponding password, or by clicking on the “Restore Wallet “below to Restore the previous account (if it was previously backed up) :

Click “Register”, it will be created successfully, and enter the management page:

Pay attention to its address is: http://127.0.0.1:9888/dashboard

If we look at the config file config.toml, we can see it there:

fast_sync = true
db_backend = "leveldb"
api_addr = "0.0.0.0:9888"
chain_id = "solonet"
[p2p]
laddr = "TCP: / / 0.0.0.0:46658"
seeds = ""
Copy the code

Notice the API_addr, which is the address for dashboard and web-API. The BaseConfig.ApiAddress will fetch the corresponding value from the configuration file after startup.

config/config.go#L41-L85

type BaseConfig struct {
    // ...
    ApiAddress string `mapstructure:"api_addr"`
    // ...
}
Copy the code

Then, on startup, the original Web API and dashboard use that address and open dashboard in a browser.

There is a strange problem, however, that no matter what the value is, the browser always opens http://localhost:9888. Why is that? Because it’s dead in the code.

In the code, http://localhost:9888 appears in three places, one of which is used to represent the dashboard’s access address, located in node/ Node.go:

node/node.go#L33-L37

const (
	webAddress               = "http://127.0.0.1:9888"
	expireReservationsPeriod = time.Second
	maxNewBlockChSize        = 1024
)
Copy the code

The webAddress here is only used when opening the browser to display dashboard from code:

node/node.go#L153-L159

func lanchWebBroser(a) {
	log.Info("Launching System Browser with :", webAddress)
	iferr := browser.Open(webAddress); err ! =nil {
		log.Error(err.Error())
		return}}Copy the code

Compared to the original through the third-party libraries, “github.com/toqueteos/webbrowser” can be activated on the node, called the system default browser, and open the specified url, convenient for the user. (Note that there are many typos in this code, such as Lanch and broser, which have been fixed in later versions)

Another place is for the command line tool bytomcli, which is oddly placed under util/util. Go: util/util.

util/util.go#L26-L28

var (
	coreURL = env.String("BYTOM_URL"."http://localhost:9888"))Copy the code

Why does it belong to BytomCLI? Because the coreURL ends up being used in a ClientCall(…) under the util package. Function to send a request from code to the specified Web API and use it to reply to the information. However, this method is used in the package where bytomCLI resides. If so, coreURL and related functions should be moved to the BytomCLI package.

The third place, with the second like very much, but in the tools/sendbulktx/core/util. Go, it is used for another sendbulktx command line tools:

tools/sendbulktx/core/util.go#L26-L28

var (
	coreURL = env.String("BYTOM_URL"."http://localhost:9888"))Copy the code

Exactly the same, right? In fact, it’s not only here, there’s a bunch of related methods and functions, which are exactly the same, which are copied from the second one.

As for the problems here, I have raised two issues:

  • The dashboard and Web API addresses are written in the config file config.toml, but are also written in code: This is a bit difficult to implement because in the configuration file, 0.0.0.0:9998 is written, but accessing it from a browser or command line tool requires a specific IP address (not 0.0.0.0), otherwise some functions will not work properly. In addition, as you will see later in the code analysis, the address of the Web API to which LISTEN corresponds from the environment variable takes precedence over the address in the configuration file. So more research is needed here to fix it properly.

  • There is a lot of duplication in the code associated with reading WebAPI: the official explanation is that the sendBulkTX tool will be separate from the BYTom project in the future, so the code is duplicated, which is acceptable if so.

What information and functionality does Dashboard provide?

Let’s take a quick look at what information and features the original Dashboard offers. Since we are not focusing on these specific features in this article, we will examine them in detail. In addition, a lot of data are not available in the account just created in front. For convenience, I have made some data in advance.

First, the key:

This shows how many keys you currently have, what their aliases are, and displays the master public key. We can create multiple keys by clicking the New button in the upper right corner, but this is not shown here.

Account:

By default, only one asset, BTM, is defined. You can add multiple assets by clicking the New button.

Balance:

It looks like I still have a lot of money.

Trade:

Shows multiple transactions, actually mined on the machine. Since the mined BTM is transferred directly from the system to our account, it can also be regarded as a transaction.

Create a transaction:

We can also create our own transactions like this, transferring one of our holdings (such as BTM) to another address.

No cost output:

The simple understanding is that every transaction associated with me is recorded, with input and output sections, where the output may be the input of another transaction. This shows the output that has not been spent (to calculate how much balance I have left)

View core status:

Define access control:

Backup and restore operations:

! [dashboard-backup]((https://i.loli.net/2018/07/23/5b552b742b739.png)

In addition, below the left column of each page, there is information about the type of chain connected (solonet in this case), as well as synchronization and the number of other nodes connected to the current node.

We don’t need to go into detail about the information and functionality shown here, but the nouns that appear here should be noted because they are core concepts. When we study the core functions of the original internal blockchain in the future, they are actually all around them. Each of these concepts may require one or more articles.

We’re going to focus on the technical implementation level today, and we’re going to start coding time.

How did biyuan implement the HTTP server?

First let’s start from the original node and work our way to where the HTTP service is started:

cmd/bytomd/main.go#L54-L57

func main(a) {
	cmd := cli.PrepareBaseCmd(commands.RootCmd, "TM", os.ExpandEnv(config.DefaultDataDir()))
	cmd.Execute()
}
Copy the code

cmd/bytomd/commands/run_node.go#L41-L54

func runNode(cmd *cobra.Command, args []string) error {
	// Create & start node
	n := node.NewNode(config)
	if_, err := n.Start(); err ! =nil {
    // ..
}
Copy the code

node/node.go#L169-L180

func (n *Node) OnStart(a) error {
	// ...
	n.initAndstartApiServer()
	// ...
}
Copy the code

InitAndstartApiServer:

node/node.go#L161-L167

func (n *Node) initAndstartApiServer(a) {
    / / 1.
	n.api = api.NewAPI(n.syncManager, n.wallet, n.txfeed, n.cpuMiner, n.miningPool, n.chain, n.config, n.accessTokens)

    / / 2.
	listenAddr := env.String("LISTEN", n.config.ApiAddress)
    env.Parse()
    
    / / 3.
	n.api.StartServer(*listenAddr)
}
Copy the code

As you can see, the method is divided into three parts:

  1. Construct one by passing in a number of parametersAPIObject. When you go in, you’ll see a lot of URL-related configuration.
  2. Let’s get it from the environmentLISTENThe corresponding value, if not, is used againconfig.tomlSpecified in theapi_addrValue as the entry address of the API service
  3. Actually starting the service

Since 2 is relatively simple, we will examine 1 and 3 in detail next.

Find the api.NewAPI method corresponding to 1:

api/api.go#L143-L157

func NewAPI(sync *netsync.SyncManager, wallet *wallet.Wallet, txfeeds *txfeed.Tracker, cpuMiner *cpuminer.CPUMiner, miningPool *miningpool.MiningPool, chain *protocol.Chain, config *cfg.Config, token *accesstoken.CredentialStore) *API {
	api := &API{
		sync:          sync,
		wallet:        wallet,
		chain:         chain,
		accessTokens:  token,
		txFeedTracker: txfeeds,
		cpuMiner:      cpuMiner,
		miningPool:    miningPool,
	}
	api.buildHandler()
	api.initServer(config)

	return api
}
Copy the code

It is mainly to pass in the various parameters hold, for later use. Then there’s api.buildHandler to configure the paths and handlers for each function point, and api.initServer to initialize the service.

Into the API. The buildHandler (). This method is a bit long, so break it down into several parts:

api/api.go#L164-L244

func (a *API) buildHandler(a) {
    walletEnable := false
    m := http.NewServeMux()
Copy the code

It appears that the HTTP service uses the HTTP package that comes with Go.

If the user’s wallet function is not disabled, the function points related to the wallet will be configured (such as account, transaction, key, etc.) :

	ifa.wallet ! =nil {
		walletEnable = true

		m.Handle("/create-account", jsonHandler(a.createAccount))
		m.Handle("/list-accounts", jsonHandler(a.listAccounts))
		m.Handle("/delete-account", jsonHandler(a.deleteAccount))

		m.Handle("/create-account-receiver", jsonHandler(a.createAccountReceiver))
		m.Handle("/list-addresses", jsonHandler(a.listAddresses))
		m.Handle("/validate-address", jsonHandler(a.validateAddress))

		m.Handle("/create-asset", jsonHandler(a.createAsset))
		m.Handle("/update-asset-alias", jsonHandler(a.updateAssetAlias))
		m.Handle("/get-asset", jsonHandler(a.getAsset))
		m.Handle("/list-assets", jsonHandler(a.listAssets))

		m.Handle("/create-key", jsonHandler(a.pseudohsmCreateKey))
		m.Handle("/list-keys", jsonHandler(a.pseudohsmListKeys))
		m.Handle("/delete-key", jsonHandler(a.pseudohsmDeleteKey))
		m.Handle("/reset-key-password", jsonHandler(a.pseudohsmResetPassword))

		m.Handle("/build-transaction", jsonHandler(a.build))
		m.Handle("/sign-transaction", jsonHandler(a.pseudohsmSignTemplates))
		m.Handle("/submit-transaction", jsonHandler(a.submit))
		m.Handle("/estimate-transaction-gas", jsonHandler(a.estimateTxGas))

		m.Handle("/get-transaction", jsonHandler(a.getTransaction))
		m.Handle("/list-transactions", jsonHandler(a.listTransactions))

		m.Handle("/list-balances", jsonHandler(a.listBalances))
		m.Handle("/list-unspent-outputs", jsonHandler(a.listUnspentOutputs))

		m.Handle("/backup-wallet", jsonHandler(a.backupWalletImage))
		m.Handle("/restore-wallet", jsonHandler(a.restoreWalletImage))
	} else {
		log.Warn("Please enable wallet")}Copy the code

The wallet feature is enabled by default, so how can users disable it? To do this, add this section of code to the config file config.toml:

[wallet]
disable = true
Copy the code

In the previous code, we used a lot of code like m.handler (“/create-account”, jsonHandler(a.createAccount)) to configure function points. What does that mean?

  1. /create-account: The path to the function, such as this, the user needs to use the address in the browser or on the command linehttp://localhost:9888/create-accountTo access the
  2. a.createAccount: used to process user access, such as getting the data provided by the user, and then returning some data to the user, will be explained in detail below
  3. jsonHandler: is an intermediate layer that converts the JSON data sent by the user into the Go type parameters required by the handler in step 2, or converts the Go data returned by step 2 into JSON for the user
  4. m.Handle(path, handler): is used to map the function point path to the corresponding handler

Here is the code for the jsonHandler in step 3:

api/api.go#L259-L265

func jsonHandler(f interface{}) http.Handler {
	h, err := httpjson.Handler(f, errorFormatter.Write)
	iferr ! =nil {
		panic(err)
	}
	return h
}
Copy the code

It uses HTTPJSON, which is a package provided in the original code, located in net/ HTTP/httpJSON. Its main function is to add a layer of translation between HTTP access and Go functions. Usually, when users interact with API through HTTP, they send and receive JSON data. However, we define the Go function in the handler in step 2. Through HTTPJSON, the two functions can be converted automatically, so that we do not need to consider JSON and HTTP protocol related problems when writing Go code. Accordingly, the handler in Step 2 has some formatting requirements to work with JsonHTTP, as detailed in the comments here: net/ HTTP /httpjson/doc.go# l3-l40. Because httpJSON involves more code, here is not detailed, there is a chance to open a special article.

Then let’s look at the code for a.createAccount in step 2:

api/accounts.go#L16-L30

func (a *API) createAccount(ctx context.Context, ins struct { RootXPubs []chainkd.XPub `json:"root_xpubs"` Quorum int `json:"quorum"` Alias string `json:"alias"` }) Response { acc, err := a.wallet.AccountMgr.Create(ctx, ins.RootXPubs, ins.Quorum, ins.Alias) if err ! = nil { return NewErrorResponse(err) } annotatedAccount := account.Annotated(acc) log.WithField("account ID", annotatedAccount.ID).Info("Created account") return NewSuccessResponse(annotatedAccount) }Copy the code

We won’t go into the details of this function here, but rather notice its format, since it needs to be used in conjunction with the jsonHandler. The format is basically, the first argument is Context, the second argument is a parameter that can be converted from JSON data, and the return value is a Response and an Error, but all four are optional.

Let’s go back to api.buildHandler() and continue:

	m.Handle("/", alwaysError(errors.New("not Found")))
	m.Handle("/error", jsonHandler(a.walletError))

	m.Handle("/create-access-token", jsonHandler(a.createAccessToken))
	m.Handle("/list-access-tokens", jsonHandler(a.listAccessTokens))
	m.Handle("/delete-access-token", jsonHandler(a.deleteAccessToken))
	m.Handle("/check-access-token", jsonHandler(a.checkAccessToken))

	m.Handle("/create-transaction-feed", jsonHandler(a.createTxFeed))
	m.Handle("/get-transaction-feed", jsonHandler(a.getTxFeed))
	m.Handle("/update-transaction-feed", jsonHandler(a.updateTxFeed))
	m.Handle("/delete-transaction-feed", jsonHandler(a.deleteTxFeed))
	m.Handle("/list-transaction-feeds", jsonHandler(a.listTxFeeds))

	m.Handle("/get-unconfirmed-transaction", jsonHandler(a.getUnconfirmedTx))
	m.Handle("/list-unconfirmed-transactions", jsonHandler(a.listUnconfirmedTxs))

	m.Handle("/get-block-hash", jsonHandler(a.getBestBlockHash))
	m.Handle("/get-block-header", jsonHandler(a.getBlockHeader))
	m.Handle("/get-block", jsonHandler(a.getBlock))
	m.Handle("/get-block-count", jsonHandler(a.getBlockCount))
	m.Handle("/get-difficulty", jsonHandler(a.getDifficulty))
	m.Handle("/get-hash-rate", jsonHandler(a.getHashRate))

	m.Handle("/is-mining", jsonHandler(a.isMining))
	m.Handle("/set-mining", jsonHandler(a.setMining))

	m.Handle("/get-work", jsonHandler(a.getWork))
	m.Handle("/submit-work", jsonHandler(a.submitWork))

	m.Handle("/gas-rate", jsonHandler(a.gasRate))
	m.Handle("/net-info", jsonHandler(a.getNetInfo))
Copy the code

You can see the definition of various functions, mainly related to block data, mining, access control, etc., which will not be detailed here.

To continue:

	handler := latencyHandler(m, walletEnable)
	handler = maxBytesHandler(handler)
	handler = webAssetsHandler(handler)
	handler = gzip.Handler{Handler: handler}

	a.handler = handler
}
Copy the code

Here we package the previously defined function points into a handler, and then layer upon layer around it, adding more functionality:

  1. latencyHandler: I can’t say exactly what it does right now, I’ll fill in later
  2. maxBytesHandler: Prevents data submitted by users from being too large. The current value is about10MB. For in addition tosigner/sign-blockOther urls are valid
  3. webAssetsHandler: Provides dashboard related front-end page resources (such as web pages, images, etc.) to users. Probably for performance and convenience, front-end files are obfuscated and embedded as stringsdashboard/dashboard.goIn, the real code is in another projectGithub.com/Bytom/dashb…We’ll look at that later
  4. gzip.Handler: Indicates whether HTTP clients are supportedgzipAnd, where supported, transmit data using GZIP compression

Then let’s go back to the main line and look at the last api.initServer(config) called in the previous NewAPI:

api/api.go#L89-L122

func (a *API) initServer(config *cfg.Config) {
	// The waitHandler accepts incoming requests, but blocks until its underlying
	// handler is set, when the second phase is complete.
	var coreHandler waitHandler
	var handler http.Handler

	coreHandler.wg.Add(1)
	mux := http.NewServeMux()
	mux.Handle("/", &coreHandler)

	handler = mux
	if config.Auth.Disable == false {
		handler = AuthHandler(handler, a.accessTokens)
	}
	handler = RedirectHandler(handler)

	secureheader.DefaultConfig.PermitClearLoopback = true
	secureheader.DefaultConfig.HTTPSRedirect = false
	secureheader.DefaultConfig.Next = handler

	a.server = &http.Server{
		// Note: we should not set TLSConfig here;
		// we took care of TLS with the listener in maybeUseTLS.
		Handler:      secureheader.DefaultConfig,
		ReadTimeout:  httpReadTimeout,
		WriteTimeout: httpWriteTimeout,
		// Disable HTTP/2 for now until the Go implementation is more stable.
		// https://github.com/golang/go/issues/16450
		// https://github.com/golang/go/issues/17071
		TLSNextProto: map[string]func(*http.Server, *tls.Conn, http.Handler){},
	}

	coreHandler.Set(a)
}
Copy the code

This approach is not suitable for this article because it is more about the HTTP layer than the focus of this article. The interesting thing is that the method creates an HTTP. Server provided by Go, and stuffs the handler that we configured so painfully earlier into it.

So here’s the boot up. We can finally go back to the latest initAndstartApiServer method, remember its third block? N.api. StartServer(*listenAddr);

api/api.go#L125-L140

func (a *API) StartServer(address string) {
	// ...
	listener, err := net.Listen("tcp", address)
	// ...
	go func(a) {
		iferr := a.server.Serve(listener); err ! =nil {
			log.WithField("error", errors.Wrap(err, "Serve")).Error("Rpc server") ()}}}Copy the code

Net. Listen is used to Listen to the incoming Web API address. After receiving the corresponding listener, it is passed to the Serve method we created in the previous HTTP.

This piece of code analysis was a pain to write, mainly because the Web API here covers almost all the functionality that was originally provided. There’s a lot more to the HTTP protocol. At the same time, because of the exposure of the interface, here is prone to security risks, so there is a lot of code involved in user input, security checks and so on. These things are important, of course, but boring from a code reading perspective, unless we’re looking at security.

The main task of this article is to investigate how Bihara provides HTTP services, and what bihara does in terms of security will be analyzed later.

What front-end framework does Dashboard use?

Compared to the original front-end code is in another independent project: https://github.com/Bytom/dashboard

Instead of going into the details of the code, we’ll just look at what front-end frameworks it uses and get a general idea.

Through github.com/Bytom/dashb… We can get a rough idea of what the front-end uses than the original:

  1. Build tools: Leverage directlynpmtheScripts
  2. Front-end framework:React + Redux
  3. CSS:bootstrap
  4. JavaScript: ES6
  5. The HTTP request:fetch-ponyfill
  6. Resource packaging:webpack
  7. Testing:mocha

In what way is the data on the Dashboard retrieved from the background?

Take the account-related code for example:

src/sdk/api/accounts.js#L16

const accountsAPI = (client) = > {
  return {
    create: (params, cb) = > shared.create(client, '/create-account', params, {cb, skipArray: true}),

    createBatch: (params, cb) = > shared.createBatch(client, '/create-account', params, {cb}),

    // ...

    listAddresses: (accountId) = > shared.query(client, 'accounts'.'/list-addresses', {account_id: accountId}),
  }
}
Copy the code

These functions mainly send HTTP requests to the Web API interface created using GO and get the corresponding reply data through the methods provided in the Fetch – Ponyfill library. They are called in the React component and the data returned is used to populate the page.

Again, more details will not be covered in this article.

Finally, after this extensive analysis, I think I have some basic impressions of how the original Dashboard was made. The rest is to conduct a detailed study of its functions in the future.