The idea of this article is introduced at: juejin.cn/post/684490… .

You can look at the implementation before you look at the code;

Just after I posted the thoughts of the most popular nuggets article collection comment analysis, I received a lot of nuggets friends like and read, which made me more confident to record the whole implementation process step by step, so that interested front end children can also be familiar with the front end. Although the whole function is simple, but also calculated to achieve the whole process, I hope to help the front of the children’s shoes to comb their own ideas; There have been a lot of doubts before, how the database and the back end connection? How can the backend fetch data from the database? How is the data returned in JSON format? How does the front end use the interface? Can I write my own interface? How does the front-end resolve the interface? How to use vue without scaffolding? How to combine jquery and Vue? How do bootstrap and Vue combine together?

Some problems that may have bosses and veteran, is very simple, may despise the way, but the entry soon, or want to know how in front and back side of people, it is difficult to understand, it is easy to run wide, or don’t have the courage to learn, because to learn, the more will find knowledge more and more involved, the more knowledge, the less you will feel, Then there is fear, and finally nothing;

First of all, I would like to state that this tutorial is to help interested children’s shoes to comb or assist, the code is not too advanced, nor is it too optimized, in a simple way to make it, is to learn to communicate, hoping that the world “s sole animals will not look disgusted

First step

Make sure you have installed the nodejs and mogodb, if you are a MAC, you can take a look at this tutorial to help you quickly build environment, believe that you can handle it: blog.csdn.net/byc233518/a…

Frame structures,

Next, use NPM to start a project called juejinSpider. I won’t go into the details, there are a lot of tutorials, don’t use scaffolding or anything like that, just simple NPM init;

After the initial completion of the related directory structure, put a piece of my project structure here, obtain not so standard, but enough for this project can be:

File directory parsing

1. Mongodb configuration

The mongodb folder stores mongodb configuration files:

Js is the database connection file, juejinSchema.js is the rebuilt Schema structure, model.js is the curD operation on the database.

2, node_modules needless to say, is the dependence of NPM install, here we use the express, mongoose, directing, superagent

3. Back-end interface processing and crawler execution files are stored in server directory:

App. js is the main file of the back-end service, listening to the request of 5000 port. I put the conctroller related to the request in a separate conctroller file. Spider. js is responsible for collecting the gold mining interface information, processing and storing it to mongodb;

4. The View folder is mainly the front view directory:

Lib folder to store the main is to use some library files, mainly include: jquery, vue, axios, ecahrts, bootstrap, masonry (waterfall flow plug-in), imagesloaded (image loading plug-ins); The use of each will be mentioned below;

In the js folder, main.js is the main JS file, where the front-end page rendering and interface requests are implemented, not to mention index.html, and the main view file, which can be opened directly without the need for packaging tools, because I don’t want to make it too complicated.

Not to mention package.json, NPM init generates a file that contains all the dependencies you need;

Code on

1. Mongodb configuration file

The first is the mongodb related configuration dbconfig.js, the main code is as follows:

Here mainly configured the mongo’s links, as well as the connection state, my mongo’s default port is 27017, in addition to create a new database, here I named juejin, building related operations, please refer to: blog.csdn.net/byc233518/a… ; In addition, it is recommended to download a mongodb visualization tool, MongoBooster;

var mongoose = require('mongoose'), DB_URL = 'mongodb://localhost:27017/juejin'; var db = mongoose.connect(DB_URL,{useMongoClient:true}); On ('connected',function () {console.log(" mongoose connection open to "+DB_URL); }); On ('error',function (err) {console.log(" mongoose connection erro "+err); }); Mongoose.connection. on('disconnected',function () {console.log(" mongoose connection disconnected "); }); module.exports = mongoose;Copy the code

2. Schema design

The next step is to design the Schema structure, because the data from the mining interface has a lot of useless data (for me anyway, I just want a few data 😂), the main code is as follows: Here, in order to prevent repeated data collection, I set the original link of the article as a unique value. However, in the process of collection, it was found that empty value still existed, but fortunately, it did not affect the overall collection, and it has not been optimized yet.

Var mongoose = require('./dbconfig.js'), // introduce mongodb configuration file Schema = mongoose. Var JuejinSchema = new Schema({author:String, // category:{author:String, // category id:String, Name :String, title:String}, collectionCount:Number, // Count:Number, OriginalUrl :{type:String, type: unique: originalUrl:{type:String, type: unique: Screenshot: true}, // screenshot:String; module.exports = mongoose.model('juejin',JuejinSchema);Copy the code

Curd operation

Here I extract it separately and put it in model.js file. Although it can be further abstracted out a DAO file, it is implemented in a file because the project is not very large. Here I only implement data insertion and query operations. There is no specific implementation for delete and update; Among them, the insert operation mainly deals with the crawler to obtain data and write it to the database, and the query mainly deals with the front end to display relevant content. The main implementation code is as follows:

var Juejin = require('./juejinSchema.js'); / / introduce Schema file / / data insertion function inserts (the conditions, the callback) {the conditions = the conditions | | {}; Juejin. Create (the conditions, the callback)} / / data query function find (the conditions, the callback) {the conditions = the conditions | | {}; Juejin.find(conditions,callback); } / / data update function update (the conditions, the update) {Juejin. Update (the conditions, the update function (err, res) {if (err) console.log('Error' + err); else console.log('Res:' + res); })} function del(conditions) {juejin.remove (conditions,function (err,res) {if(err) console.log('Error' + err);  else console.log('Res:' + res); }) } module.exports = { find:find, del:del, update:update, insert:insert };Copy the code

4. Compilation of spider files

Then began to “crawler” the main file preparation, the initial time I only consider in the background manual execution of each crawl activity, and did not think of the front of the data crawl operation; Later, I decided that it would be better to automate all the collection, so I rebuilt spider.js. At present, this method is mainly to respond to the front-end request and collect data and insert data;

var superagent = require('superagent'); Var model = require('.. /mongodb/model.js'); // Introduce the mongodb model // collect the main function of mining hot text, receive parameter sort: Spider = function (sort, callback) {var limit = 100; Var categroyList = [{"id": {"id": {"id": {"id":}} "5562 b410e4b00c57d9b94a92", "name" : "android"}, {" id ":" 5562 b415e4b00c57d9b94ac8 ", "name" : "front end"}, {" id ": {"id": "product"}, {"id": "product"}, {"id": "product"}, {"id": "product"}, {"id": "product"}, {"id": "product"}, {"id": "product"}, "5562B41DE4B00C57D9b94B0f ", "name":" Design "}, {"id": "5562B422e4B00C57D9B94B53 ", "name":" Tool resource "}, {"id": "tool resource"}, "5562B428e4b00c57D9b94b9d ", "name":" read "}, "{"id":" 5562B419e4B00C57D9B94ae2 ", "name": "back end"}, "{"id":" back end "}," 57be7c18128Fe1005FA902de ", "name": "ai"}]; for (var i = 0; i < categroyList.length; If (categroyList[I].name === = sort) {var id = categroyList[I].id; break; Var URL = 'https://timeline-merger-ms.juejin.im/v1/get_entry_by_hot? src=web&limit=' + limit + '&category=' + id; End (function (err, res) {if (err) {return err; } var result = res.body; insertTomongoDB(result, callback); }); }; Var data = val.d.tryList; var data = val.d.tryList; var data = val.d.tryList; Var insertList = []; for (var i = 0; i < data.length; i++) { var insert = { author: data[i].author, category: { id: data[i].category.id, name: data[i].category.name, title: data[i].category.title }, collectionCount: data[i].collectionCount, commentsCount: data[i].commentsCount, viewsCount: data[i].viewsCount, title: data[i].title, summaryInfo: data[i].summaryInfo, originalUrl: data[i].originalUrl, screenshot: data[i].screenshot }; insertList.push(insert) } model.insert(insertList, callback); // Insert operation}; module.exports = { spiders: spider };Copy the code

Five, back end entrance

Ok, the data acquisition part has been completed, and the back-end interface app.js has been built (don’t think the order is reversed, because I only did the data acquisition part at the beginning, in order to test whether the data can be inserted into mongodb normally). Here I use the Express framework, listening to the request of 5000 ports, and only two interfaces have been written so far. Both are GET requests, one is responsible for querying data, one is responsible for triggering data collection operations; Here I put the main controller in a separate file, controller.js.

var express = require('express'); Var app = express(); / / constructs an instance var $= the require (".. / controllers/controllers. Js'); All ('*', function(req, res, next) {res.header(" access-control-allow-origin ", "*"); res.header("Access-Control-Allow-Headers", "X-Requested-With"); res.header("Access-Control-Allow-Methods","PUT,POST,GET,DELETE,OPTIONS"); Res. The header (" X - Powered By ", '3.2.1); res.header("Content-Type", "application/json; charset=utf-8"); next(); }); App.get ('/ API /getListByCategory',$.list); / / data acquisition interface, need to get category app. Get ('/API/sendSpiderByCategory '$. Send); Var server = app.listen() {var host = server.address().address; var port = server.address().port; console.log('Example app listening at http://%s:%s', host, port); });Copy the code

6. Back-end controller

Now that the main interface to the back end is written, start writing the main controller controller.js. There are only two implementation methods in the controller, one is to get the list of data, the other is to trigger the collection of data; Model.js and spider.js need to be introduced; One is to query the data, the other is to trigger the data collection operation; The main code is as follows:

var model = require('.. /.. /mongodb/model.js'); Var spider = require('.. /spider'); List = function (req,res,next) {var param = req.query.sort; Sort model.find({'category. Name ':param},function (err,doc) {if(err){res.end(err); Res.end (json.stringify (doc)); // Return the data returned by the database; // Return the data returned by the database; }); }; Send = function (req,res,next) {var param = req.query.sort; // Resolve the parameter sort carried by the GET request // trigger the collector to run, Spiders (param,function (err,doc) {if(err){res.end(json.stringify (err)); Return} // If the insertion was successful, return OK res.end(json.stringify ({MSG :'ok'})); }); }; module.exports = { list:list, send:send };Copy the code

Vii. Preliminary operation

At this point you can run the app. Js, run directly on the command line or webstorm, see the console appear such circumstance indicates that the successful, at this point you can input the browser again: http://localhost:5000/api/getListByCategory? Sort =Android if you have no data in your database

Eight, front-end page construction

At this point, the work related to the backend and database has been completed, and the next step is the work of the front end. In the front end, I chose vue+ Bootstrap to quickly build the page. I don’t need to mention the advantages of vue and Boostrap. Recently, I was watching Rick and Morty (A station has resources), although very bloody and violent, but there are A lot of words can make people deeply reflect, ready to brush two, one time is not enough… Off topic, or come back to continue writing); I didn’t use vue’s scaffolding here, because I didn’t think it was necessary. I just quoted a part of it, so I didn’t need to do anything. I don’t need to mention the boostrap reference, because I’m using HTTP requests, so I just want to pull axios together and use the boostrap reference. I see some questions about how to use vue with jquery, so I will continue to use jquery for some time, since it is free (yes, free, use as much as you like), because it involves the use of ICONS, so I will also use Echarts. For the use of echarts, please check the official document. There has been a very detailed explanation, I wouldn’t have launched, here I use a line chart, reference links: echarts.baidu.com/demo.html#l… ; Don’t feel that line chart, bar chart, radar chart, and all kinds of charts are difficult, in fact, follow the official document bit by bit configuration is very simple, as long as you have the data, what kind of chart analysis you can make; Let’s go straight to the view’s index. HTML code:

<! < font style =" text-align: center; text-align: center; text-align: center; text-align: center; href="./lib/bootstrap.min.css"> </head> <body> <div id="main"> <div class="container"> <div class="row panel"> <div < span style =" box-sizing: border-box; color: RGB (74, 74, 74); line-height: 22px; font-size: 13px! Important; word-break: inherit! Important;" </h4> </div> <div class="row "id="menu" > <div class="col-sm-7 col-SM-offset 3"> <ul class="nav nav-pills"> <li role="presentation" class="active"><a href="#" v-on:click="getData('Android')">Android</a></li> <li <a href="#" v-on:click="getData ">< /a></li> <li role="presentation" class="active"><a href="#" v-on:click="getData('iOS')">iOS</a></li> <li role="presentation" class="active"><a href="#" V - on: click = "getData (' products')" > products < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V - on: click = "getData (' design ')" > design < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V - on: click = "getData (' tool resources')" > tool resources < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V - on: click = "getData (' reading ')" > reading < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V - on: click = "getData (' back-end ')" > end < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V - on: click = "getData (' artificial intelligence ')" > artificial intelligence < / a > < / li > < / ul > < / div > < / div > < div class = "row spiders" > < div class = "col - sm - 12" > < h4 Class ="text-center"> < span style =" max-width: 100%; clear: both; !!!!! Don't worry, we're starting to collect, and we're limited to the first 100 entries, </h4> </div> <div class="col-sm-7 col-SM -offset 3"> <ul class="nav nav nav-pills"> <li role="presentation" class="active"><a href="#" v-on:click="spiderData('android')">Android</a></li> <li role="presentation" class="active"><a A href = "#" v - on: click = "spiderData (' front end ')" > front end < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" v-on:click="spiderData('iOS')">iOS</a></li> <li role="presentation" class="active"><a href="#" V - on: click = "spiderData (' products')" > products < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V - on: click = "spiderData (' design ')" > design < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V-on :click="spiderData ">< /a></li> <li role="presentation" class="active"> V - on: click = "spiderData (' reading ')" > reading < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V - on: click = "spiderData (' back-end ')" > end < / a > < / li > < li role = "presentation" class = "active" > < a href = "#" V - on: click = "spiderData (' artificial intelligence ')" > artificial intelligence < / a > < / li > < / ul > < / div > < / div > < div class = "row" > < div id = "line" style = "width: 1200 px; height: 600px"></div> </div> <div style="height: 30px"></div> <div class="row" id="masonry"> <div class="col-sm-6 col-md-4 box" v-for="item in articleList"> <div class="thumbnail"> <img :src="item.screenshot" alt=""> <div class="caption"> <h3><a :href="item.originalUrl">{{item.title}}</a></h3> <p>{{item.summaryInfo}}</p> </div> </div> </div> </div> <div><a V - on: click = "goTop ()" href = "#" > back to top < / a > < / div > < / div > < / div > < script SRC = ". / lib/jquery. Min. Js "> < / script > < script src="./lib/bootstrap.min.js"></script> <script src="./lib/vue.js"></script> <script src="./lib/axios.min.js"></script> <script src="./lib/echarts.min.js"></script> <script src="./lib/masonry-docs.min.js"></script> <script src="./lib/imagesloaded.pkgd.min.js"></script> <script src="./js/main.js"></script> </body> </html>Copy the code

As a matter of fact, you will find that I have used two files, for example, pcro-doces.min.js and imagesload.pkgd.min.js. These two files are mainly used to display the article in the way of waterfall flow. At the same time, overlapping may occur when the image is not loaded. So I introduced imagesLoade to judge whether the image is loaded normally, if the normal load after the waterfall flow display; The bootstrap the styles used for: v3.bootcss.com/components/… ; All imported files can be pulled from my Git: portal

Nine, front-end logic implementation

The last part is the implementation of the front-end page logic, mainly in main.js, where vue and jquery syntax are mixed. If you are a code neat person, please don’t get excited. I just want to use the two at the same time, which does not violate the original intention of VUE, the main code is as follows:

$(document).ready(function () {var vm = new vue ({el: '#main', data: {articleList: [], sort: Mounted () {"use strict"; // Mounted () {"use strict"; This. GetData (' front end '); $('.spider').css('display', 'none'); }, methods: {// Initialize line chart initChart: function (obj) {var options = {// Line chart title: {text: 'Hottest nugget in history'}, // Prompt component box, coordinate axis trigger, mainly used in bar charts, line charts, etc using category axis. Tooltip: {trigger: 'axis'}, // legend: {data: [' favorites ', 'comments ',' view ']}, // Grid: {left: '3%', right: '4%', bottom: '3%', containLabel: true // Whether the grid field contains the scale label of the coordinate axis. }, // Toolbar. Built-in export image, data view, dynamic type switch, data area zoom, reset five tools. // The dataZoom component is used for area scaling, so that you can freely focus on detailed data information, either overview of the data as a whole, or remove the effects of outliers. {show: true, realtime: true, start: 0, end: 10, start: 0, end: 10, start: 0, end: 10, start: 0}, 'category', // category axis boundaryGap: false, // White space policy on both sides of the axis, the setting and performance of the category axis and non-category axis are different. AxisLabel: {// the X-axis label is displayed as a line of 8 words, to keep text from overlapping interval: 0, formatter: function (value) { var ret = ""; Var maxLength = 8; Var valLength = value. Length; Var rowN = math.ceil (valLength/maxLength); var rowN = math.ceil (valLength/maxLength); If (rowN > 1) {for (var I = 0; i < rowN; i++) { var temp = ""; Var start = I * maxLength; Var end = start + maxLength; Temp = value. Substring (start, end) + "\n"; temp = value. ret += temp; } return ret; } else { return value; YAxis: [{type: 'value', name: 'favorites'}, {type: 'value', name:' favorites'}, {type: 'value', name: 'views'}], / / data source series: [{name:' collection number, type: 'the line', / / stack: 'total' data: obj. Collect}, {name: 'comments', type: 'line', // stack:' total ', data: obj.comment}, {name: 'browse ', yAxisIndex: 1, type: 'line', // stack:' total ', data: obj.view } ] }; var ele = document.getElementById('line'); Var myChart = echarts.init(ele); // Initialize a chart instance mychart.setoption (options); GetData: function (val) {var self = this; self.sort = val; / / using axios request axios. Get (' http://localhost:5000/api/getListByCategory? sort=' + self.sort) .then(function (response) { var data = response.data; if (data.length <= 0) { $('.spider').css('display', 'block'); $('#menu').css('display', 'none'); Alert (' data does not exist in database, please query after collection '); } self.articleList = data; var arryCollect = [], arryComment = [], arryView = [], arryTitle = []; for (var i = 0; i < data.length; i++) { arryCollect.push(data[i].collectionCount); arryComment.push(data[i].commentsCount); arryView.push(data[i].viewsCount); arryTitle.push(data[i].title) } var obj = { collect: arryCollect, comment: arryComment, view: arryView, title: arryTitle }; console.log(obj); self.initChart(obj); self.loadInfo(); }); }, // loadInfo: function () {var $container = $('# selector for navigation '); $container.imagesLoaded(function () { setTimeout(function () { $container.masonry({ itemSelector: '.box' }); }, 1000)})}, // spiderData: function (val) {var self = this; / / using axios request axios. Get (' http://localhost:5000/api/sendSpiderByCategory? sort=' + val) .then(function (response) { if (response.data.msg === 'ok') { $('.spider').css('display', 'none'); $('#menu').css('display', 'block'); Alert (' Data collection succeeded '); self.getData(val); } }) }, // goTop:function () { // this.click(function (e) { // e.preventDefault(); // $(document.body).animate({scrollTop: 0}, 800); / /}); / /}}}); });Copy the code

10. You’re done

So far, the whole project is basically completed, you can directly open index.html to view, the initial is no data, will remind you to collect data, collection will be successful, and then refresh the page will find that the data has been; (At this point, app.js needs to run in the background, do not close)

At the end

If you have any problems during the construction process, you can leave me a message or directly add my wechat. I hope we can communicate with each other. I am not a big boss, so I may not be able to solve the problems you raised, but we can discuss about it. I hope that through this article can help a person to build the front and back end of children’s shoes, although simple, but the front and back end and database are used; Just like all code starts with “Hello World”, once you’re done with “Hello World”, you can expand indefinitely;

I revised the whole Git project again today and added more comments to facilitate more children’s understanding. The project address is github.com/gengchen528… If you are interested in a star, don’t be stingy. Fork is also ok, haha ~~

This article pure hand, I hope to respect my results, if you want to reprint, please contact me, thank you