This is the 8th day of my participation in the November Gwen Challenge. Check out the event details: The last Gwen Challenge 2021

1.1 What is a File System

How is file data stored

1.2 Distributed File System

  • A computer has limited storage capacity and limited concurrent throughput. How to improve performance?

  • A ton of goods I want to deliver to Turpan:

    • 1 person transport, can’t imagine
    • Fifty people, too hard;
    • 500 people, everyone is easy;

  • Is this distributed?

  • Distributed: Different service modules are deployed on different servers or the same service module is divided into multiple sub-services and deployed on different servers. Solve high concurrency problems;

  • Cluster: The same service is deployed on multiple servers to improve system availability

  • Such as:

    • The small restaurant had only one chef, who cut vegetables, washed vegetables and prepared materials. More and more guests, a chef is too busy to come over, can only invite a chef, two chefs can fry, that is, the role of two chefs is the same, so that the relationship between two chefs is “cluster”;
    • In order to let the chef concentrate on cooking, cooking to the extreme, and invited the division responsible for cutting vegetables, prepare materials and other work. The relationship between chef and chef is “distributed”;
    • A chef is too busy to provide two ingredients for two chefs, and another chef is invited. The relationship between the two chefs is “cluster”.

1.3 Mainstream Distributed File Systems

1.3.1 HDFS

  • (HadoopDistributedFileSystem) Hadoop distributed file system;
  • Highly fault-tolerant systems suitable for deployment on inexpensive machines;
  • Can provide high throughput data access, very suitable for large-scale data applications;
  • The HDFS adopts a master/slave structure. One HDFS consists of a Name node and N data nodes.
  • The name node stores metadata. A file is divided into N copies and stored on different data nodes.

1.3.2 GFS

  • GoogleFileSystem
  • Scalable distributed file systems for large, distributed applications that access large amounts of data;
  • Run on cheap common hardware, can provide fault tolerance;
  • It can provide a large number of users with overall high performance services;
  • GFS adopts a master-slave structure. A GFS cluster consists of a master and a large number of Chunkservers.
  • A file is divided into several blocks and stored in multiple block servers

1.3.3. FastDFS

  • Written and open source by Yu Qing, senior architect of Taobao;

  • Customized for the Internet, fully consider redundant backup, load balancing, linear expansion and other mechanisms, and pay attention to high availability, high performance indicators, FastDFS is easy to build a set of high-performance file server cluster to provide file upload, download and other services;

  • HDFS and GFS are common file systems. Their advantages are good development experience, high system complexity and mediocre performance.

  • In contrast, dedicated distributed file systems have poor experience, but low complexity and high performance. FastDFS is especially suitable for small files such as pictures and videos, because fastDFS does not split files, so there is no overhead of file merging.

  • Network communication socket, fast speed.

1.4 Working Principles

  • FastDFS includes TrackerServer and StorageServer.
  • Client request TrackerServer for file upload and download;
  • TrackerServer schedules the StorageServer to complete upload and download.

  • A Tracker

    • It is used for load balancing and scheduling. It manages the storage service (StorageServer), which can be understood as “big steward, tracker and dispatcher”.
    • TrackerServer can be clustered to achieve high availability with a “polling” policy.

  • Storage; Reservoir)

    • Files uploaded by the client are stored on the storage server.
    • A storage cluster is grouped. Each server in a storage group has an equal relationship and synchronizes data to back up data for high availability. Servers in different groups do not communicate with each other.
    • If the storage capacity of each server in the same group is inconsistent, the server with the smallest capacity is selected. Therefore, the software and hardware of the servers in the same group must be consistent.
    • StorageServer connects to all trackerservers in the cluster and periodically reports its status to them, such as the remaining space, file synchronization status, and file upload and download times.

1.5 Upload/Download Mechanism

  • After the client uploads a file, the storage returns the file ID to the client

  • group1/M00/02/11/aJxAeF21O5wAAAAAAAAGaEIOA12345.sh

    • Group name: Specifies the name of the storage group after the file is uploaded. After the file is successfully uploaded, the storage returns the file and the client saves the file by itself.

    • Virtual disk path:

      Storage Specifies the virtual path configured for storage. Storage_path corresponds to disk options.

      Storage_path0 corresponds to M00,

      Storage_path1 corresponds to M01,

    • Data two-level directory:

    Storage A self-created directory on the virtual disk.

    • The file name:

    Different from the upload, is generated with storage according to specific information, which contains: storage server IP address, create time stamp, size, suffix name and other information

2.FastDFS upload and download

2.1 installation

2.1.1 Installing GCC (Required during Compilation)

yum install -y gcc gcc-c++
Copy the code

2.1.2 Installing Libevent (Runtime Requirements)

yum -y	install	libevent
Copy the code

2.1.3 installation libfastcommon

Libfastcommon is official with FastDFS. Libfastcommon contains some of the basic libraries needed to run FastDFS.

  • 1. Upload libfastcommon-master.zip to /opt
Yum install -y unzip unzip libfastcommon.zip unzip libfastcommon.cd	libfastcommon-master
Copy the code
  • 2. Compile
./make.sh
Copy the code

If: make.sh does not have enough permissions, you need permissions (executable rights)

chmod	777	make.sh
Copy the code
  • 3. The installation
./make.sh	install
Copy the code

After libfastcommon is installed, the libfastcommon.so library file is generated in /usr/lib64

  • 4. Copy library files
cd	/usr/lib64
cp	libfastcommon.so	/usr/lib
Copy the code

2.1.4 installing the Tracker

Download fastDFs_v5.05.tar. gz and upload it to /opt

The tar - ZXVF FastDFS_v5.05. Tar. Gzcd	FastDFS
./make.sh
./make.sh	install
Copy the code

The installation is successful. Copy the conf files in the installation directory to /etc/fdfs/

cp	/opt/FastDFS/conf/*	/etc/fdfs/
Copy the code

2.2 configuration

  • The Tracker configuration
vim /etc/fdfs/tracker.conf
Copy the code
# port=22122If the base directory does not exist, you need to create it yourselfmkdir/ home/base_path = / home/fastdfs fastdfs)Copy the code
  • Storage configuration
vim /etc/fdfs/storage.conf
Copy the code

Basic configuration

Group_name =group1 # port= group123000# Heartbeat interval to tracker (seconds) heart_beat_interval=30# store_path =/home/fastdfs # store_path =/home/fastdfs # store_path =/home/fastdfs Multiple store_path # fDFs_storage directories do not exist, #mkdir /home/fastdfs/fdfs_storage store_path0=/home/fastdfs/fdfs_storage #mkdir /home/fastdfs/fdfs_storage store_path0=/home/fastdfs/fdfs_storage The following # store_path1 =... (M01) #store_path2=..... (M02) # tracker_server= IP tracker_server10.1.220.247:22122#tracker_server= #tracker_server=10.1.220.x:22122
Copy the code

2.3 Starting the Service

  • Start the tracker
/usr/bin/fdfs_trackerd	/etc/fdfs/tracker.conf	restart
Copy the code
  • Start the storage
/usr/bin/fdfs_storaged	/etc/fdfs/storage.conf	restart
Copy the code
  • View all running ports:
netstat	-ntlp
Copy the code

2.4 Java Project Construction

Create a Maven project using IDEA

Against 2.4.1 pom. XML

<! Fastdfs Java client -->
<dependency>
<groupId>net.oschina.zcx7878</groupId>
<artifactId>fastdfs-client-java</artifactId>
<version>1.27.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-io</artifactId>
<version>1.3.2</version>
</dependency>
Copy the code

2.4.2 Creating a Configuration File

Fastdfs-client.properties = fastdfs-client.properties = fastdfs-client.properties = fastdfs-client.properties = fastdfs-client.properties = fastdfs-client.properties = fastdfs-client.properties = fastdfs-client.properties

##fastdfs-client.properties
fastdfs.connect_timeout_in_seconds=5
fastdfs.network_timeout_in_seconds=30
fastdfs.charset=UTF-8
fastdfs.http_anti_steal_token=false
fastdfs.http_secret_key=FastDFS1234567890
fastdfs.http_tracker_http_port=80
fastdfs.tracker_servers=10.1.220.247:22122
Copy the code

2.4.3 Uploading files


package test;

import org.csource.common.NameValuePair;
import org.csource.fastdfs.*;



/ * * *@Description: File upload */
public class TestUpload{
public static void main(String[] args){
try{
// Load the configuration file
ClientGlobal.initByProperties("config/fastdfs-client.properties");

// Create a tracker client
TrackerClient trackerClient=new TrackerClient();
// Get the connection service of the tracker through the tracker client and return it
TrackerServer trackerServer=trackerClient.getConnection();
// Declare the storage service
StorageServer storageServer=null;
// Define the storage client
StorageClient1 client=new StorageClient1(trackerServer,
storageServer);

// Define file meta information
NameValuePair[] list=new NameValuePair[1];
list[0] =new NameValuePair("fileName"."1.jpg");

String fileID=client.upload_file1("F:\\img\\1.jpg"."jpg",list);
System.out.println("fileID="+fileID);
//group1/M00/00/00/CgHc918f8l6AFYp0AAWICfQnHuk889.jpg
/* group1: a server is a group M00: store_path0---->/home/fastdfs/fdfs_storage/data 00/00: two-level data directory */
trackerServer.close();
}catch(Exceptione){ e.printStackTrace(); }}}Copy the code

2.4.4 Querying Files

package test;


import org.csource.fastdfs.*;


/ * * *@Description: File query */
public class TestQuery{
public static void main(String[] args)throws Exception{

// Load the configuration file
ClientGlobal.initByProperties("config/fastdfs-client.properties");

// Create a tracker client
TrackerClient trackerClient=new TrackerClient();
// Get the connection service of the tracker through the tracker client and return it
TrackerServer trackerServer=trackerClient.getConnection();
// Declare the storage service
StorageServer storageServer=null;
// Define the storage client
StorageClient1 client=new StorageClient1(trackerServer,
storageServer);

FileInfo fileInfo=
client.query_file_info1("group1/M00/00/00/CgHc918f8l6AFYp0AAWICfQnHuk889.jpg");
if(fileInfo! =null)
System.out.println("fileInfo="+fileInfo);
else
System.out.println("Check no this file!"); trackerServer.close(); }}Copy the code

2.4.5 Downloading Files

packagetest;


importorg.csource.fastdfs.*;


importjava.io.File;
importjava.io.FileOutputStream;


/ * * *@Description: File download */
public	class	TestDownload	{
public	static	void	main(String[]	args)	throws	Exception{
// Load the configuration file
ClientGlobal.initByProperties("config/fastdfs-client.properties");
// Create a tracker client
TrackerClient	trackerClient	=	new	TrackerClient();
// Get the connection service of the tracker through the tracker client and return it
TrackerServer	trackerServer	=	trackerClient.getConnection();
// Declare the storage service
StorageServer	storageServer	=	null;
// Define the storage client
StorageClient1	client	=	new	StorageClient1(trackerServer,
storageServer);


byte[]	bytes	=
client.download_file1("group1/M00/00/00/CgHc918f8l6AFYp0AAWICfQnHuk889.jpg");

// Convert an array of bytes to a file via IO
FileOutputStream fileOutputStream=new FileOutputStream(new
File("F:/xxxxxx.jpg"));
fileOutputStream.write(bytes);
fileOutputStream.close();
trackerServer.close();
System.out.println("Download complete!"); }}Copy the code

3. Project practice

3.1 Setting up a picture server

3.1.1 Installing the Nginx Module (Storage)

1. Upload the fastdfs-nginx-module_v1.16.tar.gz file to /opt 2. Unzip the nginx module

The tar - ZXVF fastdfs - nginx - module_v1. 16. The tar. GzCopy the code

3. Modify the config file to change the /usr/local/path to /usr/

cd	/opt/fastdfs-nginx-module/src
vimconfig
Copy the code

4. Copy mod_fastdfs.conf in the fastdfs-nginx-module/ SRC directory to /etc/fdfs

cp	mod_fastdfs.conf	/etc/fdfs/
Copy the code

5. Modify the/etc/FDFS/mod_fastdfs. Conf

vim /etc/fdfs/mod_fastdfs.conf
Copy the code
base_path=/home/fastdfs
tracker_server=10.1.220.247:22122
#(n个tracker配置n行)
#tracker_server=10.1.220.x:22122Url_have_group_name =true # specify file storage path (store path configured above) store_path0=/home/fastdfs/fdfs_storageCopy the code

6. Copy libfdfsclient.so to /usr/lib

cp	/usr/lib64/libfdfsclient.so	/usr/lib/
Copy the code

7. Create the nginx/client directory

mkdir	-p /var/temp/nginx/client
Copy the code

3.1.2Nginx Installation (Tracker)

1. Upload nginx-1.14.0.tar.gz to /opt (nginx has been installed, skip this step) 2. Decompress: tar-zxvfnginx-1.14.0.tar.gz (nginx installed, skip this step) 3. Install dependency libraries (nginx installed, skip this step)

yuminstall	pcre
yuminstall	pcre-devel
yuminstall	zlib
yuminstall	zlib-devel
yuminstall	openssl
yuminstall	openssl-devel
Copy the code

4. Go to CD /opt/nginx-1.14.0 where the nginx package is decompressed

5. Installation

./configure	\
--prefix=/usr/local/nginx	\
--pid-path=/var/run/nginx/nginx.pid	\
--lock-path=/var/lock/nginx.lock	\
--error-log-path=/var/log/nginx/error.log	\
--http-log-path=/var/log/nginx/access.log	\
--with-http_gzip_static_module	\
--http-client-body-temp-path=/var/temp/nginx/client	\
--http-proxy-temp-path=/var/temp/nginx/proxy	\
--http-fastcgi-temp-path=/var/temp/nginx/fastcgi	\
--http-uwsgi-temp-path=/var/temp/nginx/uwsgi	\
--http-scgi-temp-path=/var/temp/nginx/scgi	\
--add-module=/opt/fastdfs-nginx-module/src
Copy the code

Note: the above will be the temporary file directory is specified for the/var/temp/nginx, need to create temp and nginx under/var directory: mkdir

/var/temp/nginx
Copy the code

6. Make

7. Install: MakeInstall

8. Copy the configuration file

cd/opt/FastDFS/conf cp http.conf mime.types /etc/fdfs/ Whether to override: yesCopy the code

9. Modify the nginx configuration file

cd	/usr/local/nginx/conf/
vim nginx.conf
Copy the code
server	{
listen	80;
server_name	10.1.220.247;

#charset	koi8-r;

#access_log	logs/host.access.log	main;

location	/group1/M00	{
root	/home/fastdfs/fdfs_storage/data;
ngx_fastdfs_module;
}
Copy the code

10. Close nginx and start nginx

pkill	-9	nginx
/usr/local/nginx/sbin/nginx	-c	/usr/local/nginx/conf/nginx.conf
Copy the code

11. Access nginx and view images

http://10.1.220.247 http://10.1.220.247/group1/M00/00/00/CgHc918f8l6AFYp0AAWICfQnHuk889.jpg

3.2 Creating a Front-end page

<%-- upload file --%> <%-- upload file --%> <%-- upload file --%><form action="upload"	method="post"	enctype="multipart/form-data">
<input	type="file"	name="fname">
<br>
<button>submit</button>

</form>
Copy the code

3.3 Setting up Web Services

3.3.1 pom. XML

<packaging>war</packaging>


<dependencies>
<! Servlet dependencies are referenced because there are JSP pages.
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<scope>provided</scope>
<version>2.5</version>
</dependency>
<! -- Page submitted requests, using SpringMVC -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>5.2.7. RELEASE</version>
</dependency>
<! Java fastDFS client tool -->
<dependency>
<groupId>net.oschina.zcx7878</groupId>
<artifactId>fastdfs-client-java</artifactId>
<version>1.27.0.0</version>
</dependency>
<! Upload image to FastDFS using IO tool -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-io</artifactId>
<version>1.3.2</version>
</dependency>
<! I/O tool to save images to web server -->
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.3.1</version>
</dependency>
<! -- to convert Java objects to json strings, note that version 2.7 must be used with spring5.0 or later -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.9.8</version>
</dependency>
</dependencies>


<build>
<plugins>
<plugin>
<groupId>org.apache.tomcat.maven</groupId>
<artifactId>tomcat7-maven-plugin</artifactId>
<configuration>
<port>8001</port>
<path>/</path>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
Copy the code

3.3.2 rainfall distribution on 10-12 web. XML


      
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.jcp.org/xml/ns/javaee"
xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee
http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd"
id="WebApp_ID"version="3.1">
<servlet>
<servlet-name>springMVC</servlet-name>
<servlet-
class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
<init-param>
<param-name>contextConfigLocation</param-name>
<param-value>classpath:spring/spring-mvc.xml</param-value>
</init-param>
</servlet>
<servlet-mapping>
<servlet-name>springMVC</servlet-name>
<url-pattern>/</url-pattern>
</servlet-mapping>
</web-app>
Copy the code

3.3.3 spring – the MVC. XML

<! -- Scan annotated package -->
<context:component-scanbase-package="controller"/>
<! -- Scan comments in controller: @response -->
<mvc:annotation-driven/>
<! -- Upload file parser (specify upload file size limit) -->
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
<! -- Upload file Max: 2GB-->
<propertyname="maxUploadSize"value="2048000000"/>
</bean>
Copy the code

3.3.4 File Entity Classes


    public class FileSystem implements Serializable{
private String fileId;
private String filePath;
private String fileName;
}
Copy the code

3.3.5 control layer

package controller;


import entity.FileSystem;
import org.csource.common.NameValuePair;
import org.csource.fastdfs.*;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.multipart.MultipartHttpServletRequest;

import java.io.File;
import java.util.UUID;

/ * * * /
@Controller
public class FileAction{

/ * * *@paramrequestRequest object * for a multi-part form@returnUpload the JSON object of the file object *@throwsException * * Upload process: *1, save the file to the Web server *2, upload the file from the Web server to FastDFS */


@RequestMapping("upload")
/ / MultipartHttpServletRequest: is it improved version, not only can hold the textInterest, can also be installed picture file informationpublic @ResponseBody FileSystem upload(MultipartHttpServletRequest request)throws Exception{
FileSystem fileSystem=new FileSystem();

/* save the file to the web server */
// From the page request, get the uploaded file object
MultipartFile file=request.getFile("fname");
// Get the original file name from the file object
String oldFileName=file.getOriginalFilename();
// Obtain the suffix 1.jpg from the original file name by string interception
String hou=oldFileName.substring(oldFileName.lastIndexOf(".") +1);
// To avoid overwriting the file with the same name, generate a new file name
String newFileName=UUID.randomUUID().toString()+"."+hou;
// Create a directory for the Web server to save files. (Create a D:/upload directory in advance. Otherwise, the system cannot find the path and will throw different filesOften) File toSaveFile = newFile ("D:/upload/"+newFileName);
// Convert the path to a file
file.transferTo(toSaveFile);
// Get the absolute path of the server
String newFilePath=toSaveFile.getAbsolutePath();

/* upload file from web server to FastDFS*/

ClientGlobal.initByProperties("config/fastdfs-client.properties");
TrackerClient trackerClient=new TrackerClient();
TrackerServer trackerServer=trackerClient.getConnection();
StorageServer storageServer=null;
StorageClient1 client=new StorageClient1(trackerServer,
storageServer);

NameValuePair[] list=new NameValuePair[1];
list[0]=newNameValuePair("fileName",oldFileName);
String fileId=client.upload_file1(newFilePath,hou,list);
trackerServer.close();

// Encapsulates the fileSystem data object
fileSystem.setFileId(fileId);
fileSystem.setFileName(oldFileName);
fileSystem.setFilePath(fileId);	// The image has been uploaded to FastDFS and is accessed by fileId, so fileId is the file path

returnfileSystem; }}Copy the code

3.3.6 Adding the fastDFS Configuration File

Create fastdfs-client.properties in config directory. See 2.4.2

3.3.7 Starting the fastDFS service. The test starts

/usr/bin/fdfs_trackerd	/etc/fdfs/tracker.conf	restart
/usr/bin/fdfs_storaged	/etc/fdfs/storage.conf	restart
/usr/local/nginx/sbin/nginx	-c /usr/local/nginx/conf/nginx.conf
netstat	-ntlp
systemctl	stop	firewalld.service
cd	/home/fastdfs/fdfs_storage/data/
ls
Copy the code

3.6 Typical Errors

Nginx may fail to restart the Linux server:

/usr/local/nginx/sbin/nginx	-c /usr/local/nginx/conf/nginx.conf

nginx:	nginx: [emerg] open() "/var/run/nginx/nginx.pid" failed (2:
No such file or directory)
Copy the code

Nginx configuration file: nginx configuration file:

vim /usr/local/nginx/conf/nginx.conf
pid	/usr/local/nginx/logs/nginx.pid;
Copy the code

Start nginx again, done!