【FastDFS】FastDFS distributed file system installation and use, look at this article enough!!

Writing in the front

There are many friends in the actual work, for how to store files (pictures, videos, audio, etc.) do not have a good solution. Both understand that you can’t store files on disk on a single server and that you need to make copies of your files. If you manually write the copy mechanism of the file, it is too troublesome, which involves redundant copy mechanism, server scheduling, copy detection, server node detection, file copy storage strategy, network environment detection and a series of difficult problems. If you know about Hadoop, you will naturally think of HDFS. Yes, HDFS can store files, divide large files into file blocks for storage, and automatically provide redundant copy mechanism according to configuration. We don’t have to deal with the complex redundant copy mechanism ourselves. There is no need to deal with server scheduling, copy detection, server node detection, file copy storage strategy, network environment detection and a series of difficult problems. However, HDFS is suitable for storing large files because the metadata information of files is all stored in NameNode nodes. If HDFS is used to store a large number of small files, the memory usage of NameNode will increase rapidly, and NameNode nodes will become the bottleneck of the entire cluster environment.

For a variety of reasons, TODAY I recommend a high-performance distributed file system called FastDFS, which also provides a redundant copy mechanism and can handle all of the problems mentioned above. Without further ado, let’s get to the point.

preparation

Storage server: 192.168.50.132 (liuyazhuang132) Environment: CentOS 6.5 User: root Data Directory: /fastdfs (Note: The data directory depends on your disk mount path.

The installation package:

FastDFS v5.05
Libfastcommon-master. zip (a common C library derived from FastDFS and FastDHT)
Fastdfs – nginx – module_v1. 16. The tar. Gz
Nginx – 1.6.2. Tar. Gz
Fastdfs_client_java. _v1. 25. Tar. Gz

Source code address: github.com/happyfish10… Download address: sourceforge.net/projects/fa… Official forum: bbs.chinaunix.net/forum-240-1…

Operations performed by all servers

Compile and install the required dependency packages

# yum install make cmake gcc gcc-c++
Copy the code

2. Install libfastCommon

Libfastcommon-master. zip to /usr/local/src

(2) extract

# cd /usr/local/src/
# unzip libfastcommon-master.zip
# cd libfastcommon-master
Copy the code

# ./make.sh
# ./make.sh install
Copy the code

Libfastcommon is installed in the following location by default.

/usr/lib64/libfastcommon.so
/usr/lib64/libfdfsclient.so
Copy the code

/usr/local/lib = /usr/local/lib = /usr/local/lib

# ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
# ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
# ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
# ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
Copy the code

3. Install FastDFS

(1) Upload or download the FastDFS source package (fastdfs_v5.05.tar. gz) to /usr/local/src

(2) extract

# cd /usr/local/src/
# tar - ZXVF FastDFS_v5.05. Tar. Gz
# cd FastDFS
Copy the code

# ./make.sh
# ./make.sh install
Copy the code

The default installation mode is adopted. The corresponding files and directories are as follows:

A. The service script is in:

/etc/init.d/fdfs_storaged
/etc/init.d/fdfs_tracker
Copy the code

B. The configuration file is in (Sample configuration file)

/etc/fdfs/client.conf.sample
/etc/fdfs/storage.conf.sample
/etc/fdfs/tracker.conf.sample
Copy the code

C. The command tool is in the /usr/bin/directory

fdfs_appender_test
fdfs_appender_test1
fdfs_append_file
fdfs_crc32
fdfs_delete_file
fdfs_download_file
fdfs_file_info
fdfs_monitor
fdfs_storaged
fdfs_test
fdfs_test1
fdfs_trackerd
fdfs_upload_appender
fdfs_upload_file
stop.sh
restart.sh
Copy the code

(4) Because the bin directory of the FastDFS service script is /usr/local/bin, but the actual command is installed in /usr/bin, you can go to /user/bin and run the following command to view FDFS commands:

# cd /usr/bin/
# ls | grep fdfs
Copy the code

# vi fdfs_trackerd
Copy the code

Use the find and replace command to make uniform changes :%s+/usr/local/bin+/usr/bin

# vi fdfs_storaged
Copy the code

Use the find and replace command to make uniform changes :%s+/usr/local/bin+/usr/bin

To change all /usr/local/bin files to /usr/bin, press esc, enter the colon (:), and then enter %s+/usr/local/bin+/usr/bin

Configure the FastDFS tracker (192.168.50.131)

1. Copy the FastDFS tracker sample configuration file

Copy the FastDFS tracker sample configuration file and rename it:

# cd /etc/fdfs/
Copy the code

# cp tracker.conf.sample tracker.conf
Copy the code

2. Edit the tracker configuration file

# vi /etc/fdfs/tracker.conf
Copy the code

The modified contents are as follows:

disabled=false
port=22122
base_path=/fastdfs/tracker
Copy the code

(Keep the default Settings for other parameters. For details about the Settings, see the official documentation: bbs.chinaunix.net/thread-1941…)

3. Create the base data directory (see base_path configuration)

# mkdir -p /fastdfs/tracker
Copy the code

4. Open tracker port in firewall (default: 22122)

# vi /etc/sysconfig/iptables
Copy the code

Add the following port lines:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22122 -j ACCEPT
Copy the code

Restarting the Firewall

# service iptables restart
Copy the code

5. Start the Tracker

# /etc/init.d/fdfs_trackerd start
Copy the code

(The first successful startup will create data and logs directories under /fastdfs/tracker.)

Check to see if FastDFS Tracker has started successfully.

# ps -ef | grep fdfs
Copy the code

Close the Tracker

# /etc/init.d/fdfs_trackerd stop
Copy the code

7. Set the FastDFS tracker to start

# vi /etc/rc.d/rc.local
Copy the code

Add the following

## FastDFS Tracker
/etc/init.d/fdfs_trackerd start
Copy the code

Configuring FastDFS Storage (192.168.50.132)

1. Copy the FastDFS storage sample configuration file

Copy the FastDFS storage sample configuration file and rename it

# cd /etc/fdfs/
Copy the code

# cp storage.conf.sample storage.conf
Copy the code

2. Edit the storage sample configuration file

# vi /etc/fdfs/storage.conf
Copy the code

The modified contents are as follows:

disabled=falsePort = 23000 base_path = / fastdfs/storage store_path0 = / fastdfs/storage tracker_server = 192.168.50.131:22122 http.server_port=8888Copy the code

(Keep the default Settings for other parameters. For details about the Settings, see the official documentation: bbs.chinaunix.net/thread-1941…)

3. Create the base data directory (see base_path configuration)

# mkdir -p /fastdfs/storage
Copy the code

4. Open storage port in firewall (default: 23000)

# vi /etc/sysconfig/iptables
Copy the code

Add the following port lines:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 23000 -j ACCEPT
Copy the code

Restart the firewall:

# service iptables restart
Copy the code

5. Start Storage

# /etc/init.d/fdfs_storaged start
Copy the code

(Data and logs directories will be created under /fastdfs/storage after the first successful startup.)

Check whether FastDFS Storage is started successfully

# ps -ef | grep fdfs
Copy the code

6. Close Storage

# /etc/init.d/fdfs_storaged stop
Copy the code

7, set FastDFS memory boot

# vi /etc/rc.d/rc.local
Copy the code

Add:

## FastDFS Storage
/etc/init.d/fdfs_storaged start
Copy the code

File Upload Test (192.168.50.131)

1. Modify the client configuration file in the Tracker server

# cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf
# vi /etc/fdfs/client.confBase_path = / fastdfs/tracker tracker_server = 192.168.50.131:22122Copy the code

2. Run the following command to upload files

# / usr/bin/fdfs_upload_file/etc/FDFS/client. The conf/usr/local/SRC/FastDFS_v5. 05. Tar. Gz
Copy the code

Returns the ID: group1 / M00/00/00 / wKgEfVUYNYeAb7XFAAVFOL7FJU4. Tar. Gz (can return above file ID, file upload success)

Install nginx on each storage node

Fastdfs-nginx-module function description

FastDFS uses the Tracker server to store files on the Storage server. However, file replication is required between Storage servers in the same group, resulting in synchronization delay. Suppose the Tracker server uploads the file to 192.168.50.132, and the file ID is returned to the client. In this case, the FastDFS storage cluster mechanism synchronizes the file to the same storage group 192.168.50.133. If the client uses the file ID to obtain the file from 192.168.50.133, the file cannot be accessed. Fastdfs-nginx-module redirects files to the source server for file retrieval, avoiding client file failure due to replication delay. (Unzipped Fastdfs-nginx-module for nginx installation)

2. Upload fastdfs-nginx-module_v1.16.tar.gz to /usr/local/src

3, decompression

# cd /usr/local/src/
# tar - ZXVF fastdfs - nginx - module_v1. 16. The tar. Gz
Copy the code

4. Modify the fastdfs-nginx-module config file

# cd fastdfs-nginx-module/src
# vi config
CORE_INCS="$CORE_INCS /usr/local/include/fastdfs /usr/local/include/fastcommon/"Change the value to CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"
Copy the code

(Note: this path modification is very important, otherwise nginx will report errors when compiling.)

5. Upload the current stable version Nginx(nginx-1.13.0.tar.gz) to the /usr/local/src directory

Install the dependency packages required to compile Nginx

# yum install gcc gcc-c++ make automake autoconf libtool pcre* zlib openssl openssl-devel
Copy the code

7, install Nginx (fastdfs-nginx-module)

# cd /usr/local/src/
# tar - ZXVF nginx - 1.13.0. Tar. Gz
# CD nginx - 1.13.0
# ./configure --add-module=/usr/local/src/fastdfs-nginx-module/src
# make && make install
Copy the code

Copy fastdfs-nginx-module

Copy the fastdfs-nginx-module configuration file to /etc/fdfs and modify it

# cp /usr/local/src/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/
# vi /etc/fdfs/mod_fastdfs.conf
Copy the code

Modify the following configuration:

Connect_timeout =10 base_path=/ TMP tracker_server=192.168.50.131:22122 storage_server_port=23000 group_name=group1 url_have_group_name =true
store_path0=/fastdfs/storage
Copy the code

9. Copy the FastDFS configuration file to /etc/fdfs

# cd /usr/local/src/FastDFS/conf
# cp http.conf mime.types /etc/fdfs/
Copy the code

10. Create a soft link in /fastdfs/storage and link it to the directory where the data is stored

# ln -s /fastdfs/storage/data/ /fastdfs/storage/data/M00
Copy the code

11. Configure Nginx

Nginx compact configuration example:

user root;
worker_processes 1;
events {
	worker_connections 1024;
}
http {
	include mime.types;
	default_type application/octet-stream;
	sendfile on;
	keepalive_timeout 65;
	server {
		listen 8888;
		server_name localhost;
		location ~/group([0-9])/M00 {
			#alias /fastdfs/storage/data;ngx_fastdfs_module; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; }}}Copy the code

Note: Server_port =8888 in /etc/fdfs/storage.conf, because http.server_port is 8888 by default. If you want to change it to 80, change it accordingly. B, Storage If there are multiple groups, the access path with the group name, such as /group1/ m00/00/00 / XXX, corresponds to the Nginx configuration:

location ~/group([0-9])/M00 {
	ngx_fastdfs_module;
} 
Copy the code

C. If 404 is found, change user nobody in the first line of nginx.conf to user root and restart the nginx.conf file.

12. Open Nginx port 8888 on the firewall

# vi /etc/sysconfig/iptables
Copy the code

Add:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8888 -j ACCEPT
Copy the code

Restarting the Firewall

# service iptables restart
Copy the code

13. Start Nginx

# /usr/local/nginx/sbin/nginx
Copy the code

After the startup is successful, enter:

ngx_http_fastdfs_set pid=xxx
Copy the code

(restart Nginx command: / usr/local/Nginx/sbin/Nginx – s reload)

14. Access the files uploaded during the test through the browser

http://192.168.50.132:8888/group1/M00/00/00/wKgyhFkWubGAcwrWAAVFOL7FJU4.tar.gz

Note: Do not use the kill -9 command to forcibly kill the FastDFS process. Otherwise, binlog data may be lost. Also, you can go to the linkDownload.csdn.net/detail/l102…Download the FastDFS_V5.05_ installation package and tool package

Ok, let’s call it a day! Don’t forget to give a look and forward, let more people see, learn together progress!!

Write in the last

If you find this article helpful, please search and follow the wechat official account of “Glacier Technology” to learn distributed storage technology from Glacier.

【FastDFS】FastDFS distributed file system installation and use, look at this article enough!!

Writing in the front

preparation

Operations performed by all servers

Compile and install the required dependency packages

2. Install libfastCommon

3. Install FastDFS

Configure the FastDFS tracker (192.168.50.131)

1. Copy the FastDFS tracker sample configuration file

2. Edit the tracker configuration file

3. Create the base data directory (see base_path configuration)

4. Open tracker port in firewall (default: 22122)

5. Start the Tracker

Close the Tracker

7. Set the FastDFS tracker to start

Configuring FastDFS Storage (192.168.50.132)

1. Copy the FastDFS storage sample configuration file

2. Edit the storage sample configuration file

3. Create the base data directory (see base_path configuration)

4. Open storage port in firewall (default: 23000)

5. Start Storage

6. Close Storage

7, set FastDFS memory boot

File Upload Test (192.168.50.131)

1. Modify the client configuration file in the Tracker server

2. Run the following command to upload files

Install nginx on each storage node

Fastdfs-nginx-module function description

2. Upload fastdfs-nginx-module_v1.16.tar.gz to /usr/local/src

3, decompression

4. Modify the fastdfs-nginx-module config file

5. Upload the current stable version Nginx(nginx-1.13.0.tar.gz) to the /usr/local/src directory

Install the dependency packages required to compile Nginx

7, install Nginx (fastdfs-nginx-module)

Copy fastdfs-nginx-module

9. Copy the FastDFS configuration file to /etc/fdfs

10. Create a soft link in /fastdfs/storage and link it to the directory where the data is stored

11. Configure Nginx

12. Open Nginx port 8888 on the firewall

13. Start Nginx

14. Access the files uploaded during the test through the browser

Write in the last

Related Posts

Hapi-js implements JWT-BASED login and authentication

What is JSON

Bigtable: Distributed Storage System for Structured Data (3)