FastDFS profile

FastDFS is an open source lightweight distributed file system. It solves the problems of large data storage and load balancing. Especially suitable for small and medium size [4K-500M] files as the carrier of online services.

FastDFS features

  • Group storage, flexible and simple, peer-to-peer structure, no single point
  • Files are not stored in blocks. Uploaded files correspond to files in the OS file system
  • File ids are generated by FastDFS as file access credentials, FastDFS does not require a traditional Name Server
  • Seamless with the popular Web Server, FastDFS already offers Apache and Nginx extensions
  • Medium and small files can be well supported,
  • Support massive small file storage support multiple disks,
  • Single-disk data recovery Only one copy of a file with the same content can be saved, saving disk space
  • Online capacity expansion Primary/secondary files are supported
  • V2.0 network communication adopts Libevent, which supports large concurrent access and better overall performance
  • Download files support multithreading mode, support breakpoint continuation

The composition of FastDFS

  • FastDFS consists of a Client, a Tracker Server and a Storage Server. The client (client)
The client (client)

As the originator of the business request, data interaction with the tracker server or storage node through a proprietary interface using the TCP/IP protocol.

The tracker (tracker)

The Trackerserver is used for load balancing and scheduling. When a file is uploaded, the Tracker Server can find the Storage Server according to certain policies to provide file uploading service. A tracker can be called a tracking server or a scheduling server. The tracker acts as a load balancer on access. Can be added or taken offline at any time without affecting online services.

Storage Node

The Storageserver functions as file Storage. The files uploaded by the client are finally stored on the Storageserver. The Storage Server does not implement its own file system but uses the file system of the operating system to manage files. A storage can be called a storage server. Servers on storage nodes can be added or offline at any time without affecting online services.

Environmental installation

Install the build environment

yum install git gcc gcc-c++ make automake vim wget libevent -y
Copy the code

Install the libfastCommon base library

mkdir /root/fastdfs
cd /root/fastdfs
#download
git clone https://github.com/happyfish100/libfastcommon.git --depth 1
#compile
cd libfastcommon/
./make.sh && ./make.sh install
Copy the code
  • The results of
gcc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O3 -c -fPIC -o fc_memory.lo fc_memory.c gcc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O3 -c -fPIC -o shared_buffer.lo shared_buffer.c gcc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O3 -c -fPIC -o thread_pool.lo thread_pool.c gcc -Wall -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -g -O3 -o libfastcommon.so -shared hash.lo chain.lo shared_func.lo ini_file_reader.lo logger.lo sockopt.lo  base64.lo sched_thread.lo http_func.lo md5.lo pthread_func.lo local_ip_func.lo avl_tree.lo ioevent.lo ioevent_loop.lo fast_task_queue.lo fast_timer.lo process_ctrl.lo fast_mblock.lo connection_pool.lo fast_mpool.lo fast_allocator.lo fast_buffer.lo multi_skiplist.lo flat_skiplist.lo system_info.lo fast_blocked_queue.lo id_generator.lo char_converter.lo  char_convert_loader.lo common_blocked_queue.lo multi_socket_client.lo skiplist_set.lo uniq_skiplist.lo json_parser.lo buffered_file_writer.lo server_id_func.lo fc_queue.lo fc_memory.lo shared_buffer.lo thread_pool.lo -lm -ldl -lpthread ar  rcs libfastcommon.a hash.o chain.o shared_func.o ini_file_reader.o logger.o sockopt.o base64.o sched_thread.o http_func.o md5.o pthread_func.o local_ip_func.o avl_tree.o ioevent.o ioevent_loop.o fast_task_queue.o fast_timer.o process_ctrl.o fast_mblock.o connection_pool.o fast_mpool.o fast_allocator.o fast_buffer.o multi_skiplist.o flat_skiplist.o system_info.o fast_blocked_queue.o id_generator.o char_converter.o char_convert_loader.o common_blocked_queue.o multi_socket_client.o skiplist_set.o uniq_skiplist.o json_parser.o buffered_file_writer.o server_id_func.o fc_queue.o fc_memory.o shared_buffer.o thread_pool.o mkdir -p /usr/lib64 mkdir -p /usr/lib mkdir -p /usr/include/fastcommon install -m 755 libfastcommon.so /usr/lib64 install -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h  local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h fast_allocator.h fast_buffer.h skiplist.h multi_skiplist.h flat_skiplist.h skiplist_common.h system_info.h fast_blocked_queue.h php7_ext_wrapper.h id_generator.h char_converter.h char_convert_loader.h common_blocked_queue.h multi_socket_client.h skiplist_set.h uniq_skiplist.h fc_list.h json_parser.h buffered_file_writer.h server_id_func.h fc_queue.h fc_memory.h shared_buffer.h thread_pool.h /usr/include/fastcommon if [ ! -e /usr/lib/libfastcommon.so ]; then ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so; fiCopy the code

Install FastDFS

cd /root/fastdfs
#Download the FastdFS installation packageWget HTTP: / / https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gzCopy the code
  • The results of
Saving to: "V5.11.tar.gz" [<=>] 336,939 13.2kb /s time 18s 2020-07-23 18:08:17 (18.2kb /s) - "v5.11.tar. gz" saved [336939] [root@localhost fastdfs]# ls libfastCommon v5.11.tar.gzCopy the code
#Unpack theTar -zxvf v5.11.tar. gz CD fastDFS-5.11Copy the code

Compile the installation

./make.sh && ./make.sh install

  • Check whether the installation is successful. By default, the configuration file is generated in /etc/fdfs

    [root@localhost fastdfs-5.11]# CD /etc/fdfs/ [root@localhost FDFS]# ll total amount 24 -rw-r--r-- 1 root root 1461 7月 23 18:11 Sample -rw-r--r-- 1 root root 7927 July 23 18:11 storage.conf.sample -rw-r--r-- 1 root root 105 July 23 18:11 Storage_ids.conf. sample -rw-r--r--. 1 root root 7389 July 23 18:11 trackerCopy the code

Configuration file changes

  • Copying configuration Files

    #Configuration File Preparationcp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf cp The/etc/FDFS/client. Conf. Sample/etc/FDFS/client. The conf cp/root/fastdfs/fastdfs - 5.11 / conf/HTTP. Conf/etc/FDFS cp / root/fastdfs/fastdfs - 5.11 / conf/mime types/etc/FDFSCopy the code

  • The tracker configuration file is modified

  vim /etc/fdfs/tracker.conf
  #The following table describes the changes that need to be made
  port=22122
  base_path=/home/fastdfs
Copy the code

  • Modify the storage configuration file

    Vim /etc/ff/storage. conf # Modify the following parameters: port=23000 base_path=/home/fastdfs # Store_path0 =/home/fastdfs # Tracker_server =192.168.211.136:22122 # HTTP port for accessing files (default 8888, modified as appropriate, same as in nginx) http.server_port=8888Copy the code

  • Example Create a path specified in the tracker or storage configuration file

    mkdir /home/fastdfs -p
Copy the code

Start the test

/usr/bin/fdfs_trackerd /etc/ffs/tracker. conf restart /usr/bin/fdfs_storaged /etc/ff/storage. conf restart View all running ports netstat -ntlpCopy the code
  • Check whether there is port listening
[root @ localhost FDFS] # netstat NTLP | grep FDFS TCP 0 0 0.0.0.0:0.0.0.0:23000 * 29199 / LISTEN fdfs_storaged TCP 0 0 0.0.0.0:0.0.0.0:22122 * 29076 / fdfs_trackerd LISTENCopy the code

  • Test the upload

    • First modify the client configuration
    vim /etc/fdfs/client.conf
#The following table describes the changes that need to be made
base_path=/home/fastdfs
#IP address and port number of the tracker serverTracker_server = 192.168.18.213:22122Copy the code

  • Test the upload

    PNG /usr/bin/fdfs_upload_file /etc/fff/client.conf /root/fastdfs/timg.jpgCopy the code

  • Results back

    group1/M00/00/00/wKgS1V8ZcTOANXpUAAEI46Ym6Xs928.jpg

    If the specified address is returned, the upload is successful!

Nginx module installation

So far, fastdfs has been installed successfully! If we want to access the file as a network resource. This can be done with nginx. Fastdfs provides the Fastdfs-nginx-module module to implement this function

  • Install fastdfs – nginx – module

    CD/root/fastdfs wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.20.tar.gz extract tar - XVF V1.20. Tar. GzCopy the code

  • configuration

CD fastdfs-nginx-module-1.20/ SRC vim config Change lines 5 and 15 to ngx_module_incs="/usr/include/fastdfs /usr/include/fastcommon/" CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"Copy the code
  • Reproduction mod_fastdfs. Conf/etc/FDFS /

cp mod_fastdfs.conf /etc/fdfs/

  • Modify the configuration
 vim /etc/fdfs/mod_fastdfs.conf
 #The following table describes the changes that need to be madeTracker_server = 192.168.18.213:22122 url_have_group_name = true store_path0 = / home/fastdfsCopy the code

mkdir -p /var/temp/nginx/client

Nginx installation

Download resources

CD/root/fastdfs wget http://nginx.org/download/nginx-1.15.6.tar.gz tar - ZXVF nginx - 1.15.6. Tar. Gz nginx 1.15.6 / CDCopy the code
  • Add the Fastdfs-nginx-module module
  yum -y install pcre-devel openssl openssl-devel
  #Add the Fastdfs-nginx-module module. / configure -- add - the module = / root/fastdfs/fastdfs - nginx - module - 1.20 / SRCCopy the code
  • Compile the installation
Make && make installCopy the code
  • Check whether the module is installed/usr/local/nginx/sbin/nginx -V
[root @ localhost nginx - 1.15.6] # / usr/local/nginx/sbin/nginx -v nginx version: Nginx /1.15.6 built by GCC 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) Configure arguments: - add - the module = / root/fastdfs/fastdfs - nginx - module - 1.20 / SRCCopy the code

Modify the configuration

Server {listen 8888; server_name localhost; location ~/group[0-9]/ { ngx_fastdfs_module; }}Copy the code

Start the nginx/usr/local/nginx/sbin/nginx

  • test

  • Disable firewall systemctl stop Fairewalld

  • download

    http://ip:8888/group1/M00/00/00/xxx.png
Copy the code

> http://192.168.18.213:888/group1/M00/00/00/wKgS1V8ZcTOANXpUAAEI46Ym6Xs928.jpg
Copy the code