MinIO distributed file storage cluster developed by full stack

background

Where are the upload files stored in your project?

Our project on file storage has gone through this evolution:

1. Static resource directory; At that time, the front and back ends are not separated, so they are directly in the static resource directory of the project. Before each deployment, backup the resource directory first, otherwise these files will be lost.
2. A separate file storage directory on the server; For small projects with few files to save, this approach is generally sufficient. This phase lasts for a year or two until the hard disk space on the PC runs out. Obviously, this approach does not support horizontal scaling;
3. Distributed file storage; We looked into distributed file storage when we encountered the need for multi-instance clustering to ensure high availabilityFastDFSwithMinIOAs well as cloud services (object storage of Qiuniuyun, Ali Cloud, etc.), whereasFastDFSThe configuration was complicated, so we decided to use itMinIO, easy to use, extensible.

Basic operation

MinIOThe world’s leading object storage pioneer with read/write speeds up to 183 GB/SEC and 171 GB/SEC on standard hardware. MinIO serves as the primary storage for cloud native applications, which require higher throughput and lower latency than traditional object storage. You can expand the namespace by adding more clusters, and more racks, until you reach your goal. At the same time, conform to all native cloud computing architecture and construction process, and include the latest cloud computing brand new technologies and concepts.

Object storage is nothing more than file uploads, downloads, and deletions, plus buckets. The focus here is on the high availability and scalability of the MinIO distributed cluster.

1. Barrels of management;
2. Object management (upload, download, delete);
3. Object presignature;
4. Bucket policy management;

Note: HERE I use the 7.x version for experiment and demonstration. The MinIO background management interface of the latest 8.x version is different, but after our actual production test, the interface is compatible.

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>7.1.4</version>
</dependency>
Copy the code

MinIO runs in Docker single instance

docker run -p 9000:9000 \
  --name minio1 \
  -v /opt/minio/data-single \
  -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \
  -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
  minio/minio server /data
Copy the code

MinIO is configured in Docker Compose cluster (4 instances)

• reference

IO /docs/deploy…

• configuration

Docker-compose runs four MinIO instances on one host and is reverse-proxy by Nginx to provide unified load balancing services.

Two configurations are involved:

1. docker-compose.yaml
2. nginx.conf

• docker-compose.yaml

version: '3.7'

# starts 4 docker containers running minio server instances.
# using nginx reverse proxy, load balancing, you can access
# it through port 9000.
services:
  minio1:
    image: minio/minio:RELEASE.2020-11-10T21-02-24Z
    volumes:
      - data1-1:/data1
      - data1-2:/data2
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    command: server http://minio{1... 4}/data{1... 2}
    healthcheck:
      test: ["CMD"."curl"."-f"."http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio2:
    image: minio/minio:RELEASE.2020-11-10T21-02-24Z
    volumes:
      - data2-1:/data1
      - data2-2:/data2
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    command: server http://minio{1... 4}/data{1... 2}
    healthcheck:
      test: ["CMD"."curl"."-f"."http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio3:
    image: minio/minio:RELEASE.2020-11-10T21-02-24Z
    volumes:
      - data3-1:/data1
      - data3-2:/data2
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    command: server http://minio{1... 4}/data{1... 2}
    healthcheck:
      test: ["CMD"."curl"."-f"."http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio4:
    image: minio/minio:RELEASE.2020-11-10T21-02-24Z
    volumes:
      - data4-1:/data1
      - data4-2:/data2
    expose:
      - "9000"
    environment:
      MINIO_ACCESS_KEY: minio
      MINIO_SECRET_KEY: minio123
    command: server http://minio{1... 4}/data{1... 2}
    healthcheck:
      test: ["CMD"."curl"."-f"."http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  nginx:
    image: Nginx: 1.19.2 - alpine
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "9000:9000"
    depends_on:
      - minio1
      - minio2
      - minio3
      - minio4

## By default this config uses default local driver,

## For custom volumes replace with volume driver configuration.

volumes:
  data1-1:
  data1-2:
  data2-1:
  data2-2:
  data3-1:
  data3-2:
  data4-1:
  data4-2:
Copy the code

• nginx.conf

user nginx; worker_processes auto; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; #tcp_nopush on; keepalive_timeout 65; #gzip on; # include /etc/nginx/conf.d/*.conf; upstream minio { server minio1:9000; server minio2:9000; server minio3:9000; server minio4:9000; } server { listen 9000; listen [::]:9000; server_name localhost; # To allow special characters in headers ignore_invalid_headers off; # Allow any size file to be uploaded. # Set to a value such as 1000m; to restrict file size to a specific value client_max_body_size 0; # To disable buffering proxy_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 300; Keepalive is only enabled in HTTP/1.1 proxy_http_version 1.1; proxy_set_header Connection ""; chunked_transfer_encoding off; proxy_pass http://minio; }}}Copy the code

MinIO runs in a cluster (4 instances) under Docker Compose

Docker-compose up: docker-compose up: docker-compose up -d: docker-compose DownCopy the code

High availability test of the MinIO cluster

Once the cluster is running, you can observe availability by stopping a different number of instances: downloadable, uploadable.

A stand-alone MinIO server would go down if the server hosting the disks goes offline. In contrast, a distributed MinIO setup with m servers and n disks will have your data safe as long as m/2 servers or m*n/2 or more disks are online.

For example, an 16-server distributed setup with 200 disks per node would continue serving files, up to 4 servers can be offline in default configuration i.e around 800 disks down MinIO would continue to read and write objects.

MinIO’s official recommendation is to build a four-disk cluster at least. The specific configuration of the number of machines depends on your needs. For example:

• Four hard disks per machine
• Two hard disks for two machines
• Four machines and one hard drive

Four instances in the cluster are online

If all four instances in a cluster with four instances are online, you can upload or download them.

Three instances in the cluster are online

If all three instances in a cluster with four instances are online, the service can be uploaded or downloaded. That is, the service is still available even though one instance is down.

Two instances in the cluster are online

If only two instances in a cluster with four instances are online, you can only download them, but cannot upload them.

One instance in the cluster is online

At this time, one service instance is still online, but normal upload and download services cannot be provided.

We can conclude that in a MinIO cluster with four instances running, three of them are online and both read and write can take place, two of them are online and read is guaranteed but cannot be written, and if only one instance remains, neither read nor write can take place. This is consistent with the instructions given in the official documentation (more than M /2 servers online, or more than M * N /2 disks online).

Note: After each service is stopped, you can view the logs of MinIO background to observe the online status monitoring process of each service instance for easy understanding.

• Viewing Service Logs

docker exec -it 12935e3c6264 bash
Copy the code

Possible problems

• SpringBoot upload file error: org. Springframework. Web. Multipart. MaxUploadSizeExceededException: Maximum upload size exceeded; nested exception is java.lang. IllegalStateException: org.apache.tomcat.util.http.fileupload.impl. FileSizeLimitExceededException: The field file exceeds its maximum permitted size of 1048576 bytes.

This is because the SpringBoot project limits the size of uploaded files. The default size is 1 MB. When a user uploads a file larger than 1 MB, the above error is thrown.

spring:
  servlet:
    multipart:
      maxFileSize: 10MB
      maxRequestSize: 30MB
Copy the code

• throughdocker exec -it cd34c345960c /bin/bashOCI Runtime exec failed: exec failed: container_linux.go:349: starting Container process caused “exec: “/”: permission denied”: unknown

Docker execit cd34c345960c /bin/bash docker execit cd34c345960c sh

---

If you have any questions or any bugs are found, please feel free to contact me.

Your comments and suggestions are welcome!