This is the 29th day of my participation in the August More Text Challenge

Introduction to the

What is minio?

It can build a high-performance cloud native object storage system written in go and compatible with Amazon-S3-API

It has the following characteristics:

  • High performance on standard hardware, read/write speeds up to 183GB/ SEC and 171GB/ SEC
  • Extensibility takes advantage of web zoomer knowledge to scale clusters simply by adding machines
  • Cloud native supports K8S, microservices, and multi-tenant container technologies
  • S3 compatibility Adopts S3 compatibility protocol (ali Cloud OSS and other oss also support this protocol)
  • Simple installation is very simple to use, a command to start

The following exaggerated description is from the official website, not made up by the author

The world’s fastest growing object storage system

MinIO production deployments cover the world. As the most used and downloaded object storage service in the world, its influence is growing – supported by an extraordinary community of contributors and communicators.

The installation

Thanks to the cross-platform cross-compilation function of go, you only need to type the package into the corresponding system architecture, without the server installation environment, and can run and start with one click

Minio consists of the server miniO and the client TOOL MC

Server Installation

Install on Linux

Similar to a command line operation, specifying the storage directory /data is started directly

wget http://dl.minio.org.cn/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data
Copy the code

Docker installation is recommended in this way, convenient maintenance

Here 9000 port is API call, 9001 is the Web UI page, directly visit http://ip:9001 and enter the default account secret: minioadmin/minioadmin login

docker run \
  -p 9000:9000 \
  -p 9001:9001 \
  minio/minio server /data --console-address ": 9001"
Copy the code

By default, we install the Community edition, whose Dashborad is a simple page

Install the client

The client is also a command line tool, MC

wget https://dl.min.io/client/mc/release/linux-amd64/mc
mv mc /usr/local/bin/
Copy the code

Data protection

— The content of this chapter comes from the official website documents

Minio uses erasure code and checksum to protect data from hardware failures and silent data damage. If half (N/2) of the hard disks are lost, data can still be recovered

What is Erasure code?

Erasure code is a mathematical algorithm for recovering lost and damaged data. Minio uses Reed-Solomon code to split objects into N/2 data and N/2 parity blocks. This means that if there are 12 disks, an object can be divided into 6 data blocks and 6 parity blocks. You can lose any 6 disks (whether they are stored as data blocks or parity blocks) and still recover data from the remaining disks

example

Start minio specify 4 disks, get 2 disks, check whether minio can read and write normally

minio server /data1 /data2 /data3 /data4 
Copy the code

The service side

Configuration items

By default, MinIOserver stores all configuration information in the ${HOME}/.minio/config.json file

You can override it using the –config-dir command line option

minio server --config-dir /etc/minio /data
Copy the code

The default certificate directory is ${HOME}/.minio/certs. If TLS is required, you need to place the certificate in this directory to enable HTTPS

The command

The MinIO Client (MC) provides the “admin” subcommand to perform administrative tasks for your MinIO deployment

Service The service restarts and stops all MinIO servers update Update the info information about all MinIO servers. Display the MinIO server information. User User management user group Group management group Policy Policy defined on the MinIO server Config Configuration Manage the MinIO server Configuration Heal Repair disks on the MinIO server. Buckets and object profiles generate profile data for debugging. Top Top Provides top statistics for MinIO. Trace tracing displays HTTP tracing for the MinIO serverconsoleConsole Display Console logs of the MinIO server Prometheus Prometheus Management Prometheus configuration KMS KMS Perform KMS management operationsCopy the code

Use the sample

Set the server address before using it. For details, see Client MC Usage – Example – Configuring the Server Address

Myminio bucket information MC admin info myminio bucket user information MC admin user list myminio #Copy the code

limit

Hard disk:

  • There is no limit to the maximum or minimum
  • Read/write Quorum read can lose half of the number of disks (N/2), write must be half +1 (N/2 +1)

Browser:

  • Web browsing upload maximum 5G

The API related:

  • There is no limit on the number of buckets or objects
  • The maximum value of a single object is 5, and the maximum value of a PUT operation is 5
  • The maximum number of uploaded parts is 1000
  • Part sizes range from 5M to 5G
  • Look at the list, the maximum number of return value objects and parts is 1000

A secure connection

Minio supports TLS security access by default. You need to store the private key and public key in a specified directory

The public and private keys must be named private.key and public. CRT, and placed in config at the same level as minio.

Minio supports only key/certificate in PEM format on Linux

If you want to use the Let ‘s Encrypt please enter website: docs.minio.org.cn/docs/master…

However, it is recommended to use OpenSSL generation, which is relatively simple:

Openssl genrsa -out private.key2048Public key openSSL req -new -x509 -days 3650 -key private.key -out public.crt -subj "/C=CH/ST=state/L=location/O=organization/CN=domain"
Copy the code

Used by client MC

Some operations, miniO UI basically have functions, you can directly click the button on the page to operate, but the client MC, also need to be familiar with is not

Enable shell auto-completion

Download the automatic completion configuration provided by the government and press TAB to complete the configuration

sudo wget https://raw.githubusercontent.com/minio/mc/master/autocomplete/bash_autocomplete -O /etc/bash_completion.d/mc
source /etc/bash_completion.d/mc
Copy the code

The command

The MinIO Client(MC) provides an alternative to UNIX commands such as ls, cat, cp, mirror, diff, find

When the miniO server is not configured, executing the following command is like executing a system command that operates on the files on the system, not minio

Ls Lists files and folders. MB Creates a bucket or folder. Cat displays file and object contents. Pipe redirects a STDIN to an object or file or STDOUT. Share generates the URL for sharing. Cp Copies files and objects. Mirror Mirrors buckets and folders. Find finds files based on parameters. Diff compares differences between two folders or buckets. Rm deleted files and objects. Events Notifications of managed objects. Watch listens for events on files and objects. Policy Manages access policies. Session Indicates the session saved by cp command management. Config Manages MC configuration files. Update Checks software updates. Version Displays the version information.Copy the code

Use the sample

Configuring the Server Address

mc alias set myminio http:/ / 10.10.1.32:9000 minioadmin minioadmin
Copy the code

Create a bucket

mc mb myminio/test2
Copy the code

Upload a file

mc cp /var/log/messages myminio/test2
Copy the code

Listing directory files

mc ls myminio/test2
Copy the code

Viewing file Contents

mc cat myminio/test2/messages
Copy the code

Deletes a file or object

mc rm myminio/test2/messages
Copy the code

It is indeed an alternative to UNIX commands as well as operating on Linux

Event notification

If an object is uploaded or deleted from a bucket, it is monitored by bucket event notification and can be sent out or stored using MQ, SQL,webhooks, etc

Webhooks releases minio events

Webhooks send data to defined apis using push

Configuration webhook

Localhost :3000 is a self-written API interface

"webhook": {
  "1": {
    "enable": true."endpoint": "http://localhost:3000/"
}
Copy the code

Enabling bucket Notification

Add a bucket named images and enable event notification

Thumbnail # Enable event notification ARN is ARN: miniO: SQS: US-east -1:1:webhook
mc event add myminio/images arn:minio:sqs:us-east-1:1:webhook --events put --suffix .jpg
Copy the code

View the event notification list

mc event list myminio/images
Copy the code

When uploading and deleting operations are performed, you can receive corresponding notification messages through the API of port 3000

arn:minio:sqs:us-east-1:1:webhook   s3:ObjectCreated:*   Filter: suffix=".jpg"
Copy the code

ARN’t make too much discussion, see: docs.aws.amazon.com/general/lat…

Thumnailer receives events

If you don’t want to write an interface to receive events yourself, you can use a third-party project: Thumnailer

Use Thumbnailer to listen for MinIO notifications. If a file is uploaded to the MinIO service, Thumnailer listens for the notification, generates a thumbnail image and uplots it to the MinIO service

Install the configuration

Git clone//github.com/minio/thumbnailer/NPM install # configure minio server info CD thumbnailer vim config/webhook.json # NODE_ENV=webhook nodeCopy the code

Thumnailer listening is displayed at http://localhost:3000 and corresponds to -_-

Test Event Notification

Myminio /images-thumbnail = myminio/images-thumbnail = myminio/imagesCopy the code

multi-tenant

Create accounts on the Consule UI page with permissions for read-only, write only, read and write, check, and UI administrator

If you create a bucket without setting permissions, all permissions are allowed by default. When the user has the corresponding permission, he can directly ask the bucket

Detailed examples: please refer to the website (docs.minio.org.cn/docs/master…

monitoring

The following interfaces need to be notified to port 9000 for access

Minio provides two interface paths for health check by default

  • Dynamic probe/minio/health/liveActive probe – Whether the server is working properly
  • Ready probe/minio/health/readyReadiness probe – Indicates whether the server is not accepting connections due to heavy load

Integrated into the Prometheus

The default is also to expose the interface and provide related monitoring data to Prometheus to fetch metrics

  • Prometheus data is available at/minio/prometheus/metrics

Configure Prometheus to monitor minio

scrape_configs:
- job_name: minio-job
  metrics_path: /minio/prometheus/metrics
  scheme: http
  static_configs:
  - targets: ['localhost:9000']
Copy the code

Specific indicators meaning refer to: docs.minio.org.cn/docs/master…