Minio object storage technology
MinIO is a high-performance object storage solution that natively supports Kubernetes deployment. MinIO provides apis that are compatible with Amazon Web Services S3 and supports all core S3 functionality. MinIO is distributed under the GNU Affero General Public License V3.0.
When it comes to object storage technology, we have all experienced the tug of war with Fastdfs for a long time. Even though most of us now use OSS object storage service provided by cloud service vendors, the economic cost is increasing, and the data is stored on other people’s servers. To some extent, The granularity of file resource control is extremely uncontrollable, and the data on the public network has no privacy at all. At the development level, though, you just need to integrate the corresponding SDK and use it out of the box. However, Minio is a great tool for choosing your own object storage technology. There are almost no bottlenecks in the building process and integration. Whether it’s from a traditional server installation, or Docker and Kubernetes based deployment, it’s a no-brainer.
Basic overview
MinIO supports the widest range of use cases across the largest number of environments. Cloud native since inception, MinIO’s software-defined suite runs seamlessly in the public cloud, private cloud and at the edge – making it a leader in the hybrid cloud. With industry leading performance and scalability, MinIO can deliver a range of use cases from AI/ML,analytics, backup/restore and modern web and mobile apps.
MinIO supports the widest range of use cases in the largest number of environments. Since cloud native, MinIO’s suite of software definitions runs seamlessly across public, private, and edge clouds — making it a leader in hybrid clouds. With industry-leading performance and scalability, MinIO can offer a range of use cases, including AI/ML, analytics, backup/recovery, and modern Web and mobile applications.
- Hybrid Cloud: Hybrid Cloud
- Born cloud native
- MinIO is pioneering high Performance Object storage
- Built on the Principles of Web Scale: Built on the principles of web scale
- The #1 open source object store. And the most enterprise ready. Open source object storage, and best suited for the enterprise
- The defacto standard for Amazon S3 compatibility
- 1. “Simple and powerful”
The basic characteristics of
MinIO’s enterprise-class features represent the standard for object storage space. From the AWS S3 API to S3 Select and our implementation of inline erase coding and security, our code is widely acclaimed and often copied by some of the biggest names in technology and business.
- Erasure Coding: error correction code
MinIO protects data using inline erase encoding for each object written in assembly code to provide the highest possible performance. MinIO strips objects into data and parity blocks with a user-configurable level of redundancy using the Reed-Solomon code. MinIO’s Erasure Coding performs repair at the object level, allowing multiple objects to be repaired independently.
With maximum parity of N/2, MinIO’s implementation ensures that only ((N/2)+1) operation drives are used for uninterrupted read and write operations during deployment. For example, in a 12-drive setup, MinIO shards objects into six data and six parity drives, and can reliably write new objects or rebuild existing objects, with only seven drives remaining in the deployment.
- Bitrot Protection: Bitrot Protection
Silent data corruption, or BITROT, is a serious problem faced by disk drives, causing data to corrupt without the user’s knowledge. The causes are many (drive aging, current spikes, errors in disk firmware, phantom write, read/write direction errors, driver errors, accidental overwrite), but the result is the same – data corruption.
MinIO’s optimized implementation of the HighwayHash algorithm ensures that it never reads corrupted data – it captures and fixes corrupted objects in real time. End-to-end integrity is ensured by calculating hashes on READ and validating them on WRITE from application, network to memory/drive. The implementation is designed for speed and can achieve hash speeds in excess of 10 GB/ s on a single core Intel CPU.
- Encryption: Encryption
Encrypting data on the fly is one thing; Protecting static data is another matter. MinIO supports a variety of sophisticated server-side encryption schemes to protect data wherever it is located. MinIO’s approach ensures confidentiality, integrity, and authenticity at a negligible performance cost. Server-side and client-side encryption is supported using AES-256-GCM, Chacha20-Poly1305, and AES-CBC.
The encryption object is tamper-proof using AEAD server encryption. In addition, MinIO is compatible and tested with all common key management solutions, such as HashiCorp Vault. MinIO uses a key management system (KMS) to support SSE-S3.
If a client requests SSE-S3 or enables automatic encryption, the MinIO server encrypts each object with a unique object key protected by a master key managed by KMS. Given the very low overhead, automatic encryption can be turned on for each application and instance.
- Identity Management: Identity Management
MinIO supports the most advanced standards in identity management and is integrated with OpenID connection compatibility providers as well as key external IDP vendors. This means that access is centralized, and passwords are temporary and rotational, rather than stored in configuration files and databases. In addition, access policies are fine-grained and highly configurable, which means it is easy to support multi-tenant and multi-instance deployments.
- Continuous Replication: Continuous Replication
The challenge with traditional replication methods is that they cannot scale effectively beyond hundreds of terabytes. That being said, everyone needs a replication strategy to support disaster recovery, and that strategy needs to span geographies, data centers, and the cloud.
MinIO’s continuous replication is designed for large-scale, cross-data center deployments. By using Lambda to compute notifications and object metadata, it can compute deltas quickly and efficiently. Lambda notifications ensure that changes are propagated immediately, rather than in the traditional batch mode.
Continuous replication means that if a failure occurs, data loss is kept to a minimum, even in the face of highly dynamic data sets. Finally, as MinIO does, continuous replication is multi-vendor, which means your backup location can be anywhere from the NAS to the public cloud.
- The Global Federation
Modern companies have data everywhere. MinIO allows you to combine these different instances to form a unified global namespace. Specifically, any number of MinIO servers can be combined into a single distributed pattern set, and multiple distributed pattern sets can be combined into a single MinIO server federation. Each MinIO Server Federation provides unified management and namespace.
The MinIO federation server supports an unlimited number of distributed pattern sets. The impact of this approach is that object storage can scale massively for large, geographically distributed enterprises, while retaining the ability to accommodate a variety of applications (Splunk, Teradata, Spark, Hive, Presto, TensorFlow, H20) from a single console.
- Multi-cloud Gateway: indicates the multi-cloud Gateway
All enterprises are adopting a multi-cloud strategy. This includes private clouds. Therefore, your bare-metal virtualization container and public cloud services (including non-S3 providers like Google, Microsoft, and Alibaba) must look the same. While modern applications are highly portable, the data that supports them is not.
Making this data available wherever it is located is a major challenge that MinIO solves. MinIO runs bare-metal, network-attached storage, and on every public cloud. More importantly, MinIO uses the Amazon S3 API to ensure that your view of this data looks exactly the same from an application and management perspective.
MinIO can go a step further and make your existing storage infrastructure compatible with Amazon S3. The implications are profound. Organizations can now truly unify their data infrastructure — from files to blocks, all of which are displayed as objects accessible through the Amazon S3 API without migration.
- WORM: The WORM object is locked
After WORM is enabled, MinIO disables all apis that may change object data and metadata. This means that the data once written is tamper-proof. This has practical applications for many different regulatory requirements.
Basic architecture
MinIO is designed to be cloud-native and can run as a lightweight container managed by external choreography services such as Kubernetes. The entire server is a static binary of about 40MB and is very efficient in using CPU and memory resources – even under high loads. The result is that you can co-host a large number of tenants on shared hardware.
MinIO runs on commercial servers with locally connected drives (JBOD/JBOF). All servers in the cluster have the same capabilities (fully symmetric architecture). There are no name nodes or metadata servers.
MinIO writes data and metadata together as objects without requiring a metadata database. In addition, MinIO performs all functions (erasing code, bitrot checking, encryption) as inline, strictly consistent operations. The result is that MinIO is very resilient.
Each MinIO cluster is a set of distributed MinIO servers with one process per node. MinIO runs as a single process in user space and uses lightweight coroutines to achieve high concurrency. Drives are grouped into erasure sets (16 drives per group by default), and objects are placed on these sets using deterministic hashing algorithms.
MinIO is designed for large-scale, multi-data center cloud storage services. Each tenant runs its own MinIO cluster, completely isolated from other tenants, enabling them to protect themselves from any disruption of upgrades, updates, and security events. Each tenant scales independently through a federated cluster across geographies.
MinIO server function
MinIO’s enterprise-class features represent the standard for object storage space. From the AWS S3 API to S3 Select and our implementation of inline erase coding and security, our code is widely acclaimed and often copied by some of the biggest names in technology and business.
MinIO object reserved
By default, each new write to a unique object name causes the object to be overwritten. You can configure MinIO to create a mutated version of each object to keep a complete history of that object. MinIO also supports write once read many times (WORM) locking versioned objects to ensure complete immutability for a specified duration or until explicitly unlocked.
Versioning and object locking are only applicable to distributed MinIO deployment:
- Bucket Versioning: Controls the version of buckets
MinIO supports multiple “versions” of objects in a single bucket. Writes that typically overwrite existing objects result in the creation of new versioned objects. MinIO version control protects against accidental overwrites and deletions, while supporting “undo” writes. Bucket versioning is a prerequisite for configuring object locking and retention rules.
For versioning buckets, any write that changes an object results in a new version of that object having a unique version ID. MinIO marks the “latest” version of the object that the client retrieves by default. The client can then explicitly choose to list, retrieve, or delete a particular object version.
Among them:
- Objects with a single version: MinIO adds a unique version ID to each object as part of the write operation.
2. Objects with multiple versions: MinIO keeps all versions of objects and marks the latest version as “latest”.
3. Retrieve the latest object version
4. Retrieve the specific object version
- WORM: Lock objects
MinIO object locking (” object retention “) enforces write once many reads (WORM) invariance to protect versioned objects from being deleted. MinIO supports object retention based on duration and indefinite legal retention.
MinIO object locking provides critical data retention compliance and meets Cohasset Associates’ SEC17A-4 (F), FINRA 4511(C), and CFTC 1.31(C) -(d) requirements.
Among them:
- Buckets without locks: MinIO versioning preserves the complete history of object mutations. However, an application can explicitly remove a particular version of an object.
- Locked buckets: The default 30-day WORM lock for objects in buckets ensures the shortest retention and protection period of all object versions.
3. Lock the deletion operation in the bucket: The deletion operation follows the normal behavior in versioned buckets, where MinIODeleteMarker creates an object. However, the non-deleted tag version of an object is still subject to the retention rule and is not affected by any particular deletion or overwrite attempt.
4. Lock versioning in the bucket: MinIO blocks any attempt to delete the version of a specific object held under a WORM lock. The earliest possible time a client can delete a version is when the lock expires.
Minio deployment and management
MinIO is a software-defined high-performance distributed object storage server. You can run MinIO on consumer or enterprise hardware, as well as on a variety of operating systems and architectures.
MinIO supports two deployment modes: independent and distributed:
- Standalone deployment: A single MinIO server with a single storage volume or folder. Standalone deployment is best suited for the evaluation and initial development of applications that use MinIO for object storage, or for providing an S3 access layer for a single storage volume. Standalone deployment does not provide access to the full set of MinIO’s advanced S3 features and functions.
- Distributed deployment: One or more MinIO servers with at least four total storage volumes on each server. Distributed deployment is best suited for production environments and workloads and supports all of MinIO’s core and advanced S3 features and functions. For production environments, MinIO recommends a baseline topology of four nodes and four drives.
Copyright notice: This article is the blogger’s original article, in accordance with the relevant copyright agreement, if reprinted or shared, please attach the original source link and link source.