When I was at CMB card Center, I developed an OSS(object storage Service), which was internally called FSP(File Service Platform). Positioning is very similar to qiniu/Yupai; Kind of a private OSS. OSS is something that can be complicated or simple, but once you have it, it can be a boon to developers and greatly increase development efficiency. In general,OSS is very popular among developers, easy to promote internally, and everyone is willing to use it, sometimes to the point of abuse. A lot of static resources are hosted and distributed through FSP these days.

In the few companies I’ve worked with, I’ve noticed that as long as they’re a little bigger, they’re less willing to use public cloud services. They prefer to deploy their own services, or even build their own wheels. Even services that are less coupled tend to do it themselves rather than using the public cloud, and I think most developers working on medium to large development teams will feel the same way. If this is the case for your team, I think your team also needs a more comprehensive object storage service.

I’ve had a lot of questions like:

A WEB site must have a lot of static resources: images,CSS,JS, static content and so on… However, in order to ensure the high availability of the website, we need cluster deployment. After cluster deployment, the consistency problem is obviously brought: when the new picture A is added, node 1 is saved, what about node 2? Especially in static scenarios such as CMS, how to keep static resources synchronized between multiple nodes? If hit UGC scene is even more horrible ~ do you want to save DB or local? At that time, we tried to use a scheme like RSYNC for synchronization between nodes, but because such sites are difficult to separate reads and writes, all synchronization strategies need to use mutual synchronization…… There are always problems after using an event. Against this background, most teams naturally spawn the simplest object storage service. In general, his architecture looks like this:

There was no initial problem with this architecture, where static resources were centrally stored and uniformly returned to users by static resource servers. Static resource servers can also be added to the CDN and all seems well, often many teams end their development there. But as usage increases, demand will become harder to meet:

Authorized to access

Applying Resource Isolation

Resources Automatically Expire

Multimedia Resource processing

The watermark

Preventing hotlinking

There will be a flood of requirements, even for high availability and limiting traffic, that can’t be met with the above architecture. The storage, management, configuration, migration of these resources will become your problem. And there seems to be no way around these problems. It is the need that will eventually be faced. At this point, all you need is an OSS!

Here’s a quick overview of each service:

  1. CDN: Needless to say, CDN services provided by professional CDN service providers are generally purchased for Internet caching.

  2. WEB write API: Typically used for write interaction with users, either end users or servers. I would recommend that this set of apis provide CORS support, which is too development-friendly in UGC scenarios. Developers only need to distribute token+ upload policies, and the end user uploads traffic that never passes through the business side of the application server saves a lot of development. I used to think that qiniu had a bad taste in this part of THE API, and felt that the API was very Dirty. It was not until later when I uploaded more and more things to set that I found this kind of Dirty stuff was very cool, ha ha ha! The user can provide a lot of application handling information at upload time: expiration, file handling, naming rules, etc.

  3. CDN service invocation: When some static resources are updated, CDN needs to be notified in time for cache refresh; CDN service should be notified to prefetch resources in case of sudden heavy traffic access. These calls to the CDN service can usually be handed over to the service for asynchronous calls.

  4. Web read API: This layer is usually directly connected to the CDN and provides authentication functions as standard. If you are confident of the image processing performance of your service, you can add some small functions such as image scaling.

  5. Authentication service: this service is specially used to connect to the CDN. Generally,CDN manufacturers will provide two kinds of access authentication services :(1) CDN mode based on expired signature. Both parties agree on an algorithm based on expired signature. CDN caches your source data, and the subsequent authentication is completed by CDN for you. If this can meet your needs, there is no need for the authentication service. (2) The self-defined authentication mode. In this mode, a parameter is usually specified as the authentication token, and the CDN will call your authentication service for authentication after obtaining the authentication token. So your authentication logic can be completely autonomous, based on the number of calls, based on the time, and so on. Often CDN vendors are not very willing to do this matter, and they need to negotiate a lot of talk.

  6. Clustered storage file systems: This service provides the core services of OSS, namely efficient file read and write, availability assurance, and reliability assurance. For self-build, unless the development team is very high-minded, try not to do it yourself. There are plenty of open source and commercial solutions out there :SeaWeedFS,FastDFS,TFS. All of these solutions can be used directly.

  7. Object processing service: This service is mainly used for asynchronous processing of complex tasks, such as video stitching/cutting, image watermarking/compression, bulk file packaging, and other time-consuming processing tasks. If these tasks are many and complex, it is also recommended to open a second Web service to receive these service requests from users.

  8. Background management and testing tools: THIS I did not draw on the architecture diagram, many enterprise service developers actually have this problem, to manage the background of contempt. In fact, this thing must be carefully developed for the system, it can greatly reduce your workload, try to allow users to self-manage their own object space and development testing, can let you save a lot of time.

The basic set of OSS consists of these components, which are not very difficult to develop. You can refer to some functions provided by public cloud services to add to your OWN OSS. In today’s world of front and back end separation, even some developers can use OSS directly as the front end WEB SERVER, which is much easier than configuring a bunch of Nginx and then trying to run the Maintenance CDN. This set of services put up static resources hosting and distribution problems can be greatly solved, cost-effective. In the former factory, due to compliance policies and other reasons, many of the functions I really want to join, I always want to write a set of OSS to open source, but recently saw seven Cow began to support private cloud deployment, the implementation of this plan has been reduced a lot of motivation (mainly lazy)… I hope every company has its own set of mature, reliable and easy-to-use OSS, and don’t waste precious time on trivial matters.