The stack is a cloud-based, native, site-based PaaS for data, and we have an interesting open source project on GitHub and Gitee: FLINKX, FLINKX is a unified data synchronization tool based on FLINK batch stream, which can collect static data, but also can collect real-time changing data. It is a global, heterogeneous, batch stream data synchronization engine. If you like, please give us a star! Star! Star!
Making open source projects: https://github.com/DTStack/fl…
Gitee open source projects: https://gitee.com/dtstack_dev…
If the digital transformation of the enterprise is a bicycle, technology and data are the two wheels of the bicycle. On the wheels are the strategy, culture and resources that determine where the transformation will go. No matter how right the digital transformation is and how clever the tactics are, the end result will be on these two wheels.
In the DT era, “big data” does not emphasize large amount, but multiple data sources and dimensions. Many systems now have data that cannot be used by third parties, and only by breaking down these barriers can new value be created. In the process of digital intelligence transformation, the realization of open data sharing is the premise for enterprises to achieve curve overtaking in the DT era.
1. What is data sharing service
Data sharing service can be simply summarized as sharing the data of the data middle station to other systems, or sharing the data of an organization to other organizations. There are also various ways to provide data, such as database reading, file transfer, API interface service and so on.
How to provide data service and share data through data API?
Data API, through RestfulAPI provides data services, suitable for database does not directly open to the public, through interfaces with high concurrency quick return of data services, such as the result of the China enterprise internal data processing data, through the way of data API, provided to the upper data applications, data portal, visual screen, etc.; Securities companies will stock, bond and other market data through the data API to provide external customers; New media enterprises provide information to external customers through API, mainly to solve the scenario of rapid data sharing.
How to generate data services
Traditionally, generating API interfaces requires back-end developers to write in languages such as Java or Python. From the beginning of the generation to the external release, in the middle to do some authentication, flow limit, etc., the whole process is long, and after the completion of an interface development, it is necessary for testers to test and verify, the whole process down, the investment cost is high.
Here is the process of generating a data API in the traditional way:
In recent years, in the wave of data middle platform, the supply of big data platform products is more and more available. Some excellent domestic big data product suppliers also have standardized data sharing service products, which encapsulate the data service capabilities and complete most of the functions inside the platform products. The user-oriented functions are only connecting data sources and writing query logic, which greatly shortening the API process and reducing development costs.
Using standardized products, the general development process is as follows:
In the API generation, the following steps are taken:
- API generation: the platform will encapsulate the ability to create API. Users only need to select the library and table on the Web interface, and set the request parameters and return parameters. For complex API query logic, some platforms will also provide custom SQL query logic mode to meet the user’s different scenarios.
- API publishing: the platform will integrate the API gateway without the user writing the gateway logic. After the API is published, it will be directly published on the API gateway to form the API market and provide external data services.
- API application: Users in need can directly apply for published API in API, administrator for approval, after approval, the applicant can get the API call address and request example.
- API authorization: to approve the application of API applicants;
- API management: For enterprises, a standardized product provides an enterprise-level data services management platform and an enterprise-level API marketplace.
In the process of API call, API gateway can authenticate, limit flow, decrypt data and so on.
(1) Compared with the traditional generation method, what can the standardized product bring?
- Development efficiency improvement: shorten the data API development process, one API generation only takes 3 minutes
- Manpower cost reduction: the traditional way requires back-end development write interface, now just more familiar with the development of data, by writing SQL, you can complete the development of data API.
- API interfaces serve more scenarios: In addition to the user’s original requirements, standardized products can bring more additional functionality, such as API, user flow limiting, API call monitoring, etc.
- Enterprise API market: achieve unified management of enterprise API, unified API market open data services.
(two) the stack Dtinsight DataAPI products are oriented to the above scenarios, providing data API sharing services.
DataAPI, which generates and registers APIs through dual-mode visual configuration, quickly builds OneService data services, forms an enterprise-level API market and API service management platform, and improves the efficiency of data opening and sharing.
What does DataAPI do
In addition to the scenarios that can be solved as described above, DataAPI also has more in-depth research in third-party data service, service monitoring, and data service security, providing customers with an excellent data service product.
- Dual mode generation API
API service is generated by wizard mode and custom SQL mode, and API creation is completed visually in 3 minutes.
- Third Party Service Registration
In addition to generating API services on the platform, it supports the registration of the original API services to DataAPI for unified management and release to the API market, and the unified management of all API services of the enterprise.
- API call monitoring
The platform provides monitoring of API call times, call logs and error logs from the perspective of API manager and API applicant.
- API uses permission control
You can control the user access of a single API, how many times it is called, and how long the call cycle is.
4. Data service security
Data security is a very important part in the process of data external service. DataAPI guarantees data security in three aspects:
- API call
API call provides two encryption methods, which can encrypt user information, API information and data information, and is suitable for scenarios with different security levels.
- API current limiting
In order to guarantee the security and stability of data service, the flow limit can be carried out according to the number of API calls per second and the number of API calls by users.
- Black and white list IP control
Access to IP from IP level is controlled by setting a black and white list.