As consumer expectations rise, API performance has never been more important than it is today. It is well known that more than half of web users will stop browsing a web page if it takes longer than three seconds to load.

These expectations do not necessarily meet the technical requirements of the API. In the era of big data consolidation and analysis, apis are handling more data on the back end than ever before. To truly gain a foothold in today’s digital economy, apis must be optimized for maximum efficiency. API paging is a key strategy to ensure smooth and efficient API operation.

Let’s take a look at what paging is, and then dive into API paging with sample code.

What is paging?

Search content on the search engine, browse goods on the shopping platform, read news on the news platform, review comments on weibo and other scenarios can see the function of excessive page, different occasions have different ways, such as the common page number paging, roll to the bottom of the automatic loading page.

Paging information reduces user traffic in terms of user experience and reduces server stress in terms of application performance. Imagine an API query from a database that could return millions or even billions of results, a disaster for the server and browsing.

So paging helps limit the number of results to control network traffic.

Offset paging

Offset paging is one of the simplest implementations and is typically implemented using the limit and offset commands. Offset pages are affected by the use of database SQL, and limit and offset are included in the SQL SELECT library.

The protocol for API requests using limit and offset is as follows:

GET /items? limit=20&offset=100Copy the code

Offset paging requires little programming, is also stateless on the server side and works regardless of the custom SORT_BY parameter.

The disadvantage of offset paging is that errors can occur when processing large offset values. For example, if the offset is set to 1000000, the API must scan a million database entries and then discard them.

Another disadvantage of offset paging is that adding new entries to the table can lead to confusion, which is called page drift. Take the following scenario:

  1. Start with the queryGET /items/? offset=0&limit=20
  2. Add 10 new entries to the database
  3. Run the same query again, and this only returns 5 results, because adding 10 entries to the database moves the offset back to 10, which can cause confusion on the client side.

Key set paging

Keyset paging uses the filter values of the previous page to determine the next set of items, indexing those results.

Here’s an example:

  1. The client requests the nearest project GET/items? limit=20
  2. After clicking the next page, the query looks for the minimum creation date2021-09-01 00:00:00. Then, create a query filter for the next page,GET /items? limit=20&created:lte:2021-09-01 00:00:00
  3. And so on…

The advantage of this approach is that it requires no additional back-end logic, just a limit parameter. It also has consistent sorting capabilities, even when new items are added to the database, and works smoothly for large offsets.

The search page

Search paging is the next step beyond keyset paging, adding the queries after_id and before_id to remove filter and sort constraints. Unique identifiers are more stable than low-cardinality fields, such as state enumerations or category names.

The only downside to search paging is that creating a custom sort order can be challenging.

Here’s an example:

  1. The client requests a list of recent projectsGET items? limit=20
  2. The client uses the results of the first query to request a list of the next 20 itemsGET /items? limit=20&after_id=20
  3. The client requests the next/scroll page, using the last entry on the second page as the starting pointGET /items? limit=20&after_id=40

Benefits of search pagination:

  • Separate filter logic from paging logic
  • Consistent sorting, even as new items are added to the database, of the most recently added items.
  • Even if the offset is large, it will work well.

Performance optimization

API data paging is a method of improving service response time based on reducing data transfer traffic. Access performance to the server is also different for different paging methods. Regardless of the language used behind the scenes, API paging can be broadly divided into two categories: memory paging and data source paging.

paging

This is a common way of paging, initiated the client through the API page request, the server receives the corresponding request is then transmitted to the database server build SQL query language and returns the corresponding query result data set is temporarily stored in memory, the server again in the heart of the memory paging data needed for the response to the client.

The common implementation method is simple, and the efficiency of paging query for a small amount of data is acceptable, because the query paging based on ORM mechanism will lose performance and affect efficiency in the conversion process, and the efficiency will be significantly reduced for a large amount of data.

Data source paging

Paged data is usually returned from a stored procedure (usually found in a relational database, such as Sql Server, Mysql, etc.). Paged data is sorted out at the source.

conclusion

As apis become more involved and complex, API paging performance becomes more important.