Creating an API with python: Part 13: Pagination

In my previous post, Creating an API with python: Part 12: Database Character Encoding and Connections, I made changes to the database code and schema to enable utf-8 encoding and prevent database connections from dropping. In this post, I will add pagination to the FastAPI API. Pagination is important for the scalability of the API. Without it, the returned datasets would get larger and larger as the database grows, meaning the response times would get slower and slower, and the return data more unwieldy to work with.

Prerequisites

These prerequisites are assumed for this post:

1. Creating an API with python: Part 1: GET Endpoints
2. Creating an API with python: Part 2: MariaDB Database
3. Creating an API with python: Part 3: POST Endpoints
4. Creating an API with python: Part 4: DELETE Endpoints
5. Creating an API with python: Part 5: Authentication
6. Creating an API with python: Part 6: HTTPS and Proxying
7. Creating an API with python: Part 7: CORS
8. Creating an API with python: Part 8: Multiple Account Support
9. Creating an API with python: Part 9: Authentication Scopes
10. Creating an API with python: Part 10: Integration Tests
11. Creating an API with python: Part 11: Running as a Service
12. Creating an API with python: Part 12: Database Character Encoding and Connections

As a reminder, the code for the FastAPI API is now on GitHub, with the project name taglink-api. You can checkout the latest copy of the code here: https://github.com/liz-allthecoding/taglink-api. Note that all the code changes for this blog entry can be found here: https://github.com/liz-allthecoding/taglink-api/pull/1/files. so I won’t be adding all required code changes here.

Step 1: Update config.yaml

Update config.yaml with a new config key, api_url. The value should be the url where your API is hosted, e.g. “https://some.domain/api”.

Step 2: Add a Pagination Decorator

Update manager/manager.py with a decorator that will wrap get functions and generate a response dict that includes the results and pagination data. The code added will be:

def pagination(endpoint: str):
    def deco(func):
        @wraps(func)
        def inner(*args, **kwargs):
            results = func(*args, **kwargs)
            pag_prev = None
            pag_next = None
            api_url = CONFIG.get('api_url')
            count = len(results)
            offset = kwargs.get('offset', DEFAULT_OFFSET)
            limit = kwargs.get('limit', DEFAULT_LIMIT)
            params = deepcopy(kwargs)
            for key, value in kwargs.items():
                if value is None:
                    del params[key]

            if params.get('offset'):
                del params['offset']

            if offset != 0:
                prev_offset = offset - limit
                if prev_offset <= 0:
                    prev_offset = 0
                params['offset'] = prev_offset
                encoded_params = urlencode(params)
                pag_prev = f'{api_url}/{endpoint}/?{encoded_params}'

            next_offset = offset + limit
            if count == limit:
                params['offset'] = next_offset
                encoded_params = urlencode(params)
                pag_next = f'{api_url}/{endpoint}/?{encoded_params}'

            pagination_dict = {'count': count, 'prev': pag_prev, 'next': pag_next}
            return {'results': results, 'pagination': pagination_dict}
        return inner
    return deco

Step 3: Refactor Manager GETs

Refactor the code in manager/manager.py to use the pagination decorator. I found it useful to have separate wrapper functions that are decorated with the pagination decorator, which then call the get_xs function underneath. This is so your database retrievals can be separated from the retrievals that need response data. You will also need to refactor the get_xs functions to take offset and limit parameters, and to use them in the database query. For example, the new function get_links_with_pagination is decorated with @pagination and calls get_links underneath. The get_links function now adds the limit and offset to its query. In addition, where functions call other functions, you need to add loops to get all the data, as by default only 200 records (the default limit) will be returned otherwise.

Step 4: Refactor main.py

The get_xs endpoints need refactoring to add offset and limit params, and pass them to the new get_xs_with_pagination manager functions. Note that a default offset and limit should be specified in the event that no default or limit are passed to the API.

Step 5: Refactor the Integration Tests

Refactor the integration tests to include tests for pagination, and to take the new response format into account. Note that the response format for get_xs endpoints will now be of this structure:

{
"results": [],
"pagination": {'count': 0, 'prev': "/some/url", 'next': "/some/url"}}
}

Note that the get_xs functions in test_base.py will need refactoring with offset and limit params, in order to be able to call the API with those params.

Step 6: Run the Integration Tests

Run the integration tests to verify that the API (and the tests) are working correctly.

1. Ensure that the service is running:

   $ sudo systemctl start taglink-api
   

2. On your server, change to the code directory (~/vboxshare/fastapi should be replaced with the path to your FastAPI python code):

   $ cd ~/vboxshare/fastapi
   

3. Run the integration tests:

   $ ./integration_test.sh
   

Conclusion

You should now have a FastAPI API that supports pagination. This means that as your dataset gets larger, your response times won't get slower and slower as your API tries to return more and more data.

Thanks for reading!