Building Cursor-Based Pagination with Redis and .NET for External API Integrations

When building modern APIs that integrate with external systems, a common challenge is enabling efficient, scalable pagination — especially when the external system doesn’t support pagination natively. This post walks through a practical, production-ready solution using cursor-based pagination and Redis caching in a .NET Web API environment.

The Problem

Let’s say your API needs to fetch large datasets (like order history, customer records, etc.) from an external third-party service. You might run into these issues:

No native pagination: The external system provides data as a whole without page support.
Data volatility: Between client page requests, the dataset might change, leading to inconsistent results.
High latency and cost: Fetching large datasets for each client request from the source is slow and expensive.
Need for consistent paging: Clients expect a reliable and intuitive way to browse results.

Traditional offset-based pagination (e.g., ?page=2&limit=10) can lead to skipped or repeated records when the underlying data changes. Cursor-based pagination, where the client receives a “bookmark” (cursor) for where to continue, offers a more stable solution.

The Strategy

To overcome these challenges, our architecture uses caching and encoded cursors:

Initial Fetch and Cache:
- When the client submits a query (e.g., customerId=123), we fetch the full result from the external API.
- The result set is cached in Redis using a key based on the query parameters.
Serving Pages:
- The API slices the cached result based on a cursor and returns a portion of the dataset.
- A new cursor is returned with each response so the client can request the next set.
Expiration:
- Redis cache keys are short-lived (e.g., 30 minutes), ensuring stale data is not reused too long.
Stateless Clients:
- Clients don’t need to store or track offsets — the cursor encapsulates the position.

Tools & Tech Stack

Here’s what you’ll use:

.NET Web API (C#): To expose REST endpoints.
StackExchange.Redis: For high-performance Redis access.
System.Text.Json: For efficient object serialization.
Base64 Encoding: To safely encode the offset as a cursor.

Building the Solution

1. Define Your Data Model

We’ll use a simple OrderRecord model to represent each record.

public class OrderRecord
{
    public string OrderId { get; set; }
    public DateTime CreatedAt { get; set; }
    public string CustomerId { get; set; }
}

2. Cursor Utility

Encodes/decodes page indexes so clients get a Base64 string instead of a raw integer.

public static class CursorHelper
{
    public static string Encode(int index) =>
        Convert.ToBase64String(BitConverter.GetBytes(index));

    public static int Decode(string cursor)
    {
        try
        {
            return BitConverter.ToInt32(Convert.FromBase64String(cursor), 0);
        }
        catch
        {
            return 0;
        }
    }
}

3. Redis Cache Service

This abstracts Redis logic: caching the full dataset and reading it back.

using StackExchange.Redis;
using System.Text.Json;

public class RedisCacheService
{
    private readonly IDatabase _redisDb;

    public RedisCacheService(string connectionString)
    {
        var redis = ConnectionMultiplexer.Connect(connectionString);
        _redisDb = redis.GetDatabase();
    }

    public async Task CacheResultsAsync(string key, List&lt;OrderRecord> records, TimeSpan expiry)
    {
        var json = JsonSerializer.Serialize(records);
        await _redisDb.StringSetAsync(key, json, expiry);
    }

    public async Task&lt;List&lt;OrderRecord>> GetCachedResultsAsync(string key)
    {
        var json = await _redisDb.StringGetAsync(key);
        return json.HasValue ? JsonSerializer.Deserialize&lt;List&lt;OrderRecord>>(json) : null;
    }
}

4. API Controller

Your endpoint for cursor-based paging:

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly RedisCacheService _cache;
    private readonly IExternalApiService _externalApi; // Abstracted external API call

    public OrdersController(RedisCacheService cache, IExternalApiService externalApi)
    {
        _cache = cache;
        _externalApi = externalApi;
    }

    [HttpGet]
    public async Task&lt;IActionResult> GetOrders([FromQuery] string customerId, [FromQuery] string cursor = null, [FromQuery] int pageSize = 50)
    {
        string cacheKey = $"orders:{customerId}";
        var records = await _cache.GetCachedResultsAsync(cacheKey);

        if (records == null)
        {
            records = await _externalApi.FetchOrdersAsync(customerId);
            await _cache.CacheResultsAsync(cacheKey, records, TimeSpan.FromMinutes(30));
        }

        int startIndex = string.IsNullOrEmpty(cursor) ? 0 : CursorHelper.Decode(cursor);
        var paged = records.Skip(startIndex).Take(pageSize).ToList();

        string nextCursor = (startIndex + pageSize) &lt; records.Count
            ? CursorHelper.Encode(startIndex + pageSize)
            : null;

        return Ok(new
        {
            data = paged,
            nextCursor = nextCursor
        });
    }
}

Example Response

{
  "data": [
    { "orderId": "1001", "createdAt": "2024-05-10T10:00:00Z", "customerId": "ABC123" },
    ...
  ],
  "nextCursor": "NTA=" // Base64 of offset 50
}

The client can use nextCursor=NTA= in the next request to get the next page.

Benefits of This Approach

Performance: One-time call to external service; Redis serves repeated requests.
Consistency: All paging is done over a snapshot, avoiding data drift.
Scalability: Redis handles concurrent access well.
Statelessness: The API doesn’t manage client state; cursors do.
Resilience: Short TTLs help avoid stale or inconsistent results.

Final Thoughts

This is a highly reusable and scalable architecture for any API that needs to serve large datasets retrieved from an external system that lacks pagination support. By combining Redis caching with cursor-based pagination, you provide fast, stable, and user-friendly endpoints.

You can even extend this model to include:

Filtering and sorting logic.
Metadata responses like total counts.
Multi-user cache invalidation.

Bryan Avery – Blog