How to paginate records in Airtable when using the API

Airtable’s API is great and so is their documentation. However, I came across a small stumbling block while using it to build a simple project.

The problem, Airtable returns a maximum of 100 records in each page of results. If you have more than 100 records or want to limit the records returned in each request, by specifying the pageSize parameter, you will need to paginate the results.

However, Airtable doesn’t provide helper functions for pagination so you need to build it yourself. The API also doesn’t give you a total for all the records in a base or an offset to work with (at least not when using NodeJs) so our solution has to calculate these. 

Solution

I’m using NodeJs + ExpressJs and Airtable’s Node Client to demonstrate but the logic can be applied to other frameworks or vanilla Javascript. The final function is below.

exports.getModels = (req, res) => {
  let records = [];
  const limit = 24;

  // called for every page of records
  const processPage = (partialRecords, fetchNextPage) => {
    records = [...records, ...partialRecords]
    fetchNextPage()
  }

  // called when all the records have been retrieved
  const processRecords = (err) => {
    if (err) {
      console.error(err)
      return
    }

    //process the `records` array and do something with it
    const count = records.length;
    const page = req.params.page || 1;
    const pages = Math.ceil(count / limit);
    const offset = (page * limit) - limit;

    let models = records.map( record => {
      return {
        _id: record.getId(),
        name: record.get('Full Name'),
        photo: record.get('Headshot')
      }
    }).slice(offset, limit*page);


    res.render('models', { title: 'Models', models, page, pages, count });
  }

  base('Models').select({
    view: 'Grid View',
    pageSize: 24
  }).eachPage(processPage, processRecords)

}

Let’s break it down

The Airtable API provides a select method for listing records. The select method returns a query object. To fetch the records matching that query, you can use either the eachPage or firstPage methods of the query object.

To filter, sort, and format the results we can pass an options object, in this case we only need to pass in the view and pageSize parameter. The pageSize parameter sets the number of records returned in each request and must be less than or equal to 100. A default imposed by Airtable. An example function is provided in the API Documentation.

We are going to use the eachPage method to loop through all the records in the base page by page and store this in our records array variable declared at the top. This will allow us to get the total number of records.

The eachPage method takes two callback functions — processPage and processRecords — which I have separated for ease of readability.

let records = [];

// called for every page of records
const processPage = (partialRecords, fetchNextPage) => {
  // Using ES6 destructing to add each page's record (partialRecords) to the previously fetched records
  records = [...records, ...partialRecords]
  fetchNextPage() // Call the next page of requests
}

When all the records have been fetched, eachPage will call processRecords which will set up the variables for our pagination now that we have access to all records and the total amount of records.

const count = records.length;
const page = req.params.page || req.query.page || 1;
const pages = Math.ceil(count / limit);
const offset = (page * limit) - limit;

let models = records.map( record => {
  return {
    _id: record.getId(),
    name: record.get('Full Name'),
    photo: record.get('Headshot')
  }
}).slice(offset, limit*page);

res.render('models', { title: 'Models', models, page, pages, count });

Here we get

• The total amount of records by accessing the length property and storing it in the count variable. 
• We get the current page we are on by accessing the url query parameters or if not set, default to page 1 so either /models/page/3 or models?page=3
• We then calculate the number of pages by dividing the total number of records by the amount of records we want to show which is the pageSize limit. We round it off to the next 10 to prevent decimals in the event a page has less than the limit amount of records.
• We then calculate the amount of records we want to offset or skip when traversing the records array to only show the amount we want per page. This is used in the slice method of the array. For example if the limit per page is 24 and we are on page 2, page 1 and 2 will total 48 records but we only want for 24 – 48. There in the slice method we pass 24 to start at this index in the record array and end at index 48 in the array.
• Finally we pass the models, page, pages and count variables to our view to render the page

And that’s it, I’m not going to go into detail for the entire view but I will post my navigation markup that uses these variables we calculated. The markup uses pug and I used a mixin to store the pagination markup.

mixin pagination(page, pages, count)
  .pages(role="navigation", aria-label="pagination")
    span.total-models #{count} models
    .paginate-links
      if page > 1
        a.first-page(href="/models")
          span.icon
            i.fas.fa-angle-double-left
        a.prev-page(href=`/models/page/${page - 1}`)
          span.icon
            i.fas.fa-angle-left
      .paging-input
        label(for="current-page-selector", class="is-sr-only") Current Page
        form(action="/models", method="GET")
          input.input(type="text", name="page", value=`${page}`, class="current-page", id="current-page-selector")
        span.total-pages of #{pages}
      if page < pages
        a.next-page(href=`/models/page/${parseFloat(page) + 1}`)
          span.icon
            i.fas.fa-angle-right
        a.last-page(href=`models/page/${pages}`)
          span.icon
            i.fas.fa-angle-double-right

I hope this helps someone and if you have any questions you can tweet me @ChinaraJ.