Add support for auto pagination when fetching results

[migsc]

Hi, I'm currently working on a project where this would be useful. For each method that fetches paginated results, it'd be nice if there existed a convenience method that fetched all the pages in parallel.

Take getMySavedTracks for example, which has a maxinum number of 50 results per page. We'd have a method called:

getAllMySavedTracks(maxPages, optionsEach, callbackEach, callback)

Which fetches the first page, checks the total items, and creates requests if needed for each page using a limit and offset. The parameters could work as follows:

• maxPages: The maximum number of pages to get. By default it would request all of them (unless there's a rate limit for pages Spotify imposes that I don't know of).
• optionsEach: The object for options to send in each call to getMySavedTracks. Passing limit and offset would have no effect since that's automatically determined per each page request in this method.
• callbackEach: A callback function for each page request i.e. each call to getMySavedTracks. The callback function receives the error and value parameters of course.
• callback: A callback function that fires after completion of all requests. Just going off of how Promise.all() works, you could pass it the array of values from all the resolved getMySavedTracks promises in the second parameter, and If any of the page requests fail, you only get the value of the promise that was rejected in the first parameter.

Currently, I do this on my own and it gets messy fast. Putting it in the API wrapper could save developers some headaches. If the paging could somehow be generalized, you'd end up with several clean convenience methods. The main usecase for this is for when dealing with a user who has a large music collection (personally I hoard and create too many playlists).

Let me know what you think. I'm totally willing to implement this, as I already have existing code I could refine to work within the wrapper.

[JMPerez]

Hi @mchateloin,

This sounds really interesting. I have thought of that in the past and it all depends on how smart you want the wrapper to be. At the moment it is just a mapping 1:1 with the Web API, which works well for most cases except for the ones you mentioned.

I already had a similar situation when coding the Spotify Deduplicator. In the end I would fetch the first page of items to get the length of the collection, and then would generate promises for the rest of the pages.

The main issue with having methods returning so many items is that they hide the undergoing requests. Due to the rate limiting of the Web API, one can easily exceed it when fetching all user's saved tracks or all user's playlists. Actually, that is what happened with certain users with a very large collection of music.
So I decided to throttle the promises, delaying them a little bit to make sure I wasn't making too many requests, following an approach which I moved later to its own package.

In the end I thought that the cases in which one would have to traverse many pages in a collection would be few, and I wanted for the developer to be able to set a certain rate limit, so the wrapper wouldn't blindly make a ton of requests.

I think a good start is that you create something on top of the wrapper that can generically manage pagination and get X results or all results. And internally it would handle queueing the requests and managing rate limiting.

[migsc]

Good point. I haven't run into problems with the rate limits yet so I wasn't taking that into account. I tried to find more information on those limits, but the Spotify Web API docs doesn't say much about numbers. Don't know if this is published elsewhere?

So I'm traversing playlists in my project too. And I'm looking for all the artists from each track. With your Spotify Deduplicator project, do you remember roughly what amount of requests in a short time it took to hit the rate limit? Looking at your code in spotify-dedup, I see you're setting your promise queue to 10 requests per second. Is that usually a safe bet?

[JMPerez]

Yes, the documentation doesn't specify a certain number or requests. I thought that 10 req/sec was enough, taking into account that it might happen that other user used the site at the same time, contributing to the amount of requests for that same client_id.

The good thing is that I can always go back and decrease if users complain.

[migsc]

Cool, thanks for the advice!

[wmitchel]

Is this still the recommended way to handle pagination with the new release?

[omarryhan]

I made this Typescript auto pagination helper. Unfortunately it doesn't run in parallel, but I hope some would find it useful:

interface Pagination {
  next?: string;
  items: object[];
}

export const getAllPages = async <Response extends Pagination>(
  request: Promise<Response>,
): Promise<Response> => {
  const paginatedResponse = await request;

  let currentResponse = paginatedResponse;

  while (currentResponse.next) {
    currentResponse = await spotifyApi.getGeneric(
      currentResponse.next,
    ) as Response;
    paginatedResponse.items = paginatedResponse.items.concat(currentResponse.items);
  }

  return paginatedResponse;
};

To call it:

const paginatedResponse = await getAllPages<SpotifyApi.PlaylistTrackResponse>(
  spotifyApi.getPlaylistTracks(playlistId, { limit: 50 }),
);