Geoapify
11 September, 2024 / by Geoapify / in Tutorials

JavaScript Tutorial: Batch Geocoding and API Rate Limits

The image illustrates the concept of bulk geocoding while efficiently managing API rate limits
The image illustrates the concept of bulk geocoding while efficiently managing API rate limits

Batch geocoding large datasets—whether it's thousands or millions of addresses—can be a rewarding task when done efficiently, especially when working within the rate limits of APIs like the Geoapify Geocoding API. Ignoring or exceeding these rate limits can result in failed requests, temporary bans, or even permanent suspension from the API service, causing significant disruptions in your data processing pipeline. Respecting rate limits is crucial to ensure reliable and uninterrupted access to the geocoding service.

This tutorial will guide you through efficiently handling large-scale geocoding tasks in JavaScript while adhering to API rate limits. We’ll start by building a geocoding solution from scratch and then enhance it using the popular @geoapify/request-rate-limiter library. By the end, you'll have a robust solution for geocoding high volumes of addresses without exceeding rate limits, ensuring smooth and efficient API requests.

Let's get started!

Part 1: Bulk Geocode with API Rate Limits with Vanilla JavaScript

Let's batch geocode multiple addresses using JavaScript while managing API rate limits. We will use the Geoapify Geocoding API and handle API requests efficiently by limiting the number of requests per second and implementing a retry mechanism for failed requests.

Prerequisites

  • Basic understanding of JavaScript and asynchronous programming (async/await)
  • Node.js installed for local development or a browser-based environment like JSFiddle
  • A Geoapify API key, which you can get here

Step 1.1: Setting Up the Environment

First, you'll need a Geoapify API key. Replace the placeholder API key in the code with your own:

const myAPIKey = "YOUR_GEOAPIFY_API_KEY";

Next, prepare a list of addresses that we want to batch geocode. Here’s an example list:

const addresses = [
  "North Oak Street, Bethalto, IL 62010, USA",
  "2809 College Avenue, Alton, IL 62002, USA",
  "1 Lock and Dam Way, Alton, IL 62002, USA",
  // Add more addresses here
];

Step 1.2: Creating the Geocoding Function

Now, let's create a function that sends a geocoding request to the Geoapify API for each address. The function will also handle errors and retries for failed requests.

const geocodeAPI = "https://api.geoapify.com/v1/geocode";
const maxRetries = 2;  // Retry limit for failed requests
const retryDelayInMillis = 500;  // Delay between retries

async function callGeocode(address, attempt = 1) {
  const url = `${geocodeAPI}/search?text=${encodeURIComponent(address)}&limit=1&apiKey=${myAPIKey}`;
  try {
    const response = await fetch(url);
    if (!response.ok) {
      throw new Error(`HTTP error occurred. Status: ${response.status}`);
    }
    const data = await response.json();
    return data.features.length ? data.features[0] : { error: "Not found" };
  } catch (error) {
    if (attempt <= maxRetries) {
      await new Promise(resolve => setTimeout(resolve, retryDelayInMillis));
      return callGeocode(address, attempt + 1);
    } else {
      return { error: error.toString() };
    }
  }
}

This function:

  • Sends a geocoding request for each address.
  • Retries failed requests up to maxRetries times with a delay between attempts.
  • Returns geocoded data or an error message if the address is not found.

Step 1.3: Managing API Rate Limits

Geoapify enforces rate limits, meaning you can only send a limited number of requests per second. To stay within this limit, we need to batch our requests. We can implement a function that sends requests in batches, ensuring we don't exceed the allowed requests per second (RPS).

const rpsLimit = 5;  // Maximum requests per second
const delayBetweenBatchesInMillis = 1000;  // Delay between batches

async function executeRequestsWithRPS(geocodeRequests, rps) {
  const result = new Array(geocodeRequests.length);
  const promises = [];

  for (let startIndex = 0; startIndex < geocodeRequests.length; startIndex += rps) {
    const endIndex = Math.min(startIndex + rps, geocodeRequests.length);
    console.log(`Sending from ${startIndex + 1} to ${endIndex} addresses`);

    const batch = geocodeRequests.slice(startIndex, endIndex).map((execute, index) =>
      execute().then(res => {
        result[startIndex + index] = res;
      })
    );

    promises.push(...batch);

    if (endIndex < geocodeRequests.length) {
      await new Promise(resolve => setTimeout(resolve, delayBetweenBatchesInMillis));
    }
  }

  await Promise.all(promises);
  return result;
}

This function:

  • Batches the geocoding requests based on the defined rpsLimit.
  • Sends requests in groups and waits for a specified delay between batches.

Step 1.4: Putting It All Together

Now, we combine the previous functions to create the full batch geocoding solution. We'll map over the list of addresses and execute the geocoding requests while respecting the API rate limits.

async function geocodeWithRpsLimit(addresses, rps) {
  const geocodeRequests = addresses.map(address => () => callGeocode(address));
  return await executeRequestsWithRPS(geocodeRequests, rps);
}

Try the Code on JSFiddle

You can try out the full geocoding solution directly on JSFiddle! This allows you to see the code in action without setting up a local environment. The JSFiddle version includes all the functionality for batch geocoding and rate limit management.

To test it:

  • Click the "Geocode with RPS" button to trigger the batch geocoding process.
  • Check the browser console to see the geocoded results.

This gives you a quick way to experiment with the code and adjust it to suit your project needs.

Part 2: Batch Geocoding with @geoapify/request-rate-limiter

In this part of the tutorial, we will introduce the @geoapify/request-rate-limiter library, which simplifies managing API rate limits when batch geocoding large datasets. This library automates the process of controlling the number of requests sent per second and handles retries, making it easier to process thousands or even millions of addresses efficiently.

Why Use @geoapify/request-rate-limiter?

When you're dealing with large-scale batch geocoding, managing API rate limits manually can become complex, especially as the dataset grows. The @geoapify/request-rate-limiter library was designed to help developers:

  • Manage Rate Limits: It controls the number of API requests sent per time period(second or minute), ensuring compliance with the rate limits imposed by services like the Geoapify Geocoding API.
  • Batch Processing of Requests: The library returns addresses in batches, allowing results to be saved in smaller file sizes.
  • Progress Notifications: The library provides real-time updates on the progress of the requests, helping you track completed and pending requests during the batch geocoding process.

Efficient for Large Datasets: Whether you're processing thousands or millions of addresses, the library ensures that requests are made efficiently without hitting the rate limits, minimizing downtime.

Step 2.1: Installing the Library

To get started, install the library using npm:

npm install @geoapify/request-rate-limiter

Also, install node-fetch if it's not already present in your project:

npm install node-fetch

Step 2.2: Configure the Geocoding API

You need to specify the Geoapify Geocoding API URL and your personal API key. The following code also reads a list of addresses from a file for processing:

import fs from 'fs';
import fetch from 'node-fetch';
import RequestRateLimiter from '@geoapify/request-rate-limiter';

// Read addresses from a file
const addresses = fs.readFileSync('addresses.txt', 'utf8').split('\n').filter(address => !!address);

const GEOCODING_API_URL = 'https://api.geoapify.com/v1/geocode/search?limit=1&format=json';
const API_KEY = 'YOUR_API_KEY';

Step 2.3: Create Geocoding Requests

The function createGeocodingRequest creates individual geocoding requests for each address, sending them to the Geoapify API:

const createGeocodingRequest = (address) => {
  return async () => {
    const response = await fetch(`${GEOCODING_API_URL}&text=${encodeURIComponent(address)}&apiKey=${API_KEY}`);
    if (!response.ok) {
      return { address, error: `Failed to fetch for ${address}: ${response.statusText}` }; 
    }
    const data = await response.json();
    if (data.results.length) {
      // get the first result
      return { address, result: data.results[0] };
    } else {
      return { address, error: `Address not found` };
    }
  };
};

// Prepare an array of request functions for the rate limiter
const requests = addresses.map((address) => createGeocodingRequest(address));

Step 2.4: Batch Processing of Requests

The library returns geocoding requests in batches, ensuring you can save results in smaller file sizes. Each batch of results is saved after a certain number of requests, reducing memory usage and preventing large files from being generated:

const saveBatchResults = (batch) => {
  const filename = `geocode_results_batch_from_${batch.startIndex}_to_${batch.stopIndex}.json`;
  fs.writeFileSync(filename, JSON.stringify(batch.results, null, 2));
  console.log(`Batch from ${batch.startIndex} to ${batch.stopIndex} saved as ${filename}`);
};

Step 2.5: Configure Options for Rate Limiting

Next, configure the request-rate-limiter options, including batch size and handling for progress updates. The options allow you to manage request batching and specify how many requests should be processed before saving:

const options = {
  batchSize: 1000, // Save results after every 1000 requests
  onProgress: (progress) => {
    console.log(`Progress: ${progress.completedRequests}/${progress.totalRequests} completed`);
  },
  onBatchComplete: (batch) => {
    console.log(`Batch of ${batch.results.length} requests completed.`);
    saveBatchResults(batch);
  }
};

Step 2.6: Send Geocoding Requests with Rate Limiting

Use the RequestRateLimiter.rateLimitedRequests() function to manage how often requests are sent. The rate limiter ensures that no more than 5 requests per second are sent, with a 1000-millisecond delay between batches:

RequestRateLimiter.rateLimitedRequests(requests, 5, 1000, options)
  .then((allResults) => {
    const filename = `geocode_results_all.json`;
    fs.writeFileSync(filename, JSON.stringify(allResults, null, 2));
    console.log('All requests completed.');
  })
  .catch(error => {
    console.error('Error processing requests:', error);
  });

By following these steps, you can efficiently batch geocode addresses using the @geoapify/request-rate-limiter library while ensuring compliance with API rate limits and monitoring progress along the way.

Which Rate Limit Should I Use for Batch Geocoding?

Geoapify provides flexible rate limits depending on your API plan, which directly impacts how quickly you can process large volumes of addresses. Let’s break down the rate limits and how many addresses you can process based on the plan you’re using.

Here are Geoapify Rate Limits:

  1. Free Plan:
  • Rate Limit: 5 requests per second (RPS).
  • Daily Quota: Up to 3,000 requests per day.
  • Processing Speed: At 5 RPS, you can process 300 addresses per minute, and you'll hit the daily limit of 3,000 addresses in about 10 minutes. 
    (3,000 addresses per day) ÷ (5 requests per second) = 600 seconds (10 minutes)

This plan is ideal for small-scale or infrequent geocoding tasks.

  1. Paid Plans:
  • Rate Limit: Up to 30 requests per second (RPS).
  • Daily Quota: Depends on the specific plan, with significantly higher limits than the free plan.
  • Processing Speed: At 30 RPS, you can process 1,800 addresses per minute. This allows you to handle larger datasets much more efficiently. For example, you could process 10,000 addresses in about 6 minutes. 
    (10,000 addresses) ÷ (30 requests per second) = 333 seconds (5.5 minutes)
  1. Custom Plans:
  • Rate Limit: Custom RPS can be negotiated based on your project needs.
  • Daily Quota: Higher or unlimited quotas for enterprise-level or very large projects.
  • Processing Speed: Custom plans allow you to scale up processing for millions of addresses, with higher RPS limits and the ability to process large volumes in a short time.

How Many Addresses Can I Process with Geoapify?

The number of addresses you can process with Geoapify depends on both the rate limit (requests per second) and the daily request limit, which vary by API plan.

  1. Free Plan:

    • Rate Limit: 5 requests per second (RPS).
    • Daily Limit: 3,000 requests per day.
    • Processing Speed: At 5 RPS, you can process 300 addresses per minute. It will take about 10 minutes to reach the daily limit of 3,000 addresses.
    (3,000 addresses) ÷ (5 requests per second) = 600 seconds (10 minutes)

  2. Paid Plans (e.g., API250 Plan):

    • Rate Limit: Up to 30 requests per second (RPS).
    • Daily Limit: Higher limits, such as up to 250,000 requests per day on plans like the API250.
    • Processing Speed: At 30 RPS, you can process 1,800 addresses per minute, allowing you to process 250,000 addresses in just over 2 hours.
    (250,000 addresses) ÷ (30 requests per second) = 8,333 seconds (2.3 hours)

  3. Custom Plans:

    • Rate Limit: Custom RPS based on your needs.
    • Daily Limit: Custom daily quotas to handle large-scale projects.
    • Processing Speed: With higher RPS and daily limits, custom plans allow you to process millions of addresses efficiently, depending on your specific project requirements.

What Will Happen If Rate Limit Is Exceeded?

When you exceed the rate limit on some geocoding services, such as Google Maps or Mapbox, you may immediately encounter an error like "429 Too Many Requests", which blocks further API access until the rate resets , which blocks further API access until the rate resets. However, with Geoapify, we take a more flexible approach to ensure a smooth experience for our users.

If you temporarily exceed the rate limits or daily quota, we won’t block you immediately. Instead, we continue providing service to avoid disruptions. This approach allows for occasional overuse without sudden interruptions to your workflow.

However, if your usage consistently exceeds the rate limits or daily quotas, we kindly ask you to consider upgrading your plan to better suit your needs. Regularly exceeding limits can affect service reliability, so upgrading ensures you stay within your usage boundaries.

In cases where exceeding the rate limits negatively impacts the overall performance of our system, particularly for other users, we may enforce a temporary block on your API access until the issue is resolved. This safeguard helps maintain the productivity and efficiency of our platform for all users.

Conclusion

In this tutorial, you’ve learned how to batch geocode addresses in JavaScript while efficiently managing API rate limits using both manual approaches and the @geoapify/request-rate-limiter library. We covered how to handle large datasets, process thousands or even millions of addresses, and avoid exceeding API rate limits.

By implementing rate-limiting strategies, you can prevent service disruptions, avoid errors like 429 Too Many Requests, and ensure smooth, uninterrupted geocoding. The Geoapify Geocoding API offers flexible plans to accommodate different scales of usage—from 3,000 addresses per day on the Free Plan to 250,000 or more on paid and custom plans.

Remember, staying within rate limits not only protects your access to the API but also ensures the reliability of the service for everyone. Whether you’re processing a small batch of addresses or large volumes, Geoapify’s rate limit policies and tools like the request rate limiter provide the flexibility and scalability you need.

As your project grows, you can seamlessly scale up your geocoding capacity with Geoapify by upgrading your plan, allowing you to meet your needs while maintaining a smooth user experience.