caching - Exception: Service invoked too many times for one day: urlfetch

Question

I created a script in Google Sheets, which is working well but after a while I'm getting the following error: Exception: Service invoked too many times for one day: urlfetch

I think I called the function like 200-300 times in the day, for what I checked it should be below the limit.

I read we can use cache to avoid this issue but not sure how to use it in my code.

function scrapercache(url) {
    var result = [];
    var description;
    var options = {
        'muteHttpExceptions': true,
        'followRedirects': false,
    };
  
var cache = CacheService.getScriptCache();
var properties = PropertiesService.getScriptProperties();

try {  
  let res = cache.get(url);

  if (!res) {
    // trim url to prevent (rare) errors
    url.toString().trim();
    var r = UrlFetchApp.fetch(url, options);
    var c = r.getResponseCode();

    // check for meta refresh if 200 ok
    if (c == 200) {
      var html = r.getContentText();
      cache.put(url, "cached", 21600);
      properties.setProperty(url, html);

      var $ = Cheerio.load(html); // make sure this lib is added to your project!

      // meta description
      if ($('meta[name=description]').attr("content")) {
        description = $('meta[name=description]').attr("content").trim();
      }
    }
  
    result.push([description]);    
  }
} 
catch (error) {
  result.push(error.toString());
} 
finally {
  return result;
}
 
}

how can I use cache like this to enhance my script please?

var cache = CacheService.getScriptCache();
  var result = cache.get(url);
  if(!result) {
    var response = UrlFetchApp.fetch(url);
    result = response.getContentText();
    cache.put(url, result, 21600);

Thanks in advance

Answer 1

You can implement CacheService and PropertiesService together and only retrieve the URL again after a specified amount of time.

Code Change:

Be aware that additional calls to retrieving the cache and properties will slow your function down, especially if you are doing this a few hundred times.

As the values of the cache can be a maximum of 100 KB, we will use CacheService to keep track of which URLs are to be retrieved, but PropertiesService to store the data.

You can edit your try block as so:

var cache = CacheService.getScriptCache();
var properties = PropertiesService.getScriptProperties();

try {  
  let res = cache.get(url);

  if (!res) {
    // trim url to prevent (rare) errors
    url.toString().trim();
    var r = UrlFetchApp.fetch(url, options);
    var c = r.getResponseCode();

    // check for meta refresh if 200 ok
    if (c == 200) {
      var html = r.getContentText();
      cache.put(url, "cached", 21600);
      properties.setProperty(url, html);

      var $ = Cheerio.load(html); // make sure this lib is added to your project!

      // meta description
      if ($('meta[name=description]').attr("content")) {
        description = $('meta[name=description]').attr("content").trim();
      }
    }
  
    result.push([description]);    
  }
} 
catch (error) {
  result.push(error.toString());
} 
finally {
  return result;
}

I hope this is helpful to you!

References:

• Class CacheService | Apps Script | Google Developers
• Class Cache | Apps Script | Google Developers
• Class PropertiesService | Apps Script | Google Developers

Related Questions:

• Service invoked too many times for one day: urlfetch