javascript - Trying to make a massive series of API requests, but getting malformed JSON due to number of requests

Question

I'm trying to fetch some data in Node.JS from a service used to house employee data. After getting an access token, I have a function that the token is passed to along with an array of employee IDs (1500+). When I run the code with 2-3 hardcoded IDs it works perfectly fine. When I pass it the full array I start getting the following types of errors.

This one fires an untold number of times.

FetchError: invalid json response body at https://comp.site.com/service/odatav4/public/user/user-service/v1/dataset?$filter=criteria/targetUserID%20eq%20%27123%27 reason: Unexpected end of JSON input
    ...
  type: 'invalid-json'

Before ending with the following error.

D:data-fetch-testindex.js:96
        const values = ID.data.value
                               ^
TypeError: Cannot read property 'value' of undefined

I'll include an abbreviated example of the problem section below.

const fetch = require('node-fetch')

const getAllOrgData = async (accessToken, empIDs) => {
  const fetchOrgData = async emp => {
    const orgDataHeaderContent = [
      ['Content-Type', 'application/json'],
      ['Authorization', `Bearer ${accessToken}`]
    ]
    const orgDataHeaders = new fetch.Headers(orgDataHeaderContent)
    const orgDataOptions = {
      method: 'GET',
      headers: orgDataHeaders
    }
    const response = await fetch(
      `https://comp.site.com/service/odatav4/public/user/user-service/v1/dataset?$filter=criteria/targetUserID eq '${emp}'`,
      orgDataOptions
    )
    const empOrgData = await response.json().catch(error => {
      console.error(error)
    })
    return empOrgData
  }
  const allOrgData = async () => {
    const OrgData = await empIDs.map(async ID => ({
      id: ID,
      data: await fetchOrgData(ID)
    }))
    return await Promise.all(OrgData)
  }
  return await allOrgData()
}

;(async function () {
  const headcountPath = String.raw`\local
etworkpathHeadcount.csv`
  const empIDs = await getHeadcountData(headcountPath)
  const accessToken = await getAccessToken()
  const allOrgData = await getAllOrgData(accessToken, empIDs)
    .then(data =>
      data.map(ID => {
        const id = ID.id
        const values = ID.data.value
        return values.map(record => ({ ...record, AMEI: id }))
      })
    )
    .then(data => {
      const flatData = data.flat()
      generateCSV(flatData)
    })
})()

I'm totally at a loss of what to do next.

question from:https://stackoverflow.com/questions/65852046/trying-to-make-a-massive-series-of-api-requests-but-getting-malformed-json-due

Answer 1

Waitting for answers