javascript - How to upload and read large files line by line in ExpressJS?

Question

I am trying to upload a large file to Express and I am able to read the files object successfully using express-fileupload middleware, like below.

{
  myfile: {
    name: 'somelargefile.txt',
    data: <Buffer /* details truncated */ ... 110075790 more bytes>,
    size: 110075840,
    encoding: '7bit',
    tempFilePath: '',
    truncated: false,
    mimetype: 'text/plain',
    md5: '/* details truncated */',
    mv: [Function: mv]
  }
}

However, express might take up a lot of memory if I save the entire file into the files object. How can I read the file line by line, like using stream, from the request object?

Answer 1

For large files, readFileSync can be inconvenient, as it loads the whole file in memory. A different synchronous approach is to iteratively call readSync, reading small bits of data at a time, and processing the lines as they come.

Eg:

var fs = require('fs');
var filename = 'test.txt'

var fd = fs.openSync(filename, 'r');
var bufferSize = 1024;
var buffer = new Buffer(bufferSize);

var leftOver = '';
var read, line, idxStart, idx;
while ((read = fs.readSync(fd, buffer, 0, bufferSize, null)) !== 0) {
  leftOver += buffer.toString('utf8', 0, read);
  idxStart = 0
  while ((idx = leftOver.indexOf("
", idxStart)) !== -1) {
    line = leftOver.substring(idxStart, idx);
    console.log("one line read: " + line);
    idxStart = idx + 1;
  }
  leftOver = leftOver.substring(idxStart);
}