2 September 2026

Reading a huge file line by line in Node, without loading it

Reading a huge file line by line in Node

readFileSync loads the entire file into memory before you look at a single line of it. That is fine on the 4KB fixture you tested with and fatal on the 2GB export somebody runs it against six months later - and Node's default heap will stop you well before the file size does.

import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

export async function eachLine(path, onLine) {
  const stream = createReadStream(path, { encoding: 'utf8' });

  // crlfDelay treats \r\n as ONE break rather than two, which is the
  // difference between working and silently producing empty lines on a
  // file that came off a Windows machine.
  const lines = createInterface({ input: stream, crlfDelay: Infinity });

  let n = 0;
  for await (const line of lines) {
    n += 1;
    await onLine(line, n);
  }
  return n;
}

// Usage: memory stays flat no matter how big the file is
const count = await eachLine('./huge.log', (line, n) => {
  if (line.includes('ERROR')) console.log(n, line);
});
console.log(`${count} lines`);

Why for await and not an event handler

The for await...of loop applies backpressure for free. If onLine is slow - writing to a database, calling an API - the loop waits before pulling the next line, and the stream pauses behind it.

An on('line', ...) handler does not. The stream keeps emitting whether or not your handler has finished, and the work queues up in memory - which reintroduces the exact problem you switched to streaming to avoid, only harder to see.

// Bad: the handler cannot slow the stream down
lines.on('line', async (line) => {
  await slowThing(line);     // hundreds of these now in flight at once
});

One caveat worth knowing: readline splits on newlines, so it is not a CSV parser. A quoted field containing a newline is a valid CSV row and this will cut it in half. For CSV, stream it through a real parser - the streaming principle is the same, the line-splitting is not.