2 September 2026
Reading a huge file line by line in Node, without loading it
Reading a huge file line by line in Node
readFileSync loads the entire file into memory before you look at a
single line of it. That is fine on the 4KB fixture you tested with and
fatal on the 2GB export somebody runs it against six months later - and
Node's default heap will stop you well before the file size does.
import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';
export async function eachLine(path, onLine) {
const stream = createReadStream(path, { encoding: 'utf8' });
// crlfDelay treats \r\n as ONE break rather than two, which is the
// difference between working and silently producing empty lines on a
// file that came off a Windows machine.
const lines = createInterface({ input: stream, crlfDelay: Infinity });
let n = 0;
for await (const line of lines) {
n += 1;
await onLine(line, n);
}
return n;
}
// Usage: memory stays flat no matter how big the file is
const count = await eachLine('./huge.log', (line, n) => {
if (line.includes('ERROR')) console.log(n, line);
});
console.log(`${count} lines`);
Why for await and not an event handler
The for await...of loop applies backpressure for free. If onLine is
slow - writing to a database, calling an API - the loop waits before
pulling the next line, and the stream pauses behind it.
An on('line', ...) handler does not. The stream keeps emitting whether
or not your handler has finished, and the work queues up in memory - which
reintroduces the exact problem you switched to streaming to avoid, only
harder to see.
// Bad: the handler cannot slow the stream down
lines.on('line', async (line) => {
await slowThing(line); // hundreds of these now in flight at once
});
One caveat worth knowing: readline splits on newlines, so it is not
a CSV parser. A quoted field containing a newline is a valid CSV row and
this will cut it in half. For CSV, stream it through a real parser - the
streaming principle is the same, the line-splitting is not.