A buffer is a holding area for data in transit between two places that work at different speeds. That description sounds too general to be useful, but the speed mismatch it papers over is often enormous — memory and disk differ by several orders of magnitude — and closing it is one of the cheapest performance wins available.

Why it helps

Every unbuffered write is a system call, and a system call is expensive relative to a memory copy: the CPU switches into the kernel, does the work, and switches back. Doing that once per line means paying the transition cost once per line.

A buffer collects writes in memory and hands them over in batches. The bytes written are identical; the number of crossings into the kernel is not, and that is the entire saving.

In Go

The bufio package wraps an existing reader or writer with a fixed-size byte slice.

For a writer, data accumulates in the slice until it is full, at which point the whole slice is written to the underlying destination in one operation and the buffer resets. For a reader, the opposite: a large chunk is read from the source into the slice, and reads are served from memory until it is empty and needs refilling.

The size of that slice sets the batch size, which sets how many system calls you end up making.

Measuring it

Writing the numbers 0 to 99,999 to a file, once directly and once through a bufio.Writer:

buffered.go
package main
 
import (
	"bufio"
	"fmt"
	"os"
	"time"
)
 
func main() {
	file, err := os.Create("test.txt")
	if err != nil {
		panic(err)
	}
	defer file.Close()
 
	start := time.Now()
	for i := 0; i < 100000; i++ {
		fmt.Fprintln(file, i)
	}
	fmt.Printf("Without buffer: %s\n", time.Since(start))
 
	start = time.Now()
	w := bufio.NewWriter(file)
	for i := 0; i < 100000; i++ {
		fmt.Fprintln(w, i)
	}
	w.Flush()
	fmt.Printf("With buffer:    %s\n", time.Since(start))
}
Without buffer: 9.357391ms
With buffer:    3.488088ms

Roughly 2.7× faster for the same 100,000 lines. The work done is the same; the number of trips into the kernel is not.

Treat the exact numbers as indicative rather than precise — they depend on the filesystem, the page cache, and what else the machine is doing. The ratio is the durable part, and it grows as the writes get smaller and more frequent.

The mistake everyone makes once

WARNING

A buffered writer holds data that has not been written yet. If you do not call Flush, the tail of your output is silently discarded when the program exits.

The version of this that actually bites is combining it with defer:

file, _ := os.Create("out.txt")
defer file.Close()
 
w := bufio.NewWriter(file)
defer w.Flush()

Deferred calls run last-in-first-out, so this happens to be correct — w.Flush() runs before file.Close(). Swap the two lines and the file closes first, the flush fails, and you lose data with no error unless you were checking the return value of Flush, which almost nobody does.

There is no crash and no warning. The file is just short.

Where else this applies

The pattern generalises well past file I/O, because the underlying shape — a per-operation fixed cost that batching amortises — turns up constantly:

  • Network writes, where the fixed cost is a packet rather than a syscall.
  • Database inserts, where one statement with a thousand rows beats a thousand statements.
  • Log shipping, where batching is the difference between a viable pipeline and a self-inflicted denial of service.

The trade is latency against throughput. Buffered data is data that has not arrived yet, so a large buffer on a low-traffic connection means messages sit around waiting for company. For a log file that is free. For an interactive protocol it is a bug, which is why terminals line-buffer instead.

Getting this right is mostly a matter of asking what the fixed cost per operation is, and whether anything is waiting on the result.