"less Is Enough" — Until It Isn't: Four Limits of Command-Line Craft

You press G. Ninety seconds pass.

less was supposed to be the light tool. So what was being read during those ninety seconds?

The answer isn't about the quality of the tool. It's about one thing: where the line breaks are.

Every part of this series has been about where general-purpose tools stop. This one goes at the four that are trusted most — less, Vim, pipe craft, and grep. None of them is broken. They're all excellent. And past a certain size, all four hit the same wall. The goal here is to draw that wall as a line you can see.

Up front: UwView scrolls to the end of a huge file from the moment it opens; the index builds in the background and line numbers appear when it finishes. Pro saves the index and the compression, so a 47.73 GB log you've opened once reopens instantly, with line numbers, from the second open onward (0.02–0.07 s measured on one setup; results vary — details at the end)

Except where noted, every timing and transfer rate below is environment-dependent. The numbers are worked examples and measurements meant to show the shape of the problem, not a promise about your machine.

1. less was enough — until line jumps and Japanese search

Situation

less huge.log opens instantly. So far, so good.

It falls apart on the next keystroke. G to jump to the end: it stalls. -N for line numbers: it stalls harder. 12000000g to reach a specific line: stalls again. And in a Japanese log, typing /エラー returns nothing at all.

"less is enough" was a sound judgement. On a few megabytes, nothing is lighter.

Why it happens

less is fast because it doesn't read, and slow the moment you ask something it can't answer without reading.

less treats a file as a sequence of lines, but at open time it knows exactly one thing: the first screenful. Where the file's lines begin and end is determined by newline bytes, and you cannot know where those are without reading them. There is no way to answer "at which byte does line 12,000,000 start?" by seeking. So:

• G (go to end) = read every byte to the end, counting newlines
• -N (show line numbers) = count newlines up to the current position
• 12000000g (jump to a line) = find the 12,000,000th newline from the top

All three are the same operation: read everything, count newlines. On a 50 GB file, that's 50 GB read. The ninety seconds is what that costs. less isn't being inefficient.

The Japanese wall is a different problem. less does not transcode. It only decides whether bytes are displayable (LESSCHARSET configures that decision, not a conversion). If your terminal is UTF-8 and the file is CP932, the UTF-8 bytes you typed for /エラー simply do not appear in the file's CP932 bytes. The string isn't absent; it exists as a different byte sequence — the topic of Part 3 and Part 11, showing up here as a pager keystroke.

What general-purpose tools do, and where they stop

There are standard moves.

less -n huge.log            # stop computing line numbers → G gets light (no line numbers)
LESSCHARSET=utf-8 less huge.log
iconv -f CP932 -t UTF-8 huge.log | less    # convert, then pipe in
tail -c 200M huge.log | less               # just look at the tail

The first one works. It is also a trade: -n buys speed by giving up line numbers, so you cannot have both. On any job where the incident report needs "line 12,034,551," you end up back on -N.

The third is the one that quietly costs the most. The moment you write iconv | less, what less receives is standard input, which cannot be seeked. You can still scroll back — because less is writing what flowed past into a temp file. A second 50 GB copy grows in /tmp. And to move forward you still have to push every byte through the converter, so G becomes "read everything, converting as you go." The mapping to the original line numbers survives (converting doesn't change the line count), but displaying those numbers costs exactly what it cost before.

tail -c is a genuine escape hatch, but the moment you slice, it stops being the original: the first line may begin mid-character and render as garbage, and the line numbering restarts at 1.

Three limits. One, "see the end," "show line numbers," and "jump to a line" all demand a full read. Two, the result of that full read is discarded the instant you quit — open it again tomorrow and it's another ninety seconds. Three, there is no way to resolve an encoding mismatch without inserting a conversion, and inserting one costs you both seeking and disk.

2. :e huge.log freezes — what Vim can configure, and what it can't

Situation

You type vim huge.log, hit Enter, and nothing comes back. For a stretch, Ctrl-C doesn't even land. Eventually there's a swap-file warning, or a message about memory.

The reason people reach for Vim over less is usually: search, read the surrounding lines in place, and mark something. The goal is investigation, not editing.

Why it happens

A Vim buffer is designed to hold the entire file as an array of lines.

That isn't a flaw; it's what an editor has to be. Inserting, deleting, and replacing any line requires holding the sequence as a structure. So :e reads the whole file, and while reading, builds several other things:

• A swapfile (.swp) — another copy on disk, for crash recovery
• Undo history — and another copy on disk if undofile is on
• Syntax highlighting state — deciding whether a line is inside a comment means scanning back through earlier lines; this is what makes very long lines and very large files collapse
• Fold computation — foldmethod=syntax walks everything

So :e huge.log starts reading 50 GB, writing something like 50 GB back to disk, and running a parser over every line, all at once. Ninety seconds isn't in the running.

What general-purpose tools do, and where they stop

Vim can be told to do considerably less.

vim -u NONE -N huge.log     # no vimrc, no plugins

:syntax off
:set noswapfile
:set noundofile
:set nofoldenable
:set lazyredraw
:set synmaxcol=200          " give up highlighting past column 200
:set viminfo=

The LargeFile-style plugins essentially apply exactly this automatically. The effect is real and clearly noticeable.

One thing does not shrink: the number of bytes read.

Every setting above makes the work after reading cheaper; none of them touches the premise that :e reads all of it. Vim has no partial-load model — no way to keep most of the file on disk and page in only the region you're looking at. Which means opening a 48 GB file in Vim on a 16 GB machine is not a configuration problem. It's a structural one.

So in practice, people slice.

sed -n '12000000,12000500p' huge.log > slice.txt && vim slice.txt
awk 'NR>=12000000 && NR<=12000500 {print NR": "$0}' huge.log > slice.txt   # keeps line numbers

Three limits. One, you must choose the range before you can look at it. You find out 500 lines wasn't enough only after opening, and each retry sends sed back to read from the top of the file. Two, a plain slice loses the original line numbers — the second form keeps them via NR, but now the numbers are text inside the line, so you can't paste it straight into a report. Three, slice.txt files accumulate. By the end of an investigation there are twenty unrelated fragments in /tmp, and you can no longer say which range each one was.

None of this is an argument against Vim. If the job is editing, Vim is the answer. The problem is opening Vim when you only wanted to look, and paying the full cost of an editor for it.

3. Pipe craft vs. a viewer — where the line actually is

Situation

One line of tail, head, sed, awk, sort, and uniq handles most things. It genuinely has, for years.

It snags in one specific place: when you don't yet know what to look for. You have the error message. You don't have the condition that produces it. So you can't decide what to hand grep.

You head -100 and stare. You cut a column with awk. You look at a distribution with sort | uniq -c. Then back again. Ten round trips. Twenty.

Why it happens

A pipe demands that you decide the question first.

To write awk 'NR>=a && NR<=b' you must already know a and b. To write grep 'X' you must already know X. But the first half of an investigation is precisely the part where a, b, and X are decided by looking. You're asked for a question at the stage where you don't have one yet.

The structure of a pipeline has a few more consequences:

• Intermediate state doesn't survive. After grep A | grep B | grep C, how many lines A matched and how many B matched is gone. You see the final count without knowing which condition did the work.
• Line numbers die after stage one. grep -n counts within the block that reached it, not in the original file.
• Every retry starts from the beginning. Change one character and the pipeline re-reads the file from the top.

The flip side is equally clear. A pipeline is unbeatable at repeating the same question. A one-liner goes into cron, into CI, and over ssh unchanged. When the log lives on a remote server you can't pull down, a pipeline is the only option (a constraint that connects directly to the "where do you keep it" question from Part 4).

What general-purpose tools do, and where they stop

The line, in one sentence:

If the shape of the answer is fixed, use a pipe. If you're still looking for the shape of the answer, use a viewer.

Concretely:

In practice you use both, usually as "narrow with a pipe, then read in a viewer." There's a trap in that flow, though: narrowing already cost you one full read, and it's easy not to notice. And if you drop the narrowed result into another file, Part 9 applies — that file is no longer the original.

Two limits. One, full reads pile up in proportion to the number of exploratory round trips. Ninety seconds each, twenty trips, half an hour. Two, the history of those trips dies with the terminal. Tomorrow morning starts with remembering which conditions you already ruled out.

To be honest about it: this is not a case for replacing pipes with a viewer. Work that needs no interaction belongs in a pipe. What gets replaced is only the interactive first half.

4. grep isn't slow — you're I/O-bound

Situation

grep felt slow, so you switched to rg (ripgrep). Almost no difference.

Or the reverse: you ran the same command twice and the second run was suspiciously fast. Convinced you'd made a mistake, you measured again.

Why it happens

The bottleneck isn't the CPU. It's the disk.

Fixed-string search is very cheap on a modern CPU — Boyer-Moore-family algorithms with SIMD, running close to memory bandwidth. Which means the CPU is sitting there waiting for the disk to deliver bytes.

The floor is a division problem:

This table is calculated from typical sequential-read figures, not measured. Real numbers move with the filesystem, fragmentation, encryption, competing processes, and whether the file is on the far side of NFS/SMB.

Searching 50 GB on an HDD cannot beat four minutes, no matter how clever the search program is. grep and rg performing the same is exactly what you'd expect. The difference shows up only where the work is CPU-bound: complex regular expressions, large numbers of small files (rg parallelises across them), matching that involves Unicode normalisation.

The fast second run has the same explanation. Whatever the first run pulled in landed in the kernel's page cache, so the second read came from RAM, not disk — dramatically faster, up to the amount RAM can hold. Which is why a benchmark number means nothing unless it says whether the cache was warm. That's the reason every measurement in this series carries its conditions.

What general-purpose tools do, and where they stop

First, find out which side is the bottleneck.

time grep -c 'ERROR' huge.log
# real 4m12s / user 0m21s / sys 0m38s  → the CPU is idle = I/O-bound

# measure the sequential ceiling on this machine (bypassing the cache)
dd if=huge.log of=/dev/null bs=1M count=20000 iflag=direct

# watch I/O wait while it runs
iostat -x 1        # %util pinned at 100 means saturated
vmstat 1           # a large wa column says the same

If user + sys is tiny next to real, the CPU is waiting. Swapping the search program at that point is buying a faster car to sit in the same traffic jam.

Once you know you're I/O-bound, there are exactly three things you can do:

1. Put it on faster media (HDD → SSD — reliable, but you often don't get to choose where the file lives)
2. Read in parallel (works across many files; on one huge file it plateaus depending on the medium)
3. Read fewer bytes

The third is the real one, and it splits in two. An index — record where things are, so next time you don't read everything. And compression — if I/O is the bottleneck, reading a smaller compressed byte stream and expanding it on the CPU can be faster than reading the raw file, because the CPU is the resource you have spare. That arithmetic is why "search it while it stays compressed" in Part 8 makes sense.

One limit, and it's a heavy one. Neither grep nor rg carries anything forward. The structure of the 50 GB you just spent four minutes reading vanishes when the command exits. Change the condition and it's four minutes again. The page cache helps only as far as RAM reaches, and 50 GB doesn't fit.

What all four had in common

Four tools, four different design philosophies, four different failure modes. The rightmost column is the same shape in all four rows because all four rest on the same premise: read the whole file, every time.

On a small file that premise is free. A 0.1-second full read is indistinguishable from no read at all, so the design is correct. It only breaks when that 0.1 second becomes ninety. Nothing about the operation got harder. It just stopped being worth it. This series keeps arriving at the same sentence, but this time the tools themselves are right — which makes the cause harder to see. You assume you're holding it wrong and go looking for a setting.

Three things are needed:

• Don't throw away the line boundaries you already computed. The ninety seconds in less, the :e, the pipeline round trips — all of them are counting the same newlines over again. Build it once, save it, and the second time there's nothing to count.
• Read fewer bytes. Under an I/O bottleneck, this is the only real speedup. Searching without decompressing means less to read.
• Keep the original's coordinates while you narrow. A slice, an iconv, a sed -n — each one stops being the original the moment it passes through. On any job that ends in "line number, please," that's what bites last.

Back to the ninety seconds. That wasn't less being slow. It was newlines being counted — and the count was thrown away the moment you pressed q.

The tool I use

UwView (free), which I develop, displays, scrolls, and searches huge text from the moment it opens. It doesn't load the file into memory, so files larger than RAM open fine. The index builds in the background and line numbers appear when it completes (most viewers show only the head until indexing finishes). Nothing is split and nothing is extracted, so the original stays one file, unmodified. Section 1's G — moving to the end — works from the first second.

• Switch encodings without reopening: UTF-8 / Shift-JIS (CP932) / EUC-JP / UTF-16 are detected automatically, and when detection is wrong you switch in place without rebuilding the index. Section 1's "inserting iconv costs you seeking and disk" doesn't arise.
• No editing engine: the swapfile, undo history, and syntax parsing that make Vim's :e expensive simply don't exist here. Section 2's list of settings is unnecessary. (For work that genuinely needs editing, Edit Upgrade is a separate licence.)
• Context width is changed afterwards: you don't have to decide how many surrounding lines you need before opening. Section 2's "re-run sed after discovering 500 lines wasn't enough" disappears. (Free gives ±1 line; adjustable ±N is Pro.)
• Search results open in a separate popup you can jump from into the original (write-up) — the replacement for section 3's return trip to the terminal.

And UwView Pro is where those three requirements live.

• It saves the index and the compression: a file you've opened once reopens instantly, with line numbers, from the second open onward (0.02–0.07 s measured on a 47.73 GB text file; one setup, results vary). Section 1's "ninety seconds discarded on q" and section 4's "the scan isn't carried forward" are the same problem, solved by the same feature.
• Search through the compressed cache: exactly section 4's arithmetic — under an I/O bottleneck, reading fewer bytes is the real speedup. Storage at roughly 1/9 while still searchable helps both the disk and the wait (keeping logs compressed and still searchable).
• Drill-down search: narrow a result by another term, then another. Tabs carry term (count), which is section 3's "per-stage counts don't survive." Stages after the first search only the previous window, so there's no wait, and backing out is one click on an earlier tab. The original line numbers are preserved all the way down (write-up).
• Resume tomorrow: tabs and conditions persist, so section 3's "close the terminal and the history goes with it" doesn't happen (archive × session restore workflow).

One row from section 3's table has crossed the line: frequency tallying. As of v1.5.0, drill-down search includes Tally — the equivalent of grep ... | grep -oE '...' | sort | uniq -c | sort -rn. Apply a regular expression to the hits, take the first capture group as the key, and get a ranked frequency table on top of whatever stage you've narrowed to, with the hit count, the number of distinct values, the elapsed time, and CSV export (uniq -c, in a GUI). It's one step past section 3's complaint that per-stage counts don't survive.

It stops at a frequency ranking, though — no sums, no averages, no charts. That's an analysis tool's job.

With that said, honestly: UwView is not a replacement for a shell. It won't cut arbitrary columns and compute on them, won't join files, and won't run on a schedule in cron or CI. When the work has to stay inside a remote server, a pipeline is the only answer. Section 3's line moved by exactly one row, and otherwise stands: this tool is mainly the right-hand column — exploring, context, narrowing, citing. Routine transformation and automation still belong to awk.

One more honest note. A complete first index takes real time. Section 4's I/O bottleneck applies to UwView the same as to anything else; reading 50 GB once doesn't become free. The difference is whether that result is saved and reused. If you only ever open the file once, less is enough — that judgement is still correct today. It stops being correct on the day you start reopening the same file.

And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly).

Links

• Part 1: four go-to tools that sink under huge files: https://uvp.y42u.net/en/blog/uwview-ps01-huge-file-tool-limits-en/
• Part 3: four encoding traps and how to isolate them: https://uvp.y42u.net/en/blog/uwview-ps03-japanese-encoding-traps-en/
• Part 4: deciding between delete, keep, and compress: https://uvp.y42u.net/en/blog/uwview-ps04-log-retention-decision-en/
• Part 8: keeping logs compressed and still searchable: https://uvp.y42u.net/en/blog/uwview-ps08-compressed-archive-search-en/
• Part 9: reading structured data raw: https://uvp.y42u.net/en/blog/uwview-ps09-read-raw-structured-data-en/
• Part 10: four things to set up for the you at 2 a.m.: https://uvp.y42u.net/en/blog/uwview-ps10-oncall-night-preparation-en/
• Part 11: reading legacy encodings in 2026: https://uvp.y42u.net/en/blog/uwview-ps11-legacy-encoding-euc-utf16-en/
• Part 14: the traces are in the raw log: https://uvp.y42u.net/en/blog/uwview-ps14-attack-traces-raw-logs-en/
• 3GB/10GB/48GB × HDD/SSD/internal SSD — a nine-cell benchmark: https://uvp.y42u.net/en/blog/uwview-pro-benchmark-3sizes-en/
• An honest comparison with klogg (Ver1.1.1, complete edition): https://uvp.y42u.net/en/blog/uwview-klogg-feature-comparison-v111-en/
• Drill-down search — narrowing a result by another term: https://uvp.y42u.net/en/blog/uvp-drilldown-search-en/
• uniq -c joins the GUI (Tally in drill-down search): https://uvp.y42u.net/en/blog/uvp-tally-uniq-c-drilldown-en/
• Why search results open in a separate window: https://uvp.y42u.net/en/blog/uwview-filter-popup-jump-save-context-en/
• Archive × session restore workflow: https://uvp.y42u.net/en/blog/uwview-archive-session-restore-workflow-en/
• Source code (GitHub): https://github.com/amru195704/UwView

From the developer: a full list of my apps, Kindle books, and open-source work is on GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. The behaviour of less, Vim, grep, and awk varies by implementation (GNU/BSD/busybox), version, build options, and locale settings — always confirm option names and effects against your own man pages. Except where explicitly marked as measured, the transfer rates and durations here are calculated from typical performance figures, not observed. Figures marked as measured come from one specific setup and are not a guarantee of the same result. Disk type, filesystem, fragmentation, encryption, page-cache state, and competing processes all move the numbers substantially. If you spot an error, a comment is welcome and I'll check and correct it.