Every Linux server ships with a swap partition or swap file, and almost nobody knows what theirs is doing until the night it matters. The common advice splits into two camps: "disable swap entirely, it hides memory problems" and "keep it as a safety net". Both camps are describing the same mechanism. What swap does, what it costs when the kernel reaches for it, and how to read the numbers before you buy more RAM.
The honest version first: swap on a server is not a second memory tier. It is an emergency valve, and a well-run server touches it almost never. But the valve that never gets used is still the difference between a slow minute and a lost day, and the settings around it decide which one you get.
Physical RAM splits into two broad uses: the memory your programs hold, and the page cache, the kernel's copy of recently read files. The kernel keeps that cache because a second read of a file from RAM is a thousand times faster than from disk.
When physical RAM gets tight, the kernel has two ways to free a page. It can drop a clean cache page, one that matches what is on disk already, instantly and for free. Or it can evict a page somebody is using, and the only place to put it is swap. Swap lives on your NVMe or SATA drive, so an evicted page costs a write now and a read every time the program touches it again.
This is why free -m shows a column for swap that nobody reads until the "available" figure in the memory row drops. Available is the number that matters, not "used", because used includes the cache the kernel would happily drop. A server with 2 GB "used" and 20 GB available is healthy. A server with 4 GB used, 500 MB available and swap filling is not, whatever the total says.
vmstat 1 prints two columns that describe the whole situation: si, swap pages read back in per second, and so, pages written out per second.
The mistake to avoid: panicking at "swap used: 6 GB" on its own. The question is whether anything is coming back in. If si has been 0 for a week, that 6 GB is cold pages sitting on disk doing a useful job. If si is sustained during business hours, the box is thrashing and the fix is memory or workload, not tuning.
When active working set exceeds RAM, the kernel evicts pages that are immediately wanted back. Every process run queue entry then blocks on disk. Load average climbs past your core count with CPU utilisation near zero, because the CPU is not doing work, it is waiting on pages. The server is up, answers pings, and takes 30 seconds to answer an SSH login. That is thrashing, and no amount of monitoring elegance beats having enough RAM.
The other failure mode is worse because it is silent. Without swap, the same memory pressure triggers the OOM killer, which picks the process with the biggest score, frequently your database, and kills it stone dead. A box with zero swap does not slow down under pressure. It loses processes. On a database server the recovery is a crash replay, not a slow afternoon.
On a dedicated server with 64 GB or more of RAM, the practical answer is a small swap, 4 to 8 GB, not the old "2x RAM" rule. The point is not capacity, it is headroom: enough room for the kernel to park cold pages so cache has room, and enough buffer that a memory growth spike gives you minutes to react instead of an OOM kill. Swap on NVMe costs so little at these sizes that disabling it buys nothing.
Size it from data, not habit. Watch available and si over two weeks of real traffic. If swap used sits under a couple of GB and si is flat, the current size is fine. If so climbs steadily, you are using swap as memory, and the fix is RAM or a memory leak, not a bigger swap.
vm.swappiness (default 60) biases the kernel between evicting cache pages and evicting program pages. Low values, say 10, say "prefer to drop cache, only touch anonymous pages as a last resort", which suits a database box with a big page cache. High values let the kernel swap cold daemons out more eagerly, which suits a machine running many long-idle services.
Set it with sysctl vm.swappiness=10, persist in /etc/sysctl.d/, and measure before and after. What you should not do is follow the internet folklore of vm.swappiness=0. On older kernels that disabled the OOM protection behaviour people assumed it kept, and even today a value of 0 on some distributions means "only swap to avoid OOM", which is exactly the cliff you wanted a gradient in front of. 10 gives you the intent without the surprise.
Memory is one of three resources that decide whether a workload is comfortable, and it interacts with the other two. If the working set genuinely exceeds RAM, a faster disk does help, since every swapped page pays disk latency, and the difference between NVMe and SATA for that penalty is what we broke down in our piece on IOPS versus throughput for database workloads. But it is help, not a fix.
The other half is noticing. A box that thrashes for twenty minutes at 2am hurts nobody until the nightly import blows its window, which is the failure we described in our breakdown of what CPU steal does to a server: the resource shortfall shows up in the tail first, and the tail is where the money is. Watch si, watch available, alert on sustained si rather than on swap used, and you will know about the shortfall a week before it costs you anything.
If you are sizing a London box against a real workload rather than a tier sheet, mail admin@servercabin.com with what you run and we will spec the memory from your numbers, swap included, before you commit to the machine.