Raymii.org
Quis custodiet ipsos custodes?Home | About | All pages | Cluster Status | RSS Feed
ncdu - for troubleshooting diskspace and inode issues
Published: 29-10-2017 | Author: Remy van Elst | Text only version of this article
❗ This post is over six years old. It may no longer be up to date. Opinions may have changed.
Table of Contents
In my box of sysadmin tools there are multiple gems I use for troubleshooting
servers. Since I work at a cloud provider sometimes I have to fix servers that
are not mine. One of those tools is ncdu
. It's a very usefull tool when a
server has a full disk, both full of used space or full of used inodes. This
article covers ncdu and shows the process of finding the culprit when you're out
of disk space or inodes.
Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:
I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!
Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.
You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!
ncdu
On the homepage ncdu describes itself as:
Ncdu is a disk usage analyzer with an ncurses interface. It is designed to find space hogs on a remote server where you don't have an entire graphical setup available, but it is a useful tool even on regular desktop systems. Ncdu aims to be fast, simple and easy to use, and should be able to run in any minimal POSIX-like environment with ncurses installed.
When the root disk is full, a server will behave strangely. Some daemons still
work, others will work for a bit and some refuse to start at all. If you use
ext4
the server will boot up most of the time since there is a bit of reserved
space in the filesystem.
Space and inodes
A disk can be full because of too much space being used. You can check that with
the df -h
command:
root@s1:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 19G 0 100% /
A disk can also be full because there are to many files and folders. You've ran
out of inodes. df -i
shows you that:
root@s1:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/vda1 1305600 1305600 0 100% /
In this case one of my own servers has ran out of inodes. I know where they are, one of the users has no quota and has synced an entire 10 years of Maildir, which resulted in too many small files. If I wouldn't have known where the storage was being used, ncdu would be my go to tool for finding out.
Yes you can use du
and df
. du
takes a long time and it doesn't tell you
where the inodes are used. A shell loop with find for finding inodes has the
same problem. You also can't easily navigate the results.
ncdu gives a graphical overview of the files and folders, sorted by most space
used. In the lower right corner there is a counter for the number of files in
the current folder. Using that counter, you can navigate around the filesystem
and see where the millions of inodes are used. Sorting by inode usage can also
be done with the C
key. No other tool I know of has that option.
ncdu also allows exporting the results. You can then see them on another
machine. Usefull if you have a weird ARM nas with terminal display issues. ncdu
runs remotely without the TUI, scp
over the results file and use the local
terminal to find out what's wrong.
Installing ncdu
ncdu is available for almost every operating system imaginable and probably in your package manager:
apt-get install ncdu
yum install ncdu
If you have trouble with a full disk or your package manager doesn't work, the developer even has static binaries on the page:
Sometimes you can remove a few files to make space for this download, where the package manager doesn't work.
Using ncdu
Start up ncdu. If you don't give any parameters, it will scan the current folder. If you give a filesystem path, it will scan that:
ncdu /
Will start a scan of the entire system. When the scan is finished, use the arrow
keys and ENTER to navigate around. Files and folders can be deleted right away
using d
. You can change the display from the bar to percentage or both using
the g
key. Sorting based on filesize with s
, by name with n
and by inodes
with C
. That option is very usefull when your disk has run out of inodes.
ncdu is not real time. If you remove files outside of ncdu you need to run a rescan.
Tags: articles , bash , disk , monitoring , ncdu