Yash Verma / Cloud runbook

Disk full on a Linux host

A filesystem is at or near 100% and writes are starting to fail. All playbooks.

  1. Find which filesystem is full.

    Check disk space

    Shows filesystem disk space usage and types.

    df -Th

    disk, storage

  2. Find the biggest files on it.

    Find files over 100 MB

    Finds files over 100 MB on the root filesystem, largest first.

    sudo find / -xdev -type f -size +100M -exec du -h {} + 2>/dev/null | sort -rh | head -20

    disk, files, find

  3. From the full mount point, walk down the largest directories.

    Largest items in the current directory

    Lists the 10 largest files and directories here (dotfiles excluded).

    du -hs * | sort -rh | head -10

    disk, storage, du

  4. If df says full but du doesn't add up, a process is holding deleted files open. Restart that process to free the space.

    Find deleted files still held open

    Lists files that were deleted but are still open. Their space is not freed until the process closes them, so a disk can be full while du looks fine.

    sudo lsof +L1

    disk, lsof, space

  5. Trim the systemd journal to 500 MB.

    Caution. Deletes the oldest journal entries for good.

    sudo journalctl --vacuum-size=500M
  6. On Docker hosts, see how much space images, containers, volumes and build cache use.

    docker system df
  7. Remove dangling images.

    Remove dangling images

    Removes dangling images: untagged <none> layers left behind by rebuilds.

    docker image prune -f

    cleanup, images

  8. Delete old rotated logs, after checking the list.

    Delete log files older than 30 days

    Finds .log files older than 30 days under /var/log and deletes them.

    Caution. Deletes files permanently. Run it once without -delete to see the list first.

    sudo find /var/log -type f -name "*.log" -mtime +30 -delete

    find, cleanup, logs