Resize /var partition on a remote system (Linux Debian Lenny)

At a server that I only have access via ssh and vnc to (which means no booting from a "live" CD/DVD is possible), I would like to shrink the partition assigned to /var (in order to free some space for an additional partition). What would a good and safe way to do it?

Solution (based on David Spillet answer, provided that you have enough space on / to accommodate a copy of /var):

1. stop anything that is using /var where possible
2. remount /var readonly for good measure mount -f -oremount,ro /var (didn't seem to have any effect for me)
3. copy /var to / under a different name with mkdir /vartmp; cp -av /var/* /vartmp/
4. comment out the entry for /var in /etc/fstab
5. reboot
6. move things around with mv /var /varmount; mv /vartmp /var
7. reboot, partition
8. uncomment the entry for /var in /etc/fstap
9. reboot

From serverfault Sergei Kozlov

• 1. stop any services that may be using var (lsof | grep "/var")
  2. umount /var
  3. shrink /var
  4. re-mount /var
  5. restart the services you've stopped at 1.
  Sergei Kozlov : Stuck at step 2.: lsof | grep "/var" shows no services anymore (all killed), yet trying to unmount /var I get this: "umount: /var: device is busy"

  Zoredache : You forgot step 0. Make sure your backups are up-to-date and functional.

  Geoff Fritz : Try "umount -f" -- just make sure your current working directory isn't somewhere under /var or your shell will get killed (never hurts to have a backup login shell standing by, especially on remote machines).

  Sergei Kozlov : @Geoff, here's what I get: ~ sudo umount -f /var umount2: Device or resource busy umount: /var: device is busy umount2: Device or resource busy umount: /var: device is busy It didn't kick me out, but didn't do the unmount either. Earlier I tried "fuser -km /var" - that did kick me out (although I wasn't anywhere under /var), so, obviously, my ssh session depends on /var, too.
  From Jure1873

• If you have room on your root partition for the contents of /var you could:

  1. stop anything that is using /var where possible
  2. remount /var readonly for good measure mount -f -oremount,ro /var
  3. copy /var to / under a different name with mkdir /vartmp; cp -av /var/* /vartmp/
  4. move things around with mv /var /varmount; mv /vartmp /var
  5. comment out the entry for /var in /etc/fstab
  6. reboot

  You should now have /var in place as-was on your root partition instead of its own. You can now resize the old partition as needed. Step 4 should let you rename the /var directory even though it is in use as a mount-point and otherwise busy (and proceses with open files there will track the change as the open files are not referred to by path+name, but instead by inode, once open).

  If you don't have space on / but do on /someotherfs then you could try move it there with a symlink in / like so:

  1. stop anything that is using /var where possible
  2. remount /var readonly for good measure mount -f -oremount,ro /var
  3. copy /var with cp -av /var /someotherfs/var
  4. move things around with mv /var /varmount; ln -s /someotherfs/var /var
  5. comment out the entry for /var in /etc/fstab
  6. reboot

  This is slightly more risky though as you need to be sure that /someotherfs gets mounted before /var when booting.

  If you don't have room where you want to copy it to temporarily you might be able to reduce the size of /var by removing things like cached packages (aptitude clean on Debian style setups, there is no doubt an equivalent yum command), moving other stuff away (for instance, Debian keeps the default httpdocs in /var/www so if it is still there and you have lots of data in there, move it to another partition), and deleting files from /var/log that are not very recent (backing the up first, in case you do need to refer to them later).

  As Zoredache says: what ever you do, make sure you are happy with your backup arrangement before proceeding.

  Caveat: all the above is from memory, I've not tested it anywhere, follow at own risk!

  Sergei Kozlov : Thanks for your effort explaining! Followed your first scenario with a little modification: couldn't rename /var to /varmount on step 4 ("device busy" - probably because /var is mounted straight to a partition), so first did 5), rebooted, and then "moved things around" (and rebooted again).

  David Spillett : I was sure it would let you move it (I've deleted a whole directory structure while processes had files open within it before now) - you (re)learn something every day!
  From David Spillett

• The traditional way is to boot in single-user mode. That way /var won't even be mounted by default so you can mess around with it all you like.

  Sergei Kozlov : What would be the command to reboot in single-user mode?

  reinierpost : I'd have to Google for the details, it's been a while. It's not hard to find.
  From reinierpost