[ Date Index ] [ Thread Index ] [ <= Previous by date / thread ] [ Next by date / thread => ]
Been too long out of hands on system administration. When my desktop box starts backing stuff up it occasionally glitches the interactive response (I thought it was memory before because I was usually killing the memory), reading from one disk, writing to another. Probably the proper response is buy another SSD (because it is 2020 and 1TB SSD are under £100) and shuffle the bulk of the home file system onto it and make sure the IO is hellishly faster and the bottleneck will likely be moved elsewhere (probably CPU). I may leave that till I buy a new PC. However is there anything to do in software to make interactive response better under this sort of high IO load, because the backup is important but not urgent, where as me watching YouTube, or playing chess online, is urgent but not important. # cat /sys/block/sda/queue/scheduler [mq-deadline] none (same on sdb) # cat /proc/version Linux version 4.19.0-10-amd64 (debian-kernel@xxxxxxxxxxxxxxxx) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.132-1 (2020-07-24) # dpkg -l | grep linux ....snip... 4.19.0-11 ...snip.. I read those as Debian kernel maintainer is opinionated as to my scheduler options, and I'm guessing they know more than me on this topic. I'm due a reboot because I'm a point released behind the Debian kernel. And purging a load of stale kernels I'll never want again. (Surely someone automated the apt command to remove old enough kernels by now in Debian?). Memory 8GB, ~6GB is available for buffer cache when I'm not running virtual servers, or similar. ~4GB is I disable the swap partition. The backup is "tar cfz /usbmountpoint/SOMEDATEDFILE.tar.gz /home" which is probably not optimal. It is quite big compared to RAM (~200GB compressed each backup). SuSE documentation is (a) readable (b) says the mq-deadline schedule has the same tunable parameters as the deadline scheduler. https://documentation.suse.com/sles/12-SP5/html/SLES-all/cha-tuning-io.html#sec-tuning-io-schedulers-deadline Looking in /sys/block/sda/queue/iosched/ I see my values are the same as the SuSE defaults except write_starves is "2" not "3" (hmm). I have "ionice" installed, I don't understand the manual page. I think it is saying "wrong scheduler" so probably not the tool of choice. "top" suggests that the backup is sometimes taking a fair bit of CPU, but CPU scheduling looks plausibly fair. "iotop" suggests tar is reading, and gzip is writing, and there is hardly any data written to "sdb". Trying to interpret "iostat" output is interesting, but it suggests that "sda" is the disk being backed up to (once various redirections are sorted) and "sdb" is the source disk, and it is doing a lot of reading from sdb (well it would) and a lot of writing to "sdb", then sporadically dumping the next chunk to an almost idle USB disk on "sda". Both volumes are encrypted of course. Most of the time it rolls along 10 to 20MB/s read by tar, 10 to 20MB/s written by gzip, and 1 core is 100% running gzip, and responsiveness is okay, but every so often the read by tar jumps to 50MB/s, I'm assuming this is possibly large files or something fancy like sparse files? Doesn't seem to correspond with the problem mind. Digging further, the writes to "sdb" were largely swapping out things like the content of idle browser tabs (even though not memory stressed), and then the swapping data back in is contending with the backup reads from the same disk. If I disable swap, it helps a bit with things like recovering browser tabs, but because other things are not in RAM (code?) it is still stalling to display things waiting on IO which makes everything slow and unpleasant to use. Tempted to "rsync" all the data to the backup disk first, and/or choose a method which is more aware of what hasn't changed to either reduce the work or move it off the main disk. I'd prefer to abandon compression but I don't get than many backups to the terabyte as it is. -- The Mailing List for the Devon & Cornwall LUG https://mailman.dcglug.org.uk/listinfo/list FAQ: http://www.dcglug.org.uk/listfaq