在Linux上备份大量文件

wi

1
echo "$(date) --- START" >> backup.log ; rsync -avWP --no-compress --delete-after /run/media/wi/wdssd500 /Volumes/silo/_backups/ ; echo "$(date) --- END" >> backup.log
  • 常用脚本 backup-here.sh
1
2
3
4
5
6
7
8
9
10
11
12
#!/bin/bash

BACKUP_LOG="./backup.log"
RSYNC_LOG="./rsync.log"
DESTI_DIR="."

echo "--- START BACKUP $1 here  --- $(date '+%Y-%m-%d %H:%M:%S')" >> ${BACKUP_LOG}
rsync -avPWh --delete --delete-after "$1" "${DESTI_DIR}" > ${RSYNC_LOG}

echo "--- END --- return $? @ $(date '+%Y-%m-%d %H:%M:%S')" >> ${BACKUP_LOG}

echo ""

Gustavo Arjones 的4种方法

速度

1
tar  >  parallel + rsync  >  cpio  >  rsync

rsync

1
2
3
4
5
find ${SOURCE} \
  -type d \
  -exec rsync --owner --group \
    --archive --copy-links --whole-file \
    --relative --no-compress --progress {} ${DESTINATION}/ \;

tar

For tar I had a small script, mainly to avoid escaping on find -exec:

  • transfer-with-tar.sh
1
2
3
4
5
6
7
8
9
10
#!/usr/bin/env bash

ORIGEN=$1
DESTINATION=/datadrive2

NEW_DIR=${DESTINATION}/${ORIGEN}
mkdir -p ${NEW_DIR}
(cd ${ORIGEN}; tar cf - .) | (cd ${NEW_DIR}; tar xpf -)

echo ${ORIGEN}

执行：

1
2
find ${SOURCE} -type d \
  -exec /root/transfer-with-tar.sh {} \;

cpio

1
2
find ${SOURCE} -type f -print0 2>/dev/null | \
  cpio -0admp ${DESTINATION} &>/dev/null)

rsync + parallel

1
2
3
4
5
find ${SOURCE} -type f > /tmp/backup.txt
time (cat /tmp/backup.txt | parallel -j 8 \
rsync --owner --group \
  --archive --copy-links --whole-file \
  --relative --no-compress --progress {} ${DESTINATION})