Shell脚本如何从云存储恢复数据

本文目录导读：

1. AWS S3 恢复
2. Google Cloud Storage (GCS)
3. Azure Blob Storage
4. 高级恢复脚本（多平台支持）
5. 带加密和压缩的恢复
6. 带错误处理的完整脚本
7. 使用建议

在Shell脚本中从云存储恢复数据，最常用的方法是使用对应云服务商提供的CLI工具（如AWS CLI、GCP gcloud、Azure CLI等）,以下是几个主流云平台的Shell脚本示例：

AWS S3 恢复

基础恢复脚本

#!/bin/bash
# 配置变量
BUCKET_NAME="your-bucket-name"
S3_PATH="backup/database"
LOCAL_PATH="/var/restore/data"
REGION="us-east-1"
# 检查AWS CLI是否安装
if ! command -v aws &> /dev/null; then
    echo "Error: AWS CLI is not installed"
    exit 1
fi
# 从S3下载文件/文件夹
aws s3 cp "s3://${BUCKET_NAME}/${S3_PATH}" "${LOCAL_PATH}" \
    --recursive \
    --region "${REGION}" \
    --quiet
if [ $? -eq 0 ]; then
    echo "Successfully restored from S3"
else
    echo "Error: Failed to restore from S3"
    exit 1
fi

从特定日期恢复

#!/bin/bash
# 恢复7天前的备份
DAYS_AGO=7
BUCKET="my-bucket"
PREFIX="daily-backups/"
# 获取指定日期的备份文件
BACKUP_DATE=$(date -d "${DAYS_AGO} days ago" +%Y-%m-%d)
S3_PATH="${PREFIX}${BACKUP_DATE}/"
# 同步到本地
aws s3 sync "s3://${BUCKET}/${S3_PATH}" "/restore/${BACKUP_DATE}/"

Google Cloud Storage (GCS)

使用gsutil恢复

#!/bin/bash
BUCKET="gs://your-bucket-name"
REMOTE_PATH="backups/database"
LOCAL_PATH="/restore/data"
# 检查gsutil
if ! command -v gsutil &> /dev/null; then
    echo "Error: gsutil is not installed"
    exit 1
fi
# 认证（如果未认证）
# gcloud auth login
# 从GCS恢复
gsutil -m cp -r "${BUCKET}/${REMOTE_PATH}" "${LOCAL_PATH}"
# 验证恢复
if [ $? -eq 0 ] && [ -d "${LOCAL_PATH}" ]; then
    echo "Restore from GCS completed successfully"
else
    echo "Restore failed"
    exit 1
fi

Azure Blob Storage

使用azcopy恢复

#!/bin/bash
CONTAINER="mycontainer"
BLOB_PATH="database/backups"
LOCAL_PATH="/restore/data"
# 配置连接字符串
AZURE_STORAGE_CONNECTION_STRING="your-connection-string"
# 使用azcopy恢复
azcopy copy "https://${STORAGE_ACCOUNT}.blob.core.windows.net/${CONTAINER}/${BLOB_PATH}/*" \
    "${LOCAL_PATH}" \
    --recursive
# 或使用Azure CLI
az storage blob download-batch \
    --destination "${LOCAL_PATH}" \
    --source "${CONTAINER}" \
    --pattern "${BLOB_PATH}/*"

高级恢复脚本（多平台支持）

#!/bin/bash
# 通用恢复脚本
set -euo pipefail
CLOUD_PROVIDER="${1:-aws}"  # 可选: aws, gcp, azure
SOURCE_PATH="${2:-}"
DESTINATION_PATH="${3:-.}"
if [ -z "$SOURCE_PATH" ]; then
    echo "Usage: $0 <cloud_provider> <source_path> [destination_path]"
    exit 1
fi
restore_from_aws() {
    echo "Restoring from AWS S3: ${SOURCE_PATH}"
    aws s3 sync "s3://${SOURCE_PATH}" "${DESTINATION_PATH}"
}
restore_from_gcp() {
    echo "Restoring from GCP: ${SOURCE_PATH}"
    gsutil -m cp -r "gs://${SOURCE_PATH}/*" "${DESTINATION_PATH}"
}
restore_from_azure() {
    echo "Restoring from Azure: ${SOURCE_PATH}"
    az storage blob download-batch \
        --destination "${DESTINATION_PATH}" \
        --source "$(echo ${SOURCE_PATH} | cut -d'/' -f1)" \
        --pattern "$(echo ${SOURCE_PATH} | cut -d'/' -f2-)/*"
}
# 主逻辑
case "${CLOUD_PROVIDER}" in
    aws)
        restore_from_aws
        ;;
    gcp)
        restore_from_gcp
        ;;
    azure)
        restore_from_azure
        ;;
    *)
        echo "Unsupported cloud provider: ${CLOUD_PROVIDER}"
        exit 1
        ;;
esac
if [ $? -eq 0 ]; then
    echo "Restore completed successfully to ${DESTINATION_PATH}"
else
    echo "Restore failed"
    exit 1
fi

带加密和压缩的恢复

#!/bin/bash
# 从加密的S3备份中恢复
DECRYPT_KEY="/path/to/private.key"
BACKUP_FILE="s3://bucket/encrypted_backup.tar.gz.gpg"
# 下载、解密并解压
aws s3 cp "${BACKUP_FILE}" - | \
    gpg --decrypt --batch --passphrase-file "${DECRYPT_KEY}" | \
    tar xzf - -C /restore

带错误处理的完整脚本

#!/bin/bash
set -euo pipefail
# 日志函数
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}
error_log() {
    echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $1" >&2
}
# 环境变量
export AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION:-us-east-1}"
export GOOGLE_APPLICATION_CREDENTIALS="${GOOGLE_APPLICATION_CREDENTIALS:-}"
# 恢复函数
restore_backup() {
    local BACKUP_TYPE=$1
    local TIMESTAMP=$2
    log "Starting restoration of ${BACKUP_TYPE} backup from ${TIMESTAMP}"
    case "${BACKUP_TYPE}" in
        database)
            restore_database_backup "${TIMESTAMP}"
            ;;
        files)
            restore_file_backup "${TIMESTAMP}"
            ;;
        config)
            restore_config_backup "${TIMESTAMP}"
            ;;
        *)
            error_log "Unknown backup type: ${BACKUP_TYPE}"
            return 1
            ;;
    esac
    log "Successfully restored ${BACKUP_TYPE} backup"
}
restore_database_backup() {
    local TIMESTAMP=$1
    local S3_BUCKET="my-backup-bucket"
    local LOCAL_RESTORE_DIR="/var/restore/database"
    mkdir -p "${LOCAL_RESTORE_DIR}"
    # 下载数据库备份
    aws s3 cp "s3://${S3_BUCKET}/database/${TIMESTAMP}/dump.sql.gz" \
        "${LOCAL_RESTORE_DIR}/" || {
        error_log "Failed to download database backup"
        return 1
    }
    # 解压
    gunzip -f "${LOCAL_RESTORE_DIR}/dump.sql.gz"
    # 恢复到数据库（示例）
    # mysql -u root -p${DB_PASSWORD} ${DB_NAME} < "${LOCAL_RESTORE_DIR}/dump.sql"
}
# 主程序
if [ $# -lt 2 ]; then
    echo "Usage: $0 <backup_type> <timestamp>"
    exit 1
fi
restore_backup "$1" "$2" || {
    error_log "Restoration failed"
    exit 1
}

使用建议

1. 安全性：

   • 使用IAM角色或服务账号而不是硬编码密钥
   • 敏感信息通过环境变量或密钥管理服务获取

2. 错误处理：

   • 设置set -e在错误时退出
   • 使用trap处理中断
   • 实现重试机制

3. 验证：

   • 检查文件完整性（MD5/SHA256）
   • 验证恢复后的数据可访问
   • 测试恢复流程

4. 自动化：

   • 可以集成到Cron作业
   • 使用CI/CD管道执行恢复测试
   • 实现监控和告警

5. 性能优化：

   • 使用并行传输（如aws s3的--recursive）
   • 对大文件使用分块传输
   • 考虑本地缓存策略

这些脚本可以根据你的具体需求进行调整,记得先在测试环境中验证恢复流程。