本文目录导读：
- Python + requests（最简单）
- Python + Selenium（处理动态页面）
- Linux/Mac 命令行脚本
- Node.js 脚本
- 处理JavaScript渲染的页面（使用 Pyppeteer）
- 带登录/认证的下载
- 批量下载脚本
- 处理反爬虫
- 安装依赖
- 注意事项
Python + requests（最简单）
import requests
from bs4 import BeautifulSoup
# 基本下载
url = "https://example.com"
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
response = requests.get(url, headers=headers)
response.encoding = 'utf-8'
# 保存HTML文件
with open('page.html', 'w', encoding='utf-8') as f:
f.write(response.text)
# 解析并提取文本
soup = BeautifulSoup(response.text, 'html.parser')
text = soup.get_text()
print(text)
Python + Selenium（处理动态页面）
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# 配置无头浏览器
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
driver = webdriver.Chrome(options=options)
driver.get('https://example.com')
# 获取页面内容
html = driver.page_source
text = driver.find_element_by_tag_name('body').text
# 保存
with open('page.html', 'w', encoding='utf-8') as f:
f.write(html)
driver.quit()
Linux/Mac 命令行脚本
#!/bin/bash # 使用 curl curl -o page.html https://example.com # 使用 wget wget -O page.html https://example.com # 获取页面文本（需要lynx） lynx -dump https://example.com > page.txt
Node.js 脚本
const axios = require('axios');
const cheerio = require('cheerio');
async function downloadPage(url) {
try {
const response = await axios.get(url, {
headers: {
'User-Agent': 'Mozilla/5.0'
}
});
const html = response.data;
const $ = cheerio.load(html);
// 提取文本
const text = $('body').text();
// 保存到文件
const fs = require('fs');
fs.writeFileSync('page.html', html);
fs.writeFileSync('page.txt', text);
return { html, text };
} catch (error) {
console.error('下载失败:', error);
}
}
downloadPage('https://example.com');
处理JavaScript渲染的页面（使用 Pyppeteer）
import asyncio
from pyppeteer import launch
async def download_dynamic_page(url):
browser = await launch(headless=True)
page = await browser.newPage()
# 等待页面加载
await page.goto(url, waitUntil='networkidle0')
# 获取内容
content = await page.content()
text = await page.evaluate('document.body.innerText')
# 保存
with open('page.html', 'w', encoding='utf-8') as f:
f.write(content)
with open('page.txt', 'w', encoding='utf-8') as f:
f.write(text)
await browser.close()
asyncio.get_event_loop().run_until_complete(
download_dynamic_page('https://example.com')
)
带登录/认证的下载
import requests
# 使用会话保持登录状态
session = requests.Session()
# 登录
login_data = {
'username': 'your_username',
'password': 'your_password'
}
session.post('https://example.com/login', data=login_data)
# 下载需要认证的页面
response = session.get('https://example.com/protected-page')
批量下载脚本
import requests
import time
from urllib.parse import urljoin
def batch_download(urls, delay=1):
for i, url in enumerate(urls):
try:
response = requests.get(url)
with open(f'page_{i}.html', 'w') as f:
f.write(response.text)
print(f'已下载: {url}')
time.sleep(delay) # 设置延迟避免被封
except Exception as e:
print(f'下载失败 {url}: {e}')
# 使用示例
urls = [
'https://example.com/page1',
'https://example.com/page2',
'https://example.com/page3'
]
batch_download(urls)
处理反爬虫
import requests
from fake_useragent import UserAgent
ua = UserAgent()
headers = {
'User-Agent': ua.random,
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1'
}
# 使用代理（可选）
proxies = {
'http': 'http://proxy.example.com:8080',
'https': 'https://proxy.example.com:8080'
}
response = requests.get(url, headers=headers, proxies=proxies)
安装依赖
根据你选择的方法,安装所需库：
# Python pip install requests beautifulsoup4 selenium pyppeteer # Node.js npm install axios cheerio # Linux工具 sudo apt-get install curl wget lynx
注意事项
- 遵守robots.txt：检查网站的robots.txt文件
- 控制请求频率：避免对服务器造成压力
- 设置超时：避免长时间无响应
- 处理编码：确保正确解析不同编码的页面
- 错误处理：添加异常处理机制
选择哪种方法取决于你的具体需求：
- 静态页面：用 requests/curl
- 动态页面：用 Selenium 或 Pyppeteer
- 需要登录：用 session 保持登录状态
- 批量下载：写循环脚本加延迟
