用脚本下载网页内容怎么做

本文目录导读：

1. Python + requests（最简单）
2. Python + Selenium（处理动态页面）
3. Linux/Mac 命令行脚本
4. Node.js 脚本
5. 处理JavaScript渲染的页面（使用 Pyppeteer）
6. 带登录/认证的下载
7. 批量下载脚本
8. 处理反爬虫
9. 安装依赖
10. 注意事项

Python + requests（最简单）

import requests
from bs4 import BeautifulSoup
# 基本下载
url = "https://example.com"
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}
response = requests.get(url, headers=headers)
response.encoding = 'utf-8'
# 保存HTML文件
with open('page.html', 'w', encoding='utf-8') as f:
    f.write(response.text)
# 解析并提取文本
soup = BeautifulSoup(response.text, 'html.parser')
text = soup.get_text()
print(text)

Python + Selenium（处理动态页面）

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# 配置无头浏览器
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
driver = webdriver.Chrome(options=options)
driver.get('https://example.com')
# 获取页面内容
html = driver.page_source
text = driver.find_element_by_tag_name('body').text
# 保存
with open('page.html', 'w', encoding='utf-8') as f:
    f.write(html)
driver.quit()

Linux/Mac 命令行脚本

#!/bin/bash
# 使用 curl
curl -o page.html https://example.com
# 使用 wget
wget -O page.html https://example.com
# 获取页面文本（需要lynx）
lynx -dump https://example.com > page.txt

Node.js 脚本

const axios = require('axios');
const cheerio = require('cheerio');
async function downloadPage(url) {
    try {
        const response = await axios.get(url, {
            headers: {
                'User-Agent': 'Mozilla/5.0'
            }
        });
        const html = response.data;
        const $ = cheerio.load(html);
        // 提取文本
        const text = $('body').text();
        // 保存到文件
        const fs = require('fs');
        fs.writeFileSync('page.html', html);
        fs.writeFileSync('page.txt', text);
        return { html, text };
    } catch (error) {
        console.error('下载失败:', error);
    }
}
downloadPage('https://example.com');

处理JavaScript渲染的页面（使用 Pyppeteer）

import asyncio
from pyppeteer import launch
async def download_dynamic_page(url):
    browser = await launch(headless=True)
    page = await browser.newPage()
    # 等待页面加载
    await page.goto(url, waitUntil='networkidle0')
    # 获取内容
    content = await page.content()
    text = await page.evaluate('document.body.innerText')
    # 保存
    with open('page.html', 'w', encoding='utf-8') as f:
        f.write(content)
    with open('page.txt', 'w', encoding='utf-8') as f:
        f.write(text)
    await browser.close()
asyncio.get_event_loop().run_until_complete(
    download_dynamic_page('https://example.com')
)

带登录/认证的下载

import requests
# 使用会话保持登录状态
session = requests.Session()
# 登录
login_data = {
    'username': 'your_username',
    'password': 'your_password'
}
session.post('https://example.com/login', data=login_data)
# 下载需要认证的页面
response = session.get('https://example.com/protected-page')

批量下载脚本

import requests
import time
from urllib.parse import urljoin
def batch_download(urls, delay=1):
    for i, url in enumerate(urls):
        try:
            response = requests.get(url)
            with open(f'page_{i}.html', 'w') as f:
                f.write(response.text)
            print(f'已下载: {url}')
            time.sleep(delay)  # 设置延迟避免被封
        except Exception as e:
            print(f'下载失败 {url}: {e}')
# 使用示例
urls = [
    'https://example.com/page1',
    'https://example.com/page2',
    'https://example.com/page3'
]
batch_download(urls)

处理反爬虫

import requests
from fake_useragent import UserAgent
ua = UserAgent()
headers = {
    'User-Agent': ua.random,
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}
# 使用代理（可选）
proxies = {
    'http': 'http://proxy.example.com:8080',
    'https': 'https://proxy.example.com:8080'
}
response = requests.get(url, headers=headers, proxies=proxies)

安装依赖

根据你选择的方法,安装所需库：

# Python
pip install requests beautifulsoup4 selenium pyppeteer
# Node.js
npm install axios cheerio
# Linux工具
sudo apt-get install curl wget lynx

注意事项

1. 遵守robots.txt：检查网站的robots.txt文件
2. 控制请求频率：避免对服务器造成压力
3. 设置超时：避免长时间无响应
4. 处理编码：确保正确解析不同编码的页面
5. 错误处理：添加异常处理机制

选择哪种方法取决于你的具体需求：

• 静态页面：用 requests/curl
• 动态页面：用 Selenium 或 Pyppeteer
• 需要登录：用 session 保持登录状态
• 批量下载：写循环脚本加延迟