本文目录导读：
我来分享几种Python生成文本摘要的方法,从简单到高级：
基于统计的提取式摘要
使用TextRank算法
!pip install sumy nltk scikit-learn
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.text_rank import TextRankSummarizer
import nltk
nltk.download('punkt')
# 示例文本
text = """
人工智能(AI)是计算机科学的一个分支，致力于创建能够模拟人类智能的系统。
近年来，深度学习技术的突破使得AI在图像识别、自然语言处理等领域取得了显著进展。
机器学习作为AI的核心技术，通过算法使计算机能够从数据中学习和改进。
自然语言处理让计算机能够理解和生成人类语言，应用包括机器翻译、情感分析等。
计算机视觉技术使机器能够从图像和视频中提取信息，用于自动驾驶、医疗诊断等领域。
"""
# 使用sumy库进行文本摘要
parser = PlaintextParser.from_string(text, Tokenizer("english"))
summarizer = TextRankSummarizer()
summary = summarizer(parser.document, sentences_count=2)
print("原始文本长度:", len(text))
print("")
for sentence in summary:
print(sentence)
使用Hugging Face Transformers
基于BART的生成式摘要
!pip install transformers torch
from transformers import pipeline
# 加载预训练摘要模型
summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
# 长文本示例
long_text = """
Climate change is one of the most pressing challenges facing humanity today.
The Earth's average temperature has risen significantly over the past century,
primarily due to increased greenhouse gas emissions from human activities.
Scientists warn that if current trends continue, we could see catastrophic
consequences including rising sea levels, more frequent extreme weather events,
and loss of biodiversity. However, there is hope. Renewable energy technologies
like solar and wind power are becoming more affordable and efficient. Many
countries have committed to reducing their carbon emissions and transitioning
to sustainable energy sources. Individual actions, such as reducing waste and
choosing sustainable products, can also make a difference in combating climate change.
"""
summary = summarizer(long_text, max_length=50, min_length=20, do_sample=False)
print("", summary[0]['summary_text'])
使用T5模型进行多语言摘要
from transformers import T5ForConditionalGeneration, T5Tokenizer
# 加载T5模型
model_name = "t5-small" # 可替换为 t5-base, t5-large
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)
# 中文文本示例
chinese_text = """
深度学习是机器学习的一个重要分支，它通过构建多层神经网络来学习数据的表示。
近年来，深度学习在计算机视觉、自然语言处理、语音识别等领域取得了革命性的进展。
卷积神经网络(CNN)在图像识别任务中表现优异，而循环神经网络(RNN)则擅长处理序列数据。
Transformer架构的出现进一步推动了自然语言处理的发展，BERT、GPT等预训练模型
在各种NLP任务上达到了新的性能高度。
"""
# 预处理输入
input_text = "summarize: " + chinese_text
inputs = tokenizer.encode(input_text, return_tensors="pt", max_length=512, truncation=True)
summary_ids = model.generate(
inputs,
max_length=100,
min_length=30,
num_beams=4,
length_penalty=2.0,
early_stopping=True
)
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
print("T5摘要:", summary)
基于LSA（潜在语义分析）的摘要
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
import nltk
nltk.download('punkt')
def lsa_summarize(text, num_sentences=3):
# 分句
sentences = nltk.sent_tokenize(text)
# TF-IDF向量化
vectorizer = TfidfVectorizer(stop_words='english')
tfidf_matrix = vectorizer.fit_transform(sentences)
# LSA降维
lsa = TruncatedSVD(n_components=1)
lsa_matrix = lsa.fit_transform(tfidf_matrix)
# 选择最重要的句子
sentence_scores = lsa_matrix.flatten()
ranked_sentences = np.argsort(sentence_scores)[::-1][:num_sentences]
# 按原始顺序排列
ranked_sentences.sort()
summary = [sentences[i] for i in ranked_sentences]
return ' '.join(summary)
# 使用示例
sample_text = """
Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience.
It focuses on developing computer programs that can access data and use it to learn for themselves.
The process of machine learning begins with observations or data, such as examples or direct experience.
Machine learning algorithms are often categorized as supervised or unsupervised.
Deep learning is a type of machine learning that uses neural networks with many layers.
"""
summary = lsa_summarize(sample_text, num_sentences=2)
print("LSA摘要:", summary)
完整的摘要案例 - 新闻文本处理
import re
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import sent_tokenize, word_tokenize
from collections import Counter
nltk.download('stopwords')
class TextSummarizer:
def __init__(self, language='english'):
self.stop_words = set(stopwords.words(language))
def preprocess(self, text):
# 文本清洗
text = re.sub(r'\s+', ' ', text)
text = re.sub(r'[^\w\s]', '', text)
return text.lower()
def word_frequency(self, text):
words = word_tokenize(text)
words = [word for word in words if word not in self.stop_words]
freq = Counter(words)
# 归一化词频
max_freq = max(freq.values())
for word in freq:
freq[word] /= max_freq
return freq
def sentence_scores(self, sentences, freq):
scores = {}
for i, sentence in enumerate(sentences):
word_count = len(word_tokenize(sentence))
score = 0
for word in word_tokenize(sentence.lower()):
if word in freq:
score += freq[word]
scores[i] = score / word_count if word_count > 0 else 0
return scores
def summarize(self, text, num_sentences=3):
# 分句
sentences = sent_tokenize(text)
if len(sentences) <= num_sentences:
return text
# 预处理
processed_text = self.preprocess(text)
# 计算词频
freq = self.word_frequency(processed_text)
# 计算句子得分
scores = self.sentence_scores(sentences, freq)
# 选择最佳句子
best_sentences = sorted(scores.items(), key=lambda x: x[1], reverse=True)[:num_sentences]
best_sentences.sort(key=lambda x: x[0]) # 按原文顺序
summary = ' '.join([sentences[idx] for idx, _ in best_sentences])
return summary
# 使用自定义摘要器
summarizer = TextSummarizer()
news_text = """
Scientists have discovered a new species of dinosaur in Argentina that lived
approximately 140 million years ago. The herbivorous dinosaur, named
'Chucarosaurus diripienda', belongs to the titanosaur family. It is estimated
to have been about 30 meters long and weighed up to 50 tons. The discovery was
made in the Patagonia region, which is known for its rich fossil deposits.
Researchers believe this finding could provide new insights into the evolution
of sauropod dinosaurs. The fossil remains include several well-preserved bones
that will help scientists understand more about the anatomy and lifestyle of
these giant creatures.
"""
summary = summarizer.summarize(news_text, num_sentences=2)
print("自定义摘要器结果:")
print(f"原始文本长度: {len(news_text)} 字符")
print(f"摘要长度: {len(summary)} 字符")
print(f"压缩比: {len(summary)/len(news_text)*100:.1f}%")
print("\n摘要内容:")
print(summary)
实用建议
模型选择指南：
- 速简单任务：使用sumy或自定义算法
- 英文高质量摘要：facebook/bart-large-cnn
- 多语言支持：T5系列模型
- 中文摘要：使用Hugging Face上的中文预训练模型
优化技巧：
# 文本长度控制
summary = summarizer(
long_text,
max_length=150, # 最大长度
min_length=50, # 最小长度
length_penalty=2.0, # 长度惩罚
num_beams=4, # beam search数量
early_stopping=True
)
方法取决于你的具体需求、文本类型和性能要求，对于生产环境，建议使用预训练模型,效果更好。
