怎样在PHP项目中实现BM25？

本文目录导读：

1. 方案一：纯PHP手动实现BM25
2. 方案二：使用Elasticsearch（推荐）
3. 方案三：使用MeiliSearch（替代方案）

在PHP项目中实现BM25算法，通常有两种方式：手动实现（灵活但需注意优化）或使用搜索引擎库（如Elasticsearch、MeiliSearch）,以下提供完整实现方案。

纯PHP手动实现BM25

核心算法实现

<?php
class BM25
{
    private float $k1 = 1.2;   // 饱和控制参数
    private float $b = 0.75;   // 长度归一化参数
    private float $epsilon = 0.25; // 避免除零
    private array $corpus;      // 文档集合
    private int $totalDocs;     // 文档总数
    private array $docLengths;  // 每个文档的长度（词数）
    private float $avgDocLength; // 平均文档长度
    private array $df;          // 文档频率（term在多少文档中出现）
    private array $tf;          // 词频矩阵 [docId][term] => count
    public function __construct(array $corpus, float $k1 = 1.2, float $b = 0.75)
    {
        $this->k1 = $k1;
        $this->b = $b;
        $this->corpus = $corpus;
        $this->totalDocs = count($corpus);
        $this->buildIndex();
    }
    private function buildIndex(): void
    {
        $this->docLengths = array_fill(0, $this->totalDocs, 0);
        $this->tf = array_fill(0, $this->totalDocs, []);
        $this->df = [];
        foreach ($this->corpus as $docId => $text) {
            $terms = $this->tokenize($text);
            $this->docLengths[$docId] = count($terms);
            // 统计term在文档中的频率
            $termCounts = array_count_values($terms);
            foreach ($termCounts as $term => $count) {
                $this->tf[$docId][$term] = $count;
                // 记录文档频率（首次出现）
                if (!isset($this->df[$term])) {
                    $this->df[$term] = 0;
                }
            }
            // 统计文档频率（每个文档唯一计数）
            $uniqueTerms = array_unique($terms);
            foreach ($uniqueTerms as $term) {
                $this->df[$term]++;
            }
        }
        // 计算平均文档长度
        $this->avgDocLength = array_sum($this->docLengths) / $this->totalDocs;
    }
    /**
     * 分词函数（基础实现，可根据需求替换为jieba-php等）
     */
    private function tokenize(string $text): array
    {
        // 基础英文分词：移除标点，转小写，按空格分割
        $text = mb_strtolower($text);
        $text = preg_replace('/[^a-z0-9\s]/u', ' ', $text);
        $words = preg_split('/\s+/', $text, -1, PREG_SPLIT_NO_EMPTY);
        // 简单过滤停用词（可扩展）
        $stopWords = ['the', 'a', 'an', 'is', 'are', 'was', 'were', 'in', 'on', 'at', 'to', 'for'];
        return array_diff($words, $stopWords);
    }
    /**
     * 计算单个文档的BM25分数
     */
    public function score(int $docId, string $query): float
    {
        $queryTerms = $this->tokenize($query);
        $score = 0.0;
        foreach ($queryTerms as $term) {
            if (!isset($this->df[$term])) continue;
            $n = $this->df[$term]; // 包含该term的文档数
            $tf = $this->tf[$docId][$term] ?? 0; // 该文档中term的频率
            // IDF计算
            $idf = log(($this->totalDocs - $n + 0.5) / ($n + 0.5) + 1);
            // 长度归一化
            $docLength = $this->docLengths[$docId];
            $normalized = $this->k1 * (1 - $this->b + $this->b * ($docLength / $this->avgDocLength));
            // BM25核心公式
            $score += $idf * ($tf * ($this->k1 + 1)) / ($tf + $normalized);
        }
        return $score;
    }
    /**
     * 搜索并排序所有文档
     */
    public function search(string $query): array
    {
        $results = [];
        foreach ($this->corpus as $docId => $text) {
            $score = $this->score($docId, $query);
            if ($score > 0) {
                $results[] = [
                    'docId' => $docId,
                    'text' => $text,
                    'score' => $score
                ];
            }
        }
        // 按分数降序排序
        usort($results, function($a, $b) {
            return $b['score'] <=> $a['score'];
        });
        return $results;
    }
}
// 使用示例
$corpus = [
    "BM25 is a ranking function used by search engines",
    "It estimates the relevance of documents to a search query",
    "The algorithm is based on probabilistic information retrieval",
];
$bm25 = new BM25($corpus);
$results = $bm25->search("search engine ranking");
echo "搜索结果：\n";
foreach ($results as $result) {
    printf("文档%d (%.4f): %s\n", $result['docId'], $result['score'], $result['text']);
}

中文分词支持（需安装jieba-php）

composer require fukuball/jieba-php

// 修改tokenize方法支持中文
private function tokenize(string $text): array
{
    // 英文部分
    $englishText = preg_replace('/[^a-z0-9\s]/u', ' ', mb_strtolower($text));
    $englishWords = preg_split('/\s+/', $englishText, -1, PREG_SPLIT_NO_EMPTY);
    // 中文部分（使用jieba分词）
    $chineseText = preg_replace('/[^\\x{4e00}-\\x{9fa5}]/u', ' ', $text);
    $chineseWords = [];
    if (!empty(trim($chineseText))) {
        // 初始化jieba（只需一次）
        static $jieba = null;
        if ($jieba === null) {
            $jieba = new \Fukuball\Jieba\Jieba();
            $jieba->init();
        }
        $chineseWords = $jieba->cut($chineseText);
    }
    return array_merge($englishWords, $chineseWords);
}

性能优化建议

• 缓存IDF值：预计算并缓存，避免重复log计算
• 使用数据库索引：将词频、文档频率存入MySQL/PostgreSQL
• 倒排索引：构建倒排索引加速查询

private function precomputeIdf(): void
{
    foreach ($this->df as $term => $n) {
        $this->cache[$term] = log(($this->totalDocs - $n + 0.5) / ($n + 0.5) + 1);
    }
}

使用Elasticsearch（推荐）

安装PHP客户端

composer require elasticsearch/elasticsearch

配置BM25相似度

<?php
require 'vendor/autoload.php';
use Elasticsearch\ClientBuilder;
$client = ClientBuilder::create()->build();
// 创建索引时指定使用BM25
$params = [
    'index' => 'my_index',
    'body' => [
        'settings' => [
            'similarity' => [
                'default' => [
                    'type' => 'BM25',
                    'k1' => 1.2,
                    'b' => 0.75
                ]
            ]
        ],
        'mappings' => [
            'properties' => [
                'content' => [
                    'type' => 'text',
                    'similarity' => 'BM25'
                ]
            ]
        ]
    ]
];
$client->indices()->create($params);
// 索引文档
$client->index([
    'index' => 'my_index',
    'id' => '1',
    'body' => ['content' => 'BM25 is a ranking function']
]);
// 搜索
$results = $client->search([
    'index' => 'my_index',
    'body' => [
        'query' => [
            'match' => [
                'content' => 'ranking function'
            ]
        ]
    ]
]);

调用Elasticsearch获取分数

foreach ($results['hits']['hits'] as $hit) {
    echo "Score: " . $hit['_score'] . "\n";
    echo "Content: " . $hit['_source']['content'] . "\n";
}

使用MeiliSearch（替代方案）

MeiliSearch默认使用类似BM25的算法,PHP集成更简单：

composer require meilisearch/meilisearch-php

$client = new \MeiliSearch\Client('http://localhost:7700');
$index = $client->index('docs');
// 添加文档
$docs = [
    ['id' => 1, 'title' => 'BM25 Algorithm', 'content' => '...'],
    ['id' => 2, 'title' => 'Search Ranking', 'content' => '...'],
];
$index->addDocuments($docs);
// 搜索
$results = $index->search('BM25', ['limit' => 10]);

建议：实际项目中优先选择Elasticsearch或MeiliSearch等成熟方案,仅在数据量很小或需要深度定制时才手动实现BM25。