脚本怎样保留唯一数据项

本文目录导读：

1. Python 中使用 set（最常用）
2. 处理复杂数据结构
3. Shell 脚本去重
4. JavaScript 去重
5. 处理大文件（内存高效）
6. 实际应用示例
7. 性能优化技巧
8. 选择建议

Python 中使用 set（最常用）

# 方法1：set直接去重
data = [1, 2, 2, 3, 3, 4, 5, 5]
unique_data = list(set(data))
print(unique_data)  # [1, 2, 3, 4, 5]
# 方法2：保持原顺序去重
def unique_ordered(data):
    seen = set()
    result = []
    for item in data:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result
print(unique_ordered([3, 1, 2, 3, 1, 4]))  # [3, 1, 2, 4]

处理复杂数据结构

# 字典列表去重
users = [
    {"id": 1, "name": "Alice"},
    {"id": 2, "name": "Bob"},
    {"id": 1, "name": "Alice"},  # 重复
    {"id": 3, "name": "Charlie"}
]
# 按指定字段去重
def unique_by_key(data, key):
    seen = set()
    result = []
    for item in data:
        value = item[key]
        if value not in seen:
            seen.add(value)
            result.append(item)
    return result
print(unique_by_key(users, "id"))

Shell 脚本去重

# 文本去重
sort data.txt | uniq
# 统计重复行
sort data.txt | uniq -c
# 仅显示重复行
sort data.txt | uniq -d
# 忽略大小写
sort data.txt | uniq -i
# 处理多个文件
sort file1.txt file2.txt | uniq > unique_data.txt

JavaScript 去重

// 基本数组去重
const data = [1, 2, 2, 3, 3, 4, 5, 5];
const unique = [...new Set(data)];
console.log(unique); // [1, 2, 3, 4, 5]
// 对象数组去重
const users = [
  { id: 1, name: "Alice" },
  { id: 2, name: "Bob" },
  { id: 1, name: "Alice" },
];
const uniqueUsers = [...new Map(users.map(user => [user.id, user])).values()];

处理大文件（内存高效）

# 大文件逐行处理去重
def deduplicate_large_file(input_file, output_file):
    seen = set()
    with open(input_file, 'r') as infile, open(output_file, 'w') as outfile:
        for line in infile:
            line = line.strip()
            if line not in seen:
                seen.add(line)
                outfile.write(line + '\n')
# 使用
deduplicate_large_file('large_data.txt', 'unique_data.txt')

实际应用示例

# Excel CSV 数据去重
import csv
def deduplicate_csv(input_file, output_file, key_column=0):
    seen = set()
    with open(input_file, 'r') as infile, open(output_file, 'w', newline='') as outfile:
        reader = csv.reader(infile)
        writer = csv.writer(outfile)
        # 写入标题行
        header = next(reader)
        writer.writerow(header)
        # 去重数据行
        for row in reader:
            key = row[key_column]
            if key not in seen:
                seen.add(key)
                writer.writerow(row)
deduplicate_csv('data.csv', 'clean_data.csv', key_column=0)

性能优化技巧

# 使用字典保持顺序且性能好
def fast_unique(items):
    return list(dict.fromkeys(items))
# 大数据量使用
# pip install pandas
import pandas as pd
def pandas_unique(file_path):
    df = pd.read_csv(file_path)
    return df.drop_duplicates()
# 并行处理
from multiprocessing import Pool
def parallel_unique(chunks):
    seen = set()
    for chunk in chunks:
        seen.update(chunk)
    return seen

选择建议

需要根据你的具体场景选择最合适的方法！