使用正则表达式清理人工智能文本

为什么要为 AI 清理文本？

一个杂乱的示例

raw = "<p>Contact us at info@shop.com or visit https://shop.com NOW!!!   Thanks</p>"

去除 HTML 标签

import re

text = re.sub("<[^>]+>", "", raw)
print(text)   # "Contact us at info@shop.com or visit https://shop.com NOW!!!   Thanks"

去除 URL

import re

text = re.sub("https?://\S+", "", text)
print(text)   # URL removed

去除电子邮件地址

import re

text = re.sub("\S+@\S+\.\S+", "", text)
print(text)   # email removed

使用掩码而不是删除

import re

masked = re.sub("\S+@\S+\.\S+", "[EMAIL]", "reach me at a@b.com please")
print(masked)   # "reach me at [EMAIL] please"

去除特殊字符

import re

text = re.sub("[^A-Za-z0-9 ]", "", "Hello!! @world #2026")
print(text)   # "Hello world 2026"

规范化空白字符

import re

text = re.sub("\s+", " ", "Hello    world\n\n  again").strip()
print(text)   # "Hello world again"

转换为小写以及顺序的重要性

text = text.lower()
# Pipeline order: tags -> urls -> emails -> specials -> whitespace -> lowercase

构建清理流程

import re

def clean_text(s):
    s = re.sub("<[^>]+>", " ", s)          # html tags
    s = re.sub("https?://\S+", " ", s)     # urls
    s = re.sub("\S+@\S+\.\S+", " ", s)    # emails
    s = re.sub("[^A-Za-z0-9 ]", " ", s)     # special chars
    s = re.sub("\s+", " ", s).strip()       # whitespace
    return s.lower()

print(clean_text(raw))

将清理流程应用于 DataFrame

import pandas as pd

df["clean"] = df["text"].apply(clean_text)
print(df[["text", "clean"]].head())

快速检查：空白字符规范化

回顾：使用正则表达式清理文本