Fast String Processing with Polars — Scam Emails Dataset | by Antons Tocilins-Ruberts | May, 2023

Clear, course of and tokenise texts in milliseconds utilizing in-built Polars string expressions

10 min learn

·

10 hours in the past

With the big scale adoption of Giant language Fashions (LLMs) it may appear that we’re previous the stage the place we needed to manually clear and course of textual content information. Sadly, me and different NLP practitioners can attest that that is very a lot not the case. Clear textual content information is required at each stage of NLP complexity — from fundamental textual content analytics to machine studying and LLMs. This submit will showcase how this laborious and tedious course of will be considerably sped up utilizing Polars.

Polars is a blazingly quick Information Body library written in Rust that’s extremely environment friendly with dealing with strings (attributable to its Arrow backend). Polars shops strings within the Utf8 format utilizing Arrow backend which makes string traversal cache-optimal and predictable. Additionally, it exposes a number of in-built string operations underneath the str namespace which makes the string operations parallelised. Each of those elements make working with strings extraordinarily straightforward and quick.

The library shares a number of syntaxis with Pandas however there are additionally a number of quirks that you just’ll have to get used to. This submit will stroll you thru working with strings however for a complete overview I extremely advocate this “Getting Started” information because it offers you overview of the library.

You could find all of the code on this GitHub repo, so be sure to drag it if wish to code alongside (don’t neglect to ⭐ it). To make this submit extra sensible and enjoyable, I’ll showcase how we are able to clear a small rip-off e-mail dataset which will be discovered on Kaggle (License CC BY-SA 4.0). Polars will be put in utilizing pip — pip set up polars and the really useful Python model is 3.10 .

The objective of this pipeline is to parse the uncooked textual content file right into a DataFrame that can be utilized for additional analytics/modelling. Listed below are the general steps that might be carried out:

1. Learn in textual content information
2. Extract related fields (e.g. sender e-mail, object, textual content, and so on.)
3. Extract helpful options from these fields (e.g. size, % of digits, and so on)
4. Pre-process textual content for additional evaluation
5. Carry out some fundamental textual content analytics

With out additional ado, let’s start!

Studying Information

Assuming that the textual content file with emails is saved as fraudulent_emails.txt , right here’s the operate used to learn them in:

def load_emails_txt(path: str, split_str: str = "From r  ") -> listing[str]:
with open(path, "r", encoding="utf-8", errors="ignore") as file:
textual content = file.learn()
emails = textual content.cut up(split_str)
return emails

In the event you discover the textual content information you’ll see that the emails have two major sections

• Metadata (begins with From r ) that incorporates e-mail sender, topic, and so on.
• Electronic mail textual content (begins after Standing: O or Standing: RO )

I’m utilizing the primary sample to separate the continual textual content file into an inventory of emails. General, we must always be capable of learn in 3977 emails that we put right into a Polars DataFrame for additional evaluation.

emails = load_emails_txt("fradulent_emails.txt")
emails_pl = pl.DataFrame({"emails": emails})
print(len(emails))
>>> 3977

Extracting Related Fields

Now the difficult half begins. How can we extract related fields from this mess of a textual content information? Sadly, the reply is regex.

Sender and Topic

Upon additional inspection of metadata (beneath) you possibly can see that it has fields From: and Topic: that are going to be very helpful for us.

From r  Wed Oct 30 21:41:56 2002
Return-Path: <james_ngola2002@maktoob.com>
X-Sieve: cmu-sieve 2.0
Return-Path: <james_ngola2002@maktoob.com>
Message-Id: <200210310241.g9V2fNm6028281@cs.CU>
From: "MR. JAMES NGOLA." <james_ngola2002@maktoob.com>
Reply-To: james_ngola2002@maktoob.com
To: webmaster@aclweb.org
Date: Thu, 31 Oct 2002 02:38:20 +0000
Topic: URGENT BUSINESS ASSISTANCE AND PARTNERSHIP
X-Mailer: Microsoft Outlook Specific 5.00.2919.6900 DM
MIME-Model: 1.0
Content material-Sort: textual content/plain; charset="us-ascii"
Content material-Switch-Encoding: 8bit
Standing: O

In the event you maintain scrolling the emails, you’ll discover that there are just a few codecs for the From: area. The primary format you see above the place we’ve got each title and e-mail. The second format incorporates solely the e-mail e.g. From: 123@abc.com or From: “123@abc.com” . With this in thoughts, we’ll want three regex patterns — one for topic, and two for sender (title with e-mail and simply e-mail).

email_pattern = r"From:s*([^<ns]+)"
subject_pattern = r"Topic:s*(.*)"
name_email_pattern = r'From:s*"?([^"<]+)"?s*<([^>]+)>'

Polars has an str.extract methodology that may evaluate the above patterns to our textual content and (you guessed it) extract the matching teams. Right here’s how one can apply it to the emails_pl DataFrame.

emails_pl = emails_pl.with_columns(
# Extract the primary match group as e-mail
pl.col("emails").str.extract(name_email_pattern, 1).alias("sender_name"),
# Extract the second match group as e-mail
pl.col("emails").str.extract(name_email_pattern, 2).alias("sender_email"),
# Extract the topic 
pl.col("emails").str.extract(subject_pattern, 1).alias("topic"),
).with_columns(
# In instances the place we did not extract e-mail
pl.when(pl.col("sender_email").is_null())
# Attempt one other sample (simply e-mail)
.then(pl.col("emails").str.extract(email_pattern, 1))
# If we do have an e-mail, do nothing
.in any other case(pl.col("sender_email"))
.alias("sender_email")
)

As you possibly can see moreover str.extract we’re additionally utilizing a pl.when().then().in any other case() expressions (Polars model of if/else) to account for a second e-mail solely sample. In the event you print out the outcomes you’ll see that most often it ought to’ve labored appropriately (and extremely quick). We now have sender_name , sender_email and topic fields for our evaluation.

Electronic mail Textual content

As was famous above, the precise e-mail textual content begins after Standing: O (opened) or Standing: RO (learn and opened) which signifies that we are able to utilise this sample to separate the e-mail into “metadata” and “textual content” components. Under you possibly can see the three steps that we have to take to extract the required area and the corresponding Polars methodology to carry out them.

1. Exchange Standing: RO with Standing: O in order that we solely have one “cut up” sample — use str.exchange
2. Break up the precise string by Standing: O — use str.cut up
3. Get the second ingredient (textual content) of the ensuing listing — use arr.get(1)

emails_pl = emails_pl.with_columns(
# Apply operations to the emails column
pl.col("emails")
# Make these two statuses the identical
.str.exchange("Standing: RO", "Standing: O", literal=True)
# Break up utilizing the standing string
.str.cut up("Standing: O")
# Get the second ingredient
.arr.get(1)
# Rename the sector
.alias("email_text")
)

Et voilà! We now have extracted necessary fields in just some milliseconds. Let’s put all of it into one coherent operate that we are able to later use within the pipeline.

def extract_fields(emails: pl.DataFrame) -> pl.DataFrame:
email_pattern = r"From:s*([^<ns]+)"
subject_pattern = r"Topic:s*(.*)"
name_email_pattern = r'From:s*"?([^"<]+)"?s*<([^>]+)>'
emails = (
emails.with_columns(
pl.col("emails").str.extract(name_email_pattern, 2).alias("sender_email"),
pl.col("emails").str.extract(name_email_pattern, 1).alias("sender_name"),
pl.col("emails").str.extract(subject_pattern, 1).alias("topic"),
)
.with_columns(
pl.when(pl.col("sender_email").is_null())
.then(pl.col("emails").str.extract(email_pattern, 1))
.in any other case(pl.col("sender_email"))
.alias("sender_email")
)
.with_columns(
pl.col("emails")
.str.exchange("Standing: RO", "Standing: O", literal=True)
.str.cut up("Standing: O")
.arr.get(1)
.alias("email_text")
)
)
return emails

Now, we are able to transfer on to the characteristic technology half.

Characteristic Engineering

From private expertise, rip-off emails are typically very detailed and lengthy (since scammers are attempting to win your belief) so the character size of an e-mail goes to be fairly informative. Additionally, they closely use exclamation factors and digits, so calculating the proportion of non-characters in an e-mail can be helpful. Lastly, scammers love to make use of caps lock, so let’s calculate the proportion of capital letters as properly. There are in fact, many extra options we may create however to not make this submit too lengthy, let’s simply give attention to these two.

The primary characteristic will be very simply created utilizing an in-built str.n_chars() operate. The 2 different options will be computed utilizing regex and str.count_match(). Under you will discover the operate to calculate these three options. Much like the earlier operate, it makes use of with_columns() clause to hold over the previous options and create the brand new ones on high of them.

def email_features(information: pl.DataFrame, col: str) -> pl.DataFrame:
information = information.with_columns(
pl.col(col).str.n_chars().alias(f"{col}_length"),
).with_columns(
(pl.col(col).str.count_match(r"[A-Z]") / pl.col(f"{col}_length")).alias(
f"{col}_percent_capital"
),
(pl.col(col).str.count_match(r"[^A-Za-z ]") / pl.col(f"{col}_length")).alias(
f"{col}_percent_digits"
),
)
return information

Textual content Cleansing

In the event you print out just a few of the emails we’ve extracted, you’ll discover some issues that have to be cleaned. For instance:

• HTML tags are nonetheless current in a number of the emails
• Numerous non-alphabetic characters are used
• Some emails are written in uppercase, some in lowercase, and a few are blended

Identical as above, we’re going to make use of common expressions to wash up the information. Nevertheless, now the tactic of selection is str.replace_all as a result of we wish to exchange all of the matched cases, not simply the primary one. Moreover, we’ll use str.to_lowercase() to make all textual content lowercase.

emails_pl = emails_pl.with_columns(
# Apply operations to the emails textual content column
pl.col("email_text")
# Take away all the information in <..> (HTML tags)
.str.replace_all(r"<.*?>", "")
# Exchange non-alphabetic characters (besides whitespace) in textual content
.str.replace_all(r"[^a-zA-Zs]+", " ")
# Exchange a number of whitespaces with one whitespace 
# We have to do that due to the earlier cleansing step
.str.replace_all(r"s+", " ")
# Make all textual content lowercase
.str.to_lowercase()
# Preserve the sector's title
.keep_name()
)

Now, let’s refactor this chain of operations right into a operate, in order that it might be utilized to the opposite columns of curiosity as properly.

def email_clean(
information: pl.DataFrame, col: str, new_col_name: str | None = None
) -> pl.DataFrame:
information = information.with_columns(
pl.col(col)
.str.replace_all(r"<.*?>", " ")
.str.replace_all(r"[^a-zA-Zs]+", " ")
.str.replace_all(r"s+", " ")
.str.to_lowercase()
.alias(new_col_name if new_col_name isn't None else col)
)
return information

Textual content Tokenisation

As a closing step within the pre-processing pipeline, we’re going to tokenise the textual content. Tokenisation goes to occur utilizing the already acquainted methodology str.cut up() the place as a cut up token we’re going to specify a whitespace.

emails_pl = emails_pl.with_columns(
pl.col("email_text").str.cut up(" ").alias("email_text_tokenised")
)

Once more, let’s put this code right into a operate for our closing pipeline.

def tokenise_text(information: pl.DataFrame, col: str, split_token: str = " ") -> pl.DataFrame:
information = information.with_columns(pl.col(col).str.cut up(split_token).alias(f"{col}_tokenised"))
return information

Eradicating Cease Phrases

In the event you’ve labored with textual content information earlier than, that cease phrase elimination is a key step in pre-processing tokenised texts. Eradicating these phrases permits us to focus the evaluation solely on the necessary components of the textual content.

To take away these phrases, we first have to outline them. Right here, I’m going to make use of a default set of cease phrases from nltk library plus a set of HTML associated phrases.

stops = set(
stopwords.phrases("english")
+ ["", "nbsp", "content", "type", "text", "charset", "iso", "qzsoft"]
)

Now, we have to discover out if these phrases exist within the tokenised array, and in the event that they do, we have to drop them. For this we’ll want to make use of the arr.eval methodology as a result of it permits us to run the Polars expressions (e.g. .is_in ) in opposition to each ingredient of the tokenised listing. Ensure that to learn the remark beneath to grasp what the every line does as this a part of the code is extra difficult.

emails_pl = emails_pl.with_columns(
# Apply to the tokenised column (it is a listing)
pl.col("email_text_tokenised")
# For each ingredient, test if it is not in a stopwords listing and solely then return it
.arr.eval(
pl.when(
(~pl.ingredient().is_in(stopwords)) & (pl.ingredient().str.n_chars() > 2)
).then(pl.ingredient())
)
# For each ingredient of a brand new listing, drop nulls (beforehand gadgets that have been in stopwords listing)
.arr.eval(pl.ingredient().drop_nulls())
.keep_name()
)

As ordinary, let’s refactor this little bit of code right into a operate for our closing pipeline.

def remove_stopwords(
information: pl.DataFrame, stopwords: set | listing, col: str
) -> pl.DataFrame:
information = information.with_columns(
pl.col(col)
.arr.eval(pl.when(~pl.ingredient().is_in(stopwords)).then(pl.ingredient()))
.arr.eval(pl.ingredient().drop_nulls())
)
return information

Whereas this sample may appear fairly difficult it’s properly price it to make use of the pre-defined str and arr expressions to optimise the efficiency.

Full Pipeline

Up to now, we’ve outlined pre-processing capabilities and noticed how they are often utilized to a single column. Polars gives a really helpful pipe methodology that permits us to chain Polars operations specified as operate. Right here’s how the ultimate pipeline appears to be like like:

emails = load_emails_txt("fradulent_emails.txt")
emails_pl = pl.DataFrame({"emails": emails})
emails_pl = (
emails_pl.pipe(extract_fields)
.pipe(email_features, "email_text")
.pipe(email_features, "sender_email")
.pipe(email_features, "topic")
.pipe(email_clean, "email_text")
.pipe(email_clean, "sender_name")
.pipe(email_clean, "topic")
.pipe(tokenise_text, "email_text")
.pipe(tokenise_text, "topic")
.pipe(remove_stopwords, stops, "email_text_tokenised")
.pipe(remove_stopwords, stops, "subject_tokenised")
)

Discover that now we are able to simply apply all of the characteristic engineering, cleansing, and tokenisation capabilities to all of the extracted columns and never simply the e-mail textual content like within the examples above.

In the event you’ve received up to now — nice job! We’ve learn in, cleaned, processed, tokenised, and did fundamental characteristic engineering on ~4k textual content data in underneath a second (not less than on my Mac M2 machine). Now, let’s benefit from the fruits of our labor and do some fundamental textual content evaluation.

To start with, let’s have a look at the phrase cloud of the e-mail texts and marvel in any respect the foolish issues we are able to discover.

# Phrase cloud operate
def generate_word_cloud(textual content: str):
wordcloud = WordCloud(
max_words=100, background_color="white", width=1600, top=800
).generate(textual content)
plt.determine(figsize=(20, 10), facecolor="okay")
plt.imshow(wordcloud)
plt.axis("off")
plt.tight_layout(pad=0)
plt.present()
# Put together information for phrase cloud
text_list = emails_pl.choose(pl.col("email_text_tokenised").arr.be a part of(" "))[
"email_text_tokenised"
].to_list()
all_emails = " ".be a part of(text_list)
generate_word_cloud(all_emails)

Financial institution accounts, subsequent of kin, safety firms, and decease kin — it’s got all of it. Let’s see how these will appear like for textual content clusters created utilizing easy TF-IDF and Ok-Means.

# TF-IDF with 500 phrases
vectorizer = TfidfVectorizer(max_features=500)
transformed_text = vectorizer.fit_transform(text_list)
tf_idf = pd.DataFrame(transformed_text.toarray(), columns=vectorizer.get_feature_names_out())
# Cluster into 5 clusters
n = 5
cluster = KMeans(n_clusters=n, n_init='auto')
clusters = cluster.fit_predict(tf_idf)
for c in vary(n):
cluster_texts = np.array(text_list)[clusters==c]
cluster_text = ' '.be a part of(listing(cluster_texts))
generate_word_cloud(cluster_text)

Under you possibly can see just a few attention-grabbing clusters that I’ve recognized:

In addition to these, I additionally discovered just a few non-sense clusters which implies that there’s nonetheless room for enhancements when it comes textual content cleansing. Nonetheless, it appears to be like like we have been capable of extract helpful clusters, so let’s name it successful. Let me know which clusters you discover!

This submit has lined all kinds of pre-processing and cleansing operations that Polars library means that you can do. We’ve seen tips on how to use Polars to:

• Extract particular patterns from texts
• Break up texts into lists primarily based on a token
• Calculate lengths and the variety of matches in texts
• Clear texts utilizing regex
• Tokenise texts and filter for cease phrases

I hope that this submit was helpful to you and also you’ll give Polars an opportunity in your subsequent NLP undertaking. Please take into account subscribing, clapping and commenting beneath.

Not a Medium Member but?

Radev, D. (2008), CLAIR assortment of fraud e-mail, ACL Information and Code Repository, ADCR2008T001, http://aclweb.org/aclwiki

Undertaking Github https://github.com/aruberts/tutorials/tree/main/metaflow/fraud_email

Polars Person Information https://pola-rs.github.io/polars-book/user-guide/