[ad_1]
Step-by-step rationalization of using Transformer encoders to categorise textual content
Transformer is, for sure, one of the vital necessary breakthroughs within the area of deep studying. The encoder-decoder structure of this mannequin has confirmed to be highly effective in cross-domain purposes.
Initially, Transformer was used solely for language modeling duties, reminiscent of machine translation, textual content era, textual content classification, question-answering, and many others. Nonetheless, not too long ago, Transformer has additionally been used for laptop imaginative and prescient duties, reminiscent of picture classification, object detection, and semantic segmentation.
Given its reputation and the existence of quite a few Transformer-based subtle fashions reminiscent of BERT, Imaginative and prescient-Transformer, Swin-Transformer, and the GPT household, it’s essential for us to know the interior workings of the Transformer structure.
On this article, we’ll dissect solely the encoder a part of Transformer, which can be utilized primarily for classification functions. Particularly, we’ll use the Transformer encoders to categorise texts. With out additional ado, let’s first check out the dataset that we’re going to make use of on this article.
The dataset that we’re going to make use of is the e-mail dataset. You may obtain this dataset on Kaggle by way of this link. This dataset is licensed below CC0: Public Area, which implies that you need to use and distribute this dataset freely.
import math
import torch
import torch.nn as nn
import torchtext
import pandas as pd
from sklearn.model_selection import train_test_split
from torch.utils.knowledge import DataLoader
from tqdm import tqdm
from torchtext.knowledge.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iteratormachine = torch.machine("cuda" if torch.cuda.is_available() else "cpu")
df = pd.read_csv('spam_ham.csv')
df_train, df_test = train_test_split(df, test_size=0.2, random_state=42)
print(df_train.head())
# Output
'''
Class Message
1978 spam Reply to win £100 weekly! The place will the 2006 ...
3989 ham Good day. Type of out on the town already. That . So ...
3935 ham How come guoyang go n inform her? Then u instructed her?
4078…
[ad_2]
Source link
