[ad_1]
That is an up to date model of a chunk initially posted on the author’s blog.
Launched in 2018, BERT has shortly turn out to be one of the crucial standard fashions for pure language processing, with the unique paper accumulating tens of hundreds of citations. Its success recipe lies in a easy coaching methodology referred to as masked language modeling. Randomly masks ~15% of the phrases in a textual content, after which ask the mannequin to get better these based mostly on the encircling context. This easy thought, scaled on billions of phrases, allowed BERT to develop a deep understanding of human language, reusable for different duties equivalent to query answering, textual content summarization, classification of textual content paperwork, and many others. Given the spectacular success of BERT for written language, it’s no shock that researchers try to use the identical recipe to different modalities of language, like human speech.
May we substitute the textual content enter in BERT with a speech sequence, masks part of it, and equally practice the mannequin to get better what’s lacking?
Sadly, it’s not as easy as one would possibly assume. Written language may be naturally damaged down into phrases (or sub-words) from a finite vocabulary of discrete models. Speech, initially a sound wave, is recorded as a sequence of steady values and doesn’t present such pure sub-units. In concept, we might use phonetic techniques like phones to discretize speech, however this may first require people to label your complete dataset beforehand stopping us from pre-training on a big amount of unlabeled information as BERT does.
On this article, we are going to introduce the wav2vec 2.0 and HuBERT fashions. They each supply totally different options to the above downside. They each:
- Rework the unique steady acoustic information right into a sequence of discrete models.
- Masks a random half of the sequence and practice the mannequin to get better it.
- Use a BERT-like Transformer encoder to mannequin the context of the speech sequence.
Nevertheless, they do these items in extensively alternative ways. We are going to take a look at each fashions intimately to know all of the subtleties and design decisions that come into play when making use of a BERT-like method to speech or acoustic information.
This mannequin from Meta AI is a sequel to the first version, initially manufactured from two massive convolutional networks referred to as the encoder and context community. Model 2.0 replaces the context community with the Transformer-based BERT structure. The core thought behind wav2vec 2.0 is to show the mannequin to do two issues in parallel:
- Quantize steady speech information into discrete models robotically.
- Guess the proper discrete models (discovered from step 1) at randomly masked areas.
The problem of doing these two issues collectively is learn how to stop the mannequin from dishonest. For the reason that mannequin each chooses the bottom fact targets (step 1) and guesses these targets (step 2), it might collapse right into a naive method with out studying something concerning the construction of spoken language. For instance, the mannequin might assign the identical goal to each a part of the speech sequence, after which all the time predict this goal. Wav2vec 2.0 introduces a set of ways to mitigate this threat. Certainly one of these replaces BERT’s unique cross-entropy loss with a contrastive loss. We are going to look into the small print of the coaching loss, however first, let’s take a step again and take a look at the massive image.
Above is an summary of the wav2vec 2.0 structure and its pre-training course of. There are 4 vital parts on this diagram: the characteristic encoder, context community, quantization module, and a contrastive loss (pre-training goal). We are going to open the hood and look intimately at every one.
Function encoder
The characteristic encoder’s job is to cut back the dimensionality of the audio information, changing the uncooked waveform right into a sequence of characteristic vectors Z0, Z1, Z2, …, ZT each 20 milliseconds. Its structure is straightforward: a 7-layer convolutional neural community (single-dimensional) with 512 channels at every layer.
The waveform is normalized earlier than being despatched to the community, and the kernel width and strides of the convolutional layers lower as we get increased within the community. The characteristic encoder has a complete receptive area of 400 samples or 25 ms of audio (audio information is encoded at a pattern price of 16 kHz).
Quantization module
Wav2vec 2.0 proposes to robotically study discrete speech models, by sampling from a Gumbel-Softmax distribution. Doable models are manufactured from codewords sampled from codebooks (teams). Codewords are then concatenated to kind the ultimate speech unit. Wav2vec makes use of 2 teams with 320 attainable phrases in every group, therefore a theoretical most of 320 x 320 = 102,400 speech models.
The latent options are multiplied by the quantization matrix to provide the logits: one rating for every of the attainable codewords in every codebook. The Gumbel-Softmax trick permits sampling a single codeword from every codebook, after changing these logits into possibilities. It’s just like taking the argmax, besides that the operation is absolutely differentiable. Furthermore, a small randomness impact, whose impact is managed by a temperature argument, is launched to the sampling course of to facilitate coaching and codewords utilization.
Context community
The core of wav2vec 2.0 is its Transformer encoder, which takes as enter the latent characteristic vectors and processes it by 12 Transformer blocks for the BASE model of the mannequin, or 24 blocks for the LARGE model. To match the interior dimension of the Transformer encoder, the enter sequence first must undergo a characteristic projection layer to extend the dimension from 512 (output of the CNN) to 768 for BASE or 1,024 for LARGE. I can’t describe additional the Transformer structure right here and invite you to learn The Illustrated Transformer for the small print.
One distinction from the unique Transformer structure is how positional data is added to the enter. For the reason that self-attention operation of the Transformer doesn’t protect the order of the enter sequence, fastened pre-generated positional embeddings have been added to the enter vectors within the unique implementation. The wav2vec mannequin as a substitute makes use of a grouped convolution layer to study relative positional embeddings by itself.
Pre-training & contrastive loss
The pre-training course of makes use of a contrastive activity to coach on unlabeled speech information. A masks is first randomly utilized within the latent house to ~50% of the projected latent characteristic vectors. Masked positions are then changed by the identical educated vector Z’M earlier than being fed to the Transformer community.
The ultimate context vectors then undergo the final projection layer to match the dimension of the quantized speech models Qt. For every masked place, 100 unfavourable distractors are uniformly sampled from different positions in the identical sentence. The mannequin then compares the similarity (cosine similarity) between the projected context vector C’t and the true constructive goal Qp together with all unfavourable distractors Qñ. The contrastive loss then encourages excessive similarity with the true constructive goal and penalizes excessive similarity scores with unfavourable distractors.
Variety loss
Throughout pre-training, one other loss is added to the contrastive loss to encourage the mannequin to make use of all codewords equally usually. This works by maximizing the entropy of the Gumbel-Softmax distribution, stopping the mannequin from all the time selecting from a small sub-group of all accessible codebook entries. You could find extra particulars within the original paper.
Positive-tuning and downstream duties
This concludes our tour of wav2vec 2.0 and its pre-training course of. The ensuing pre-trained mannequin can then be used for a wide range of speech downstream duties: automated speech recognition, emotion detection, speaker recognition, language detection, and many others. In wav2vec 2.0’s original paper, the authors immediately fine-tuned the mannequin for speech recognition with a CTC loss, including a linear projection on prime of the context community to foretell a phrase token at every timestep. They demonstrated that fine-tuning the mannequin this manner on just one hour of labeled speech information might beat the earlier state-of-the-art techniques educated on 100 instances extra labeled information.
Now let’s take a look at our second mannequin. HuBERT’s predominant thought is to find discrete hidden models (the Hu within the title) to remodel speech information right into a extra “language-like” construction. These hidden models may very well be in comparison with phrases or tokens in a textual content sentence.
The tactic attracts inspiration from the DeepCluster paper in pc imaginative and prescient, the place photographs are assigned to a given variety of clusters earlier than re-using these clusters as “pseudo-labels” for coaching the mannequin in a self-supervised means. In HuBERT’s case, clustering just isn’t utilized on photographs however on quick audio segments (25 milliseconds), and the ensuing clusters turn out to be the hidden models that the mannequin can be educated to foretell.
Variations with wav2vec 2.0
At first look, HuBERT seems similar to wav2vec 2.0: each fashions use the identical convolutional community adopted by a transformer encoder. Nevertheless, their coaching processes are very totally different, and HuBERT’s efficiency, when fine-tuned for automated speech recognition, both matches or improves upon wav2vec 2.0. Listed below are the important thing variations to bear in mind:
HuBERT makes use of the cross-entropy loss, as a substitute of the extra complicated mixture of contrastive loss + variety loss utilized by wav2vec 2.0. This makes coaching simpler and extra steady since this is identical loss that was used within the original BERT paper.
HuBERT builds targets through a separate clustering course of, whereas wav2vec 2.0 learns its targets concurrently whereas coaching the mannequin (through a quantization course of utilizing Gumbel-softmax). Whereas wav2vec 2.0 coaching might appear less complicated because it consists of solely a single step, in follow, it may possibly turn out to be extra complicated because the temperature of the Gumbel-softmax should be fastidiously adjusted throughout coaching to forestall the mannequin from “dishonest” and sticking to a small subset of all accessible targets.
HuBERT re-uses embeddings from the BERT encoder to enhance targets, whereas wav2vec 2.0 solely makes use of the output of the convolutional community for quantization. Within the HuBERT paper, the authors present that utilizing such embeddings from intermediate layers of the BERT encoder results in higher targets high quality than utilizing the CNN output.
When it comes to mannequin structure, the BASE and LARGE variations of HuBERT have the identical configuration because the BASE and LARGE variations of wav2vec 2.0 (95 million and 317 million parameters respectively). Nevertheless, an X-LARGE model of HuBERT can be used with twice as many transformer layers as within the LARGE model, with virtually 1 billion parameters.
Coaching course of
The coaching course of alternates between two steps: a clustering step to create pseudo-targets, and a prediction step the place the mannequin tries to guess these targets at masked positions.
Step 1: Uncover “hidden models” targets by clustering
Step one is to extract the hidden models (pseudo-targets) from the uncooked waveform of the audio. The K-means algorithm is used to assign every phase of audio (25 milliseconds) to certainly one of Ok clusters. Every recognized cluster will then turn out to be a hidden unit, and all audio frames assigned to this cluster can be assigned with this unit label. Every hidden unit is then mapped to its corresponding embedding vector that can be utilized throughout the second step to make predictions.
A very powerful resolution for clustering is into which options to remodel the waveform for clustering. Mel-Frequency Cepstral Coefficients (MFCCs) are used for the primary clustering step, as these options have been proven to be comparatively environment friendly for speech processing. Nevertheless, for subsequent clustering steps, representations from an intermediate layer of the HuBERT transformer encoder (from the earlier iteration) are re-used.
Extra exactly, the sixth transformer layer is used for clustering throughout the second iteration of the HuBERT BASE mannequin (the BASE mannequin is barely educated for 2 iterations in complete). Moreover, HuBERT LARGE and X-LARGE are educated for a 3rd iteration by re-using the ninth transformer layer from the second iteration of the BASE mannequin.
Combining clustering of various sizes
The authors additionally experiment with a mixture of a number of clustering with a unique variety of clusters to seize targets of various granularity ( vowel/consonant vs sub-phone states for instance). They present that utilizing cluster ensembles can enhance efficiency by a small margin. You could find extra particulars on this within the original paper.
Step 2: Predict noisy targets from the context
The second step is identical as for the unique BERT: coaching with the masked language modeling goal. Round 50% of transformer encoder enter options are masked, and the mannequin is requested to foretell the targets for these positions. For this, the cosine similarity is computed between the transformer outputs (projected to a decrease dimension) and every hidden unit embedding from all attainable hidden models to provide prediction logits. The cross-entropy loss is then used to penalize flawed predictions.
The loss is barely utilized to the masked positions because it has been proven to carry out higher when utilizing noisy labels. The authors show this experimentally by making an attempt to foretell solely masked targets, solely un-masked ones, or each collectively.
Like wav2vec 2.0 earlier than it, HuBERT representations may be reused to enhance efficiency on many downstream duties equivalent to automated speech recognition. Whereas these two fashions are just one a part of all BERT-like approaches for speech illustration, they provide an awesome introduction to the present state-of-the-art. Certainly, most of at the moment’s massive Transformer-based fashions for speech use comparable ideas and concepts.
Extra usually, the ideas behind each Wav2vec 2.0 and HuBERT open the door to the broader use of NLP algorithms for speech (not simply BERT), initially designed just for textual content sequence. Many of those NLP approaches allow environment friendly coaching on unlabeled information.
Coaching on unlabeled acoustic information permits scaling up speech fashions to very massive sizes. Whereas present mannequin sizes aren’t but corresponding to what we see for text-based fashions like GPT-3 (175 billion parameters), they’re quickly catching up. Increasingly large-scale speech datasets are being launched, like Common Voice, recorded in numerous languages. These will assist practice higher and extra resilient speech fashions that generalize properly in all situations and environments, a key problem for speech processing.
Writer Bio
Jonathan Boigne is a machine studying engineer specializing in pure language processing and speech, which he often writes about on his weblog at jonathanbgn.com. His core pursuits embody self-supervised studying, switch studying, and learn how to leverage large-scale unlabeled information to enhance efficiency on real-world issues the place labels are scarce.
[ad_2]
Source link