Vanilla RNNs for Sentence Classification

In our usecase, we will use the above vanilla RNN to create a “Sentence Encoding” i.e. a vector of values that represent the given sentence and then pass that vector through another linear layer to make our final binary prediction of whether the given tweets is about a real disaster or not.

The network architecture looks as follows.

import pytorch_lightning as pl

class DisasterTweetsClassifierRNN(pl.LightningModule):

    def __init__(self, rnn_hidden_size, num_classes,
                 dropout_p, pretrained_embeddings, learning_rate,
                 num_layers, bidirectional, aggregate_hiddens, aggregation_func='max'):
        super().__init__()

        self.save_hyperparameters('num_classes', 'dropout_p', 'learning_rate',
                                  'rnn_hidden_size', 'num_layers', 'bidirectional',
                                  'aggregate_hiddens', 'aggregation_func'
                                  )

        embedding_dim = pretrained_embeddings.size(1)
        num_embeddings = pretrained_embeddings.size(0)

        self.emb = torch.nn.Embedding(embedding_dim=embedding_dim,
                                      num_embeddings=num_embeddings,
                                      padding_idx=0,
                                      _weight=pretrained_embeddings)

        self.rnn = torch.nn.RNN(embedding_dim, rnn_hidden_size, num_layers=num_layers, bidirectional=bidirectional)
        rnn_output_size = rnn_hidden_size
        if bidirectional:
            rnn_output_size = rnn_hidden_size * 2
        self._dropout_p = dropout_p
        self.fc1 = torch.nn.Linear(rnn_output_size, rnn_hidden_size)
        self.fc2 = torch.nn.Linear(rnn_hidden_size, num_classes)

        self.loss = torch.nn.CrossEntropyLoss(reduction='none')

• We are logging the hyperparameters of our network and initializing our embedding layer with the pretrained embeddings.
• We then create our vanilla RNN object with the required parameters.
• One parameter that gets passed to the RNN object is the number of layers. In order to generate more predictive power we can stack multiple RNNs on top of each other, this parameter configures the number of those stacked RNN layers.
• We also pass the bidirectional flag to the RNN object. Like we discussed earlier, RNN processes input sentence sequentially, we can process the sentence from beginning to end or end to beginning. Usually, processing from both the directions and using the concatenated vector of both directions leads to higher predictive power.
• Finally we have 2 linear layers which transform the RNN output into our 2 layer output which can be used to calculate cross entropy loss.

Gated Recurrent Units (GRU) for Sentence Classification

Vanilla RNNs suffer from the problem of Vanishing Gradients. In simple words, the basic RNNs discussed above are more impacted by near word effects instead of words which are far away. So, for sentences with longer length, at each RNN step, the model is only able to carry information from the previous few words and not from the words which are farther away but may have more relevance to the word in current step.
The issue mainly arises because we completely modify hidden state and weights at each step based on the currently incoming word and the hidden state vector becomes an information bottleneck. GRUs try to fix this problem by controlling how much of the previous hidden vector is forgotten and how much of it is kept/updated. Another popular variant to fix this problem is the LSTM (Long Short Term Memory) network which tries to keep a separate vector (cell state) as “memory” to hold information over long sequences. However, I am using GRU in my work in favour of limited compute power.

This lecture by Abigail See at Stanford is the best resource IMO to understand the issues with Vanilla RNNs and how GRUs and LSTMs work to try and resolve them.

For our purposes, we create the GRU network as follows -

import pytorch_lightning as pl

class DisasterTweetsClassifierGRU(DisasterTweetsClassifierRNN):
    def __init__(self, rnn_hidden_size, num_classes,
                 dropout_p, pretrained_embeddings, learning_rate,
                 num_layers, bidirectional, aggregate_hiddens, aggregation_func='max'):
        super().__init__(rnn_hidden_size, num_classes,
                         dropout_p, pretrained_embeddings, learning_rate,
                         num_layers, bidirectional, aggregate_hiddens, aggregation_func=aggregation_func)

        embedding_dim = pretrained_embeddings.size(1)
        self.rnn = torch.nn.GRU(embedding_dim, rnn_hidden_size, num_layers=num_layers, bidirectional=bidirectional)

• Note that this class has the vanilla RNN class as a parent so it borrows almost all the implementation from there.
• The only difference in this GRU network is the usage of torch.nn.GRU instead of torch.nn.RNN in order to process the input sentences.