Proposal: specials revamp

[mttk]

Issues with current Specials:

• User can't modify the string value of the special
• Text can't be printed out with join after reverse numericalize due to enum type (common use case)

original_text = ' '.join(vocab.reverse_numericalize(batch_x.text)) # itos contains both `str` and `SpecialVocabSymbols`

Idea:
Make Specials subclass str. Inheritance from Special base class is an identifier that a string is a Special. Each Special has a method that knows how to apply it to a token and/or sequence. Example:

class EOS(Special):
    def apply(self, sequence_or_token):
        # Core special, handled by Vocab
        if type(sequence_or_token) is str:
            raise ValueError("EOS can only be applied to sequence")
        elif type(sequence_or_token) is list:
            # Extend with self
            return sequence_or_token + [self.data]

class UNK(Special):
    def apply(self, sequence_or_token):
        # Core special, handled by Vocab
        pass

class Special(str):
    @abc.abstractmethod
    def apply(self, sequence_or_token):
        # Method is used ONLY in Vocab.numericalize
        if type(sequence_or_token) is str:
            # Apply to token
            pass
        elif type(sequence_or_token) in (list, tuple):
            # Apply to sequence
            pass

This allows us to:

eos = EOS('<eos>')
sequence = ['this', 'is', 'a', 'sequence']
print(' '.join(eos.apply(sequence)))
>>> this is a sequence <eos>

So the user can define (1) the string for the special and (2) is handled.
Up for discussion:

1. Do we introduce an inheritance in specials? A natural inheritance is

• CoreSpecial (PAD, UNK -- behavior hardcoded in vocab, apply isn't used)
• TokenSpecial (Applied on token-level for efficiency. Example: substituting numbers with or masking tokens can be a special instead of a hook)
• SequenceSpecial (anything that works on sequence level: EOS, BOS, maybe MASK)

2. Referencing specials

• The Vocab needs to find the core specials in order to provide them to Field (e.g. for padding)
• The Vocab.padding_index method has to check for if PAD in self.stoi/self.specials (TBD: maybe make list of specials an attribute of vocab)
  • Proposal (maybe bad): make __hash__ and __equals__ of specials trigger on concrete class, and not string
    • Required: there can be only one of each special in the Vocab (natural)
    • Checking for if PAD in stoi would essentially check if stoi[idx] == PAD.__class__ instead of == str(PAD) (illustrative)
• Alternative:
  - Check

for special in self.specials:
  if type(special) is PAD:
    return self.stoi[special]

- Requires storing all specials as attribute (probably nicer)

[mttk]

@FilipBolt @ivansmokovic @mariosasko

[FilipBolt]

I agree with both problems listed above and agree with the solution.

I do propose slight modifications:
using __add__ instead of apply, such that you could use specials and concatenate them as you would regular strings/lists:

eos = EOS('<eos>')
sequence = ['this', 'is', 'a', 'sequence']
print(' '.join(eos + sequence)))

Questions:

Do we introduce an inheritance in specials?

I like the idea of inputting NLP domain knowledge into SpecialsDesign and enforcing some things, such that EOS can only be applied to sequences, etc. Do note, that this does require extra docs and explanations. Can you just explain a bit TokenLevel and the efficiency part? One thing I'm not a fan of is that having classes implies having different instances, which isn't the case where. I don't see a case where you will need two instances of EOS.

Referencing specials

I think how we reference specials is key here. Why do you propose to override the __eq__ for string. Wouldn't using a string be good enough by default? You could just use if stoi[idx] == PAD() in that case, or do I misunderstand something?

[mttk]

I like the __add__ idea.

Re inheritance: sequence level specials have to be applied to the sequence (prepend, append, etc). That's fine. Token-level specials (e.g. <MASK>) are applied to single tokens. The difference is if you have multiple token-level specials (TBH I can't think of multiple so this might be overengineering) you can instead of:

for special in specials:
  special.apply(sequence)

do this:

for token in sequence:
  for special in token_specials:
    special.apply(token)

for a bit more efficiency.

Re referencing: if the user defines a different string for PAD (e.g. specials = (PAD('<custom_pad>)),), we cannot retrieve its concrete string (except by storing upon Vocab construction by iterating over specials), but we do know that (1) there should only be one padding symbol and that (2) its concrete string representation doesn't really matter.