Ph.D: Thesis Defense: Online: CDS: 15, March 2023 “Methods for Improving Data-efficiency and Trustworthiness using Natural Language Supervision”

Traditional strategies to build machine learning based classification  systems employ discrete labels as targets. This limits the usefulness of  such systems in two ways. First, the generalizability of these systems  is limited to labels present and well represented in the training data.  Second, with increasingly larger neural network models gaining  acceptability, supervision with discrete labels alone does not lead to a  straightforward interface for generating explanations for the decisions  taken by such systems. Natural Language (NL) Supervision (NLS), in the  form of task descriptions, examples, label descriptions and explanations  for labelling decisions, provides a way to overcome these bottlenecks.  Working in this paradigm, we propose novel methods for improving  data-efficiency and trustworthiness:

(1) Data Efficiency using NLS: Word Sense Disambiguation (WSD) using  Sense Definition Embeddings

WSD, a long-standing open problem in Natural Language Processing (NLP),  typically presents itself with small training corpora with long tails of  label distributions. Existing supervised methods didn’t generalize well  to rare or unseen classes while NL supervision based systems did worse  on overall (standard) evaluation benchmarks. We propose Extended WSD  Incorporating Sense Embeddings (EWISE), a supervised model to perform  WSD by predicting over a continuous sense embedding space as opposed to  a discrete label space. This allows EWISE to generalize over both seen  and unseen senses, thus achieving generalized zero-shot learning. To  obtain target sense embeddings, EWISE utilizes NL sense definitions  along with external knowledge in WordNet relations. EWISE achieved new  state-of-the-art WSD performance at the time of publication,  specifically by improving on zero-shot and few-shot learning.

 (2) Trustworthiness using NLS: Natural Language Inference (NLI) with  Faithful NL Explanations

Generated NL explanations are expected to be faithful, i.e., they should  correlate well with the model’s internal decision making. In this work,  we focus on the task of NLI and address the following question: can we  build NLI systems which produce labels with high accuracy, while also  generating faithful explanations of its decisions? We propose  Natural-language Inference over Label-specific Explanations (NILE), a  novel NLI method which utilizes auto-generated label-specific NL  explanations to produce a label along with its faithful explanation. Our  evaluation of NILE also supports the claim that accurate systems capable  of providing testable explanations of their decisions can be designed.

(3) Improving the NLS interface of Large Language Models (LLM)

 LLMs, pre-trained on unsupervised corpora, have proven to be successful  as zero-shot and few-shot learners on downstream tasks using only a  textual interface. This enables a promising NLS interface. A typical  usage involves augmenting an input example along with some priming text  comprising of task descriptions and training examples and processing the  output probabilities to make predictions. In this work, we further  explore priming-based few-shot learning and make the following  contributions:

(a) Reordering Examples Helps during Priming-based Few-Shot Learning: We  show that presenting training examples in the right order is key for  generalization. We introduce PERO (Prompting with Examples in the Right  Order), where we formulate few-shot learning as search over the set of  permutations of the training examples. We demonstrate the effectiveness  of the proposed method on the tasks of sentiment classification, natural  language inference and fact retrieval. We show that PERO can learn to  generalize efficiently using as few as 10 examples, in contrast to  existing approaches.

 (b) Answer-level Calibration (ALC) Helps Free-form Multiple Choice  Question Answering (QA): We consider the QA format, where we need to  choose from a set of free-form textual choices of unspecified lengths,  given a context. We present ALC, where our main suggestion is to model  context-independent biases in terms of the probability of a choice  without the associated context and to subsequently remove these biases  using an unsupervised estimate of similarity with the full context. ALC  improves zero-shot and few-shot performance on several benchmarks while  also providing a more reliable estimate of performance.