Recommender systems are fundamental to numerous personalization services. However, traditional systems often lack the ability to proactively explore user interests, explain their choices, or adapt immediately to user feedback, leaving users unsatisfied with the recommendation results. Conversational recommender systems, instead, offer a promising future with the capability of understanding both explicit textual feedback and implicit behavior patterns. Such systems accurately model user sequential interactions, proactively engage with users to explore preferences and provide explanations and context for their recommendations through conversations. The research on conversational recommender systems marks a crucial step in the evolution from traditional recommender systems to truly personalized intelligent agents.

In this dissertation, our research focuses on three pillars in developing effective conversational recommender systems: (1) Accuracy: modeling the dynamic and evolving nature of user preferences within sequential behaviors to ensure recommendations better match the user’s needs. (2) Explainability: improving the quality and expressiveness of explanations that accompany recommendations to enhance user understanding and trust. (3) Interactivity: exploring multi-round conversational capabilities to support complex recommendation scenarios, including managing bundled item recommendations and seamlessly integrating with Large Language Models for a natural-language-driven user experience.By advancing conversational recommender systems with the components for accuracy, explainability, and interactivity, this research paves the way for future research to create powerful conversational recommender systems that reshape how users discover and interact with products and services.

The amount of data available to train modern machine learning systems has been increasing rapidly, so much so that we're using, e.g., entirety of the publicly available text data to train state-of-the-art (SoTA) large language models (LLMs), interaction data from billions of users to train SoTA recommender systems, etc. Training of such large machine learning systems on such large datasets entails a high (i) computational runtime, (ii) economical cost, and (iii) carbon footprint; all of which we aim to minimize for different reasons.

While a large body of literature develops "model-centric" techniques to better model a given dataset, in this thesis, we develop a "data-centric" viewpoint, where we are interested in techniques that can appropriately summarize a given training dataset, such that models can be trained equally effectively on the data summary vs. training on the much larger original dataset. In addition to being more efficient overall, data-efficient techniques further aim to improve the trained model's quality by stripping away the low-quality and noisy sources of information in the original dataset.

More specifically, we develop techniques from two disparate data summarization ideologies: (i) data pruning (a.k.a. coreset construction) techniques that sample the most relevant portions from the dataset using various grounded heuristics, and (ii) data distillation techniques that generate synthetic data-points which summarize the underlying information in the dataset, and are optimized end-to-end using meta-learning. We restrict our scope to training (i) language models on textual datasets, and (ii) recommender systems on user-item interaction datasets.

By pushing the frontier of data-efficient training of machine learning systems, we believe our research can effectively contribute to the practical success of such widely-deployed systems, as well as provide a better understanding for the research community to build future work on.

The recent proliferation of machine learning (ML) agents that interact with humans through natural text conversation (e.g. smart phone assistants, chat bots) is predicated on the unprecedented public availability of user-contributed text (e.g. blogs, product reviews) and behavioral traces (e.g. purchases, social media interactions).Current methods for building conversational agents have seen success in highly structured fields like automated help desks and reservation booking. However, it remains challenging to apply these ML systems to help users with daily tasks in more natural and intuitive ways. For example, current recommender systems cannot fluidly engage with users for multiple rounds of conversation.

In this dissertation we focus primarily on developing technologies that allow intelligent agents to engage with users in trustworthy, personalized, and interactive ways through the medium of text.Specifically, my work focuses on 1) explainable dialog models to facilitate meaningful interviews; 2) a language modeling framework to infer user preferences from dialog; and 3) a bot-play framework for training explainable and personalized recommender systems to understand and reflect user feedback over multiple turns of conversation. I finally present two case studies on applying the aforementioned technologies to build personalized interactive agents that generate and edit instructional texts (e.g. cooking recipes) to assist users in their daily lives.

Engaging with a TV show in the age of the Internet often means avoiding show-related content for months out of fear of being spoiled. While spoiler detection research shows promising results for protecting viewers from generic spoilers, these approaches don't actually solve the problem of users avoiding show-related content during their watch. This is because what constitutes a spoiler is different depending on where a viewer is in the show, and spoiler detection on its own is too coarse to capture this complexity. Instead, we propose the task of spoiler recognition, which seeks to assign an episode number to a spoiler, given a show. We pose this task as semantic text matching and present a dataset of comments and episode summaries for evaluating model performance. The dataset consists of ~3.1K and ~2.8K manually-labeled test and validation comments respectively, and over 200K auto-labeled comments for training. We experimentally demonstrate the utility of this training set and use it to benchmark the performance of BigBird, Nyströmformer, and Longformer on this task. Specifically, we cross-encode summaries with comments and examine the mean reciprocal rank scores. Our results find Longformer to be best suited for this task. We also perform an error analysis to shed some light on the kinds of challenges spoiler recognition poses. In total, we present this dataset and these results to facilitate future research into spoiler recognition.

Product reviews are a form of user feedback that provide richer information than traditional signals in recommender systems, such as star ratings and implicit feedback.Meanwhile, a review is also a justification for the user's rating of a product. Previous works show that recommendation models can predict user ratings more accurately by jointly learning to generate reviews. The generated reviews can also serve as recommendation explanations, making the recommender system more interpretable. However, existing works evaluate these generated explanations using traditional natural language generation metrics only, overlooking trustworthiness, an important aspect of model explanations. In this thesis, we focus on two properties of trustworthy recommendation explanations: faithfulness, how truthfully do explanations reflect the decision process of the model, and factuality, whether the generated content accurately reflects the characteristics of the corresponding product. Specifically, this thesis includes two directions: (1) we propose a set of methods for evaluating the faithfulness and semantic coherency of recommendation explanations, and (2) we develop a personalized retrieval-augmented model that can generate factual and informative reviews to explain its recommendation predictions.

Teaching machines to describe visual images is one of the most long-standing challenges in the field of Machine Learning. This thesis tackles the problem of describing images via natural language: how to build machine learning models to read visual images and describe their content as well as answer relevant questions. From the application perspective, strong image captioning systems can contribute to applications such as visual question answering, dialogue systems and visual-based robotics. For a long term goal, if we can build such systems (e.g., GPT-4V and beyond), they would be a crucial step towards building Artificial General Intelligence: computers that can perceive and explore the world as humans do.

This thesis focus on neural models: building vision understanding and language generation models with deep neural networks. It mainly consists of three parts.

First, we will introduce concept bottleneck models, a class of models that build concept layers for visual understanding. We will present our work on learning a concise concept space, and follow-up applications for medical imaging to gain robustness.

In the second part of this thesis, we investigate how we can build practical image captioning systemsbased on different neural text generation architectures, from LSTM to transformers and pre-trained language models. In particular, we will cover four different tasks: 1) how we can describe the visual difference of two images; 2) how we can write medical reports given Chest X-rays to assist doctors; 3) how to generate personalized explanations for recommender systems; 4) how to augment text generation with visual imagination generated from vision diffusion models.

In the third part, we will discuss recent advances, future directions and openquestions in this field, focusing on aspects of datasets, models, and applications. We will also introduce some of our on-going attempts for these directions: for example, how to navigate phone screens and complete mobile tasks with GPT-4V.

In summary, my research contributes to the field of vision and language, specifically visual understanding via natural language, from the aspects of data curation, algorithm designing, model training, as well as various downstream applications.

Recent technological innovations (e.g. e-commerce platforms, automated retail stores) have enabled dramatic changes in people's shopping experiences, as well as the accessibility to incredible volumes of consumer-product interaction data. As a result, machine learning (ML) systems can be widely developed to help people navigate relevant information and make decisions. Traditional ML systems have achieved great success on various well-defined problems such as speech recognition and facial recognition. Unlike these tasks where datasets and objectives are clearly benchmarked, modeling consumer behavior can be rather complicated; for example, consumer activities can be affected by real-time shopping contexts, collected interaction data can be noisy and biased, interests from multiple parties (both consumers and producers) can be involved in the predictive objectives.

The primary goal of this dissertation is to address the obstacles in modeling consumer activities through computational approaches, but with careful considerations from economic and societal perspectives. Intellectually, such models help us to understand the forces that guide consumer behavior. Methodologically, I build algorithms capable of processing massive interaction datasets by connecting well-developed ML techniques and well-established economic theories. Practically, my work has applications ranging from recommender systems, e-commerce and business intelligence.

Conventional recommendation models often use the user-item interaction matrix (e.g. ratings) to predict user preferences and generate recommendations. However, it ignores abundant signals and context (e.g. visual signals or temporal context) existing in real-world applications. Moreover, efficiency becomes an essential factor when building large-scale recommendation engines. In this thesis, we seek to extend the conventional recommendation frameworks to adapt new and large-scale application scenarios. Specifically, this thesis includes three directions: (i) Visually-aware Recommendation: we extend recommendation models to visual domains. We develop CNN-based end-to-end learning approaches to make personalized image recommendations and complementary product recommendations. Moreover, beyond recommending existing products, we develop GAN-based preference models to generate new products that are preferred by users; (ii) Dynamic Recommendation: we adapt recommendation models to dynamic environments where user preferences constantly shift. We develop Markov-chain-based and self-attention-based sequential models to respond to the change of users' interests quickly and make more accurate recommendations. (iii) Efficient Recommendation: we consider both time efficiency and space efficiency, where the former seeks to optimize the serving latency, and the latter seeks to reduce the memory consumption of recommendation models.

In the rapidly evolving field of Natural Language Processing (NLP), the advent of Large Language Models (LLMs) marks a significant milestone, setting new standards in language understanding and generation. This thesis focuses on augmenting and evaluating LLMs, introducing ToolkenGPT, a novel method to integrate external tools via tool embeddings to enrich model functionality and adaptability and RepoBench, a benchmark for assessing the proficiency of LLMs in handling repository-level code auto-completion. Additionally, this thesis rethinks approaches towards tabular data reasoning, exploring how LLMs can be better tailored to understand and interpret structured data formats effectively.

Understanding and modeling heterogeneous human interaction data is essential to build next-generation natural (e.g. conversational, personalized) interactive systems. Textual data (e.g. dialogues, product reviews), relational data (e.g. knowledge bases) and behavioral data (e.g. purchases, clicks) are ubiquitous sources of data describing human interactions. Recent advances in natural language processing and human behavior modeling have led to drastic development in interactive systems as recommender systems, search engines, conversational agents, fitness trackers, etc. However, it remains challenging for such systems to provide personalized and explainable outputs, which are critical to improving the user experience.

In this thesis, we aim to study two main sources of human interaction data, i.e., text and sequences, to improve behavioral prediction, enhance model interpretability, and increase model usability. In particular, my work focuses on studying personalized models to estimate and interpret user interactions and exploring explainable models to understand and learn from user-generated text. The primary contributions are 1) a personalized text generation model which can facilitate assistive writing; 2) a personalized model that generates textual justification for modern recommender systems; 3) a personalized algorithm that models workout activity sequences, predicts user heart rate profile and provides fitness-aware recommendations; 4) an algorithm that learns to attend on essential terms in complex questions and retrieves relevant evidence to boost performance in open-domain question answering; 5) a visual-semantic embedding based method to report abnormal findings on chest X-rays.