TAPI Lab

Discovering and Controlling for Latent Confounds in Text Classification Using Adversarial Domain Adaptation

2019-04-01T00:00:00+00:00

Abstract

In text classification, the testing data often systematically differ from the training data, a problem called dataset shift. In this paper, we investigate a type of dataset shift we call confounding shift. Such a setting exists when two conditions are met: (a) there is a confound variable Z that influences both text features X and class label Y ; (b) the relationship between Z and Y changes from training to testing. While recent work in this area has required confounds to be known ahead of time, this is unrealistic for many settings. To address this shortcoming, we propose a method both to discover and to control for potential confounds. The approach first uses neural network-based topic modeling to discover potential confounds that differ between training and testing data, then uses adversarial training to fit a classification model that is invariant to these discovered confounds. We find the resulting method to improve over state-of-the-art domain adaptation method, while also producing results that are competitive with those obtained when confounds are known ahead of time.

Citation

@InProceedings{landeiro19discovering,
  author =       {Virgile Landeiro and Tuan Tran and Aron Culotta},
  title =        {Discovering and Controlling for Latent Confounds in Text Classification Using Adversarial Domain Adaptation},
  booktitle = {Proceedings of the SIAM International Conference on Data Mining (SDM19)},
  year =         2019,
}

When do Words Matter? Understanding the Impact of Lexical Choice on Audience Perception using Individual Treatment Effect Estimation

2019-01-01T00:00:00+00:00

Abstract

Studies across many disciplines have shown that lexical choice can affect audience perception. For example, how users describe themselves in a social media profile can affect their perceived socio-economic status. However, we lack general methods for estimating the causal effect of lexical choice on the perception of a specific sentence. While randomized controlled trials may provide good estimates, they do not scale to the potentially millions of comparisons necessary to consider all lexical choices. Instead, in this paper, we first offer two classes of methods to estimate the effect on perception of changing one word to another in a given sentence. The first class of algorithms builds upon quasi-experimental designs to estimate individual treatment effects from observational data. The second class treats treatment effect estimation as a classification problem. We conduct experiments with three data sources (Yelp, Twitter, and Airbnb), finding that the algorithmic estimates align well with those produced by randomized-control trials. Additionally, we find that it is possible to transfer treatment effect classifiers across domains and still maintain high accuracy.

Citation

@InProceedings{zhang2019words,
  author =       {Zhao Wang and Aron Culotta},
  title =        {When do words matter? {U}nderstanding the Impact of Lexical Choice on Audience Perception using Individual Treatment Effect Estimation},
  booktitle = {Proceedings of the Thirty Third National Conference on Artificial Intelligence (AAAI 2019)},
  year =         2019,
}

Forecasting the presence and intensity of hostility on Instagram using linguistic and social features

2018-06-01T00:00:00+00:00

Abstract

Online antisocial behavior, such as cyberbullying, harassment, and trolling, is a widespread problem that threatens free discussion and has negative physical and mental health consequences for victims and communities. While prior work has proposed automated methods to identify hostile comments in online discussions, these methods work retrospectively on comments that have already been posted, making it difficult to intervene before an interaction escalates. In this paper we instead consider the problem of forecasting future hostilities in online discussions, which we decompose into two tasks: (1) given an initial sequence of non-hostile comments in a discussion, predict whether some future comment will contain hostility; and (2) given the first hostile comment in a discussion, predict whether this will lead to an escalation of hostility in subsequent comments. Thus, we aim to forecast both the presence and intensity of hostile comments based on linguistic and social features from earlier comments. To evaluate our approach, we introduce a corpus of over 30K annotated Instagram comments from over 1,100 posts. Our approach is able to predict the appearance of a hostile comment on an Instagram post ten or more hours in the future with an AUC of .82 (task 1), and can furthermore distinguish between high and low levels of future hostility with an AUC of .91 (task 2).

Citation

@InProceedings{liu2018forecasting,
  author =       {Ping Liu and Joshua Guberman and Libby Hemphill and {\bf Aron Culotta},
  title =        {Forecasting the presence and intensity of hostility on Instagram using linguistic and social features},
  booktitle = {Proceedings of the Twelfth International AAAI Conference on Web and Social Media (ICWSM 2018)},
  year =         2018,
}

Robust Text Classification under Confounding Shift

2018-04-01T00:00:00+00:00

Abstract

As statistical classifiers become integrated into real-world applications, it is important to consider not only their accuracy but also their robustness to changes in the data distribution. Although identifying and controlling for confounding variables Z – correlated with both the input X of a classifier and its output Y – has been assiduously studied in empirical social science, it is often neglected in text classification. This can be understood by the fact that, if we assume that the impact of confounding variables does not change between the time we fit a model and the time we use it, then prediction accuracy should only be slightly affected. We show in this paper that this assumption often does not hold and that when the influence of a confounding variable changes from training time to pre- diction time (i.e. under confounding shift), the classifier accuracy can degrade rapidly. We use Pearl’s back-door adjustment as a predictive framework to develop a model robust to confounding shift under the condition that Z is observed at training time. Our approach does not make any causal conclusions but by experimenting on 6 datasets, we show that our approach is able to outperform baselines 1) in controlled cases where confounding shift is manually injected between fitting time and prediction time 2) in natural experiments where confounding shift appears either abruptly or gradually 3) in cases where there is one or multiple confounders. Finally, we discuss multiple issues we encountered during this research such as the effect of noise in the observation of Z and the importance of only controlling for confounding variables.

Citation

@article{virgile2018reducing,
  author = {Virgile Landeiro and Aron Culotta},
  title = {Robust Text Classification under Confounding Shift},
  journal = {Journal of Artificial Intelligence},
  year = {2018},
  volume = {63},
}

Learning from noisy label proportions for classifying online social data

2018-01-01T00:00:00+00:00

Abstract

Inferring latent attributes (e.g., demographics) of social media users is important to improve the accuracy and validity of social media analysis methods. While most existing approaches use either heuristics or supervised classification, recent work has shown that accurate classification models can be trained using supervision from population statistics. These learning with label proportion (LLP) models are ft on bags of instances and then applied to individual accounts. However, it is well known that many social media sites such as Twitter are not a representative sample of the population; thus, there are many sources of noise in these label proportions (e.g., sampling bias). This can in turn degrade the quality of the resulting model. In this paper, we investigate classification algorithms that use population statistical constraints such as demographics, names, and social network followers to ft classifiers to predict individual user attributes. We propose LLP methods that explicitly model the noise inherent in these label proportions. On several real and synthetic datasets, we find that combining these enhancements together can significantly reduce averaged classification error by 7%, resulting in methods that are robust to noise in the provided label proportions.

Citation

@Article{ehsan2017learning,
author={Ehsan Mohammady Ardehaly and  Aron Culotta},
title="Learning from noisy label proportions for classifying online social data",
journal="Social Network Analysis and Mining",
year="2018",
volume="8",
number="1",
pages="2--22",
issn="1869-5469",
doi="10.1007/s13278-017-0478-6",
}

Mining the Demographics of Political Sentiment from Twitter Using Learning from Label Proportions

2017-08-01T00:00:00+00:00

Abstract

Opinion mining and demographic attribute inference have many applications in social science. In this paper, we propose models to infer daily joint probabilities of multiple latent attributes from Twitter data, such as political sentiment and demographic attributes. Since it is costly and time-consuming to annotate data for traditional supervised classification, we instead propose scalable Learning from Label Proportions (LLP) models for demographic and opinion inference using U.S. Census, national and state political polls, and Cook partisan voting index as population level data. In LLP classification settings, the training data is divided into a set of unlabeled bags, where only the label distribution in of each bag is known, removing the requirement of instance-level annotations. Our proposed LLP model, Weighted Label Regularization (WLR), provides a scalable generalization of prior work on label regularization to support weights for samples inside bags, which is applicable in this setting where bags are arranged hierarchically (e.g., county level bags are nested inside of state-level bags). We apply our model to Twitter data collected in the year leading up to the 2016 U.S. presidential election, producing estimates of the relationships among political sentiment and demographics over time and place. We find that our approach closely tracks traditional polling data stratified by demographic category, resulting in error reductions of 28-44% over baseline approaches. We also provide descriptive evaluations showing how the model may be used to estimate interactions among many variables and to identify linguistic temporal variation, capabilities which are typically not feasible using traditional polling methods.

Citation

@InProceedings{ehsan2017mining,
  author =       {Ehsan Ardehaly and Aron Culotta},
  title =        {Mining the Demographics of Political Sentiment from {T}witter Using Learning from Label Proportions},
  booktitle = {Proceedings of the 17th {IEEE} International Conference on Data Mining (ICDM)},
  year =         2017,
}

Co-training for Demographic Classification Using Deep Learning from Label Proportions

2017-08-01T00:00:00+00:00

Abstract

Deep learning algorithms have recently produced state-of-the-art accuracy in many classification tasks, but this success is typically dependent on access to many annotated training examples. For domains without such data, an attractive alternative is to train models with light, or distant supervision. In this paper, we introduce a deep neural network for the Learning from Label Proportion (LLP) setting, in which the training data consist of bags of unlabeled instances with associated label distributions for each bag. We introduce a new regularization layer, Batch Averager, that can be appended to the last layer of any deep neural network to convert it from supervised learning to LLP. This layer can be implemented readily with existing deep learning packages. To further support domains in which the data consist of two conditionally independent feature views (e.g. image and text), we propose a co-training algorithm that iteratively generates pseudo bags and refits the deep LLP model to improve classification accuracy. We demonstrate our models on demographic attribute classification (gender and race/ethnicity), which has many applications in social media analysis, public health, and marketing. We conduct experiments to predict demographics of Twitter users based on their tweets and profile image, without requiring any user-level annotations for training. We find that the deep LLP approach outperforms baselines for both text and image features separately. Additionally, we find that co-training algorithm improves image and text classification by 4% and 8% absolute F1, respectively. Finally, an ensemble of text and image classifiers further improves the absolute F1 measure by 4% on average.

Citation

@InProceedings{ehsan2017training,
  author =       {Ehsan Ardehaly and Aron Culotta},
  title =        {Co-training for Demographic Classification Using Deep Learning from Label Proportions},
  booktitle = {Proceedings of the ACUMEN Workshop at the 17th {IEEE} International Conference on Data Mining (ICDM)},
  year =         2017,
}

Are Words Commensurate with Actions? Quantifying Commitment to a Cause from Online Public Messaging

2017-06-01T00:00:00+00:00

Abstract

Public entities such as companies and politicians increasingly use online social networks to communicate directly with their constituencies. Often, this public messaging is aimed at aligning the entity with a particular cause or issue, such as the environment or public health. However, as a consumer or voter, it can be difficult to assess an entity’s true commitment to a cause based on public messaging. In this paper, we present a text classification approach to categorize a message according to its commitment level toward a cause. We then compare the volume of such messages with external ratings based on entities’ actions (e.g., a politician’s voting record with respect to the environment or a company’s rating from environmental non-profits). We find that by distinguishing between low- and high- level commitment messages, we can more reliably identify truly committed entities. Furthermore, by measuring the discrepancy between classified messages and external ratings, we can identify entities whose public messaging does not align with their actions, thereby providing a methodology to identify potentially “inauthentic” messaging campaigns.

Citation

@InProceedings{wang2017words,
  author =       {Zhao Wang and Jennifer Cutler and Aron Culotta},
  title =        {Are Words Commensurate with Actions? {Q}uantifying Commitment to a Cause from Online Public Messaging},
  booktitle = {Proceedings of the ACUMEN Workshop at the 17th {IEEE} International Conference on Data Mining (ICDM)},
  year =         2017,
}

Controlling for Unobserved Confounds in Classification Using Correlational Constraints

2017-05-01T00:00:00+00:00

Abstract

As classifiers become integrated into real-world applications, it is important to consider not only their accuracy but also their robustness to changes in the data distribution. We consider the case where there is an unobserved confounding variable $z$ that influences both the features $x$ and the class variable $y$. When the influence of $z$ changes from training to testing data, classifier accuracy can degrade rapidly. In our approach, we assume that we can predict the value of $z$ at training time with some error. The prediction for $z$ is then fed to Pearl’s back-door adjustment to build our model. Because of the attenuation bias caused by measurement error in $z$, standard approaches to controlling for $z$ are ineffective. In response, we propose a method to properly control for the influence of $z$ by first estimating its relationship with the class variable $y$, then updating predictions for $z$ to match that estimated relationship. By adjusting the influence of $z$, we show that we can build a model that exceeds competing baselines on accuracy as well as on robustness over a range of confounding relationships.

Citation

@InProceedings{virgile2017controlling,
  author =       {Virgile Landeiro and Aron Culotta},
  title =        {Controlling for Unobserved Confounds in Classification Using Correlational Constraints},
  booktitle = {Proceedings of the Eleventh International AAAI Conference on Web and Social Media (ICWSM 2017)},
  year =         2017
}

Identifying leading indicators of product recalls from online reviews using positive unlabeled learning and domain adaptation

2017-04-30T00:00:00+00:00

Abstract

Consumer protection agencies are charged with safeguarding the public from hazardous products, but the thousands of products under their jurisdiction make it challenging to identify and respond to consumer complaints quickly. In this paper, we propose a system to mine Amazon.com reviews to identify products that may pose safety or health hazards. Since labeled data for this task are scarce, our approach combines positive unlabeled learning with domain adaptation to train a classifier from consumer complaints submitted to an online government portal. We find that our approach results in an absolute F1 score improvement of 8% over the best competing baseline. Furthermore, when we apply the classifier to Amazon reviews of known recalled products, we identify safety hazard reports prior to the recall date for 45% of the products. This suggests that the system may be able to provide an early warning system to alert consumers to hazardous products before an official recall is announced.

Demo: https://shreeshbhat.github.io/recalls

Citation

@InProceedings{bhat2017identifying,
  author =       {Shreesh Kumara Bhat and Aron Culotta},
  title =        {Identifying leading indicators of product recalls from online reviews using positive unlabeled learning and domain adaptation},
  booktitle = {Proceedings of the Eleventh International AAAI Conference on Web and Social Media (ICWSM 2017)},
  year =         2017
}

Using online social networks to measure consumers’ brand perception

2017-03-01T00:00:00+00:00

Abstract

The ability to measure and monitor specific dimensions of brand image has a range of useful applications in marketing, from developing competitive strategy to identifying strength and weaknesses to evaluating the effectiveness of marketing initiatives. Nevertheless, obtaining reliable measurements is an ongoing challenge for marketers. Traditional methods such as administering surveys can be expensive and biased, and are limited in scale, both in terms of the number of brands and dimensions that can be tracked, and the frequency with which the measurements can be updated. The explosion of social media in recent years has created an enormous secondary data trail that is available for analysis. However, the most common analytics approaches, such as those that rely on user-generated text, are difficult to apply due to the scarcity of relevant conversations, as well as the ambiguity, variety, and often rapid changes in linguistic terms used by consumers. This paper describes a recent advance in marketing science that makes use of brand social network connections to make highly scalable inferences about brand image. This promising new approach provides many potential advantages, including the ability to fully automate monitoring for a large number of brands over a wide range of dimensions.

Citation

@article{cutler2017using,
  title={Using online social networks to measure consumers’ brand perception},
  author={Cutler, Jennifer and Culotta, Aron},
  journal={Applied Marketing Analytics},
  volume={2},
  number={4},
  pages={312--321},
  year={2017},
  publisher={Henry Stewart Publications}
}

#Polar Scores: Measuring Partisanship Using Social Media Content

2016-07-22T00:00:00+00:00

Abstract

We present a new approach to measuring political polarization, including a novel algorithm and open source Python code, which leverages Twitter content to produce measures of polarization for both users and hashtags. #Polar scores provide advantages over existing measures because they (1) can be calculated throughout the legislative cycle, (2) allow for easy differentiation between users with similar scores, (3) are chamber-agnostic, and (4) are a generic approach that can be applied beyond the U.S. Congress. #Polar scores leverage available information such as party labels, word frequency, and hashtags to create an accessible, straightforward algorithm for estimating polarity using text.

Citation

@article{hemphill2016polar,
author = {Libby Hemphill and Aron Culotta and Matthew Heston},
title = {\#Polar Scores: Measuring Partisanship Using Social Media Content},
journal = {Journal of Information Technology \& Politics},
volume = {0},
number = {ja},
pages = {null},
year = {2016},
doi = {10.1080/19331681.2016.1214093},
}

Cold-Start Recommendations for Audio News Stories Using Matrix Factorization

2016-05-02T00:00:00+00:00

Abstract

We investigate a suite of recommendation algorithms for audio news listening applications. This domain presents several challenges that distinguish it from more commonly studied applications such as movie recommendations: (1) we do not receive explicit rating feedback, instead only observing when a user skips a story; (2) new stories arrive continuously, increasing the importance of making recommendations for items with few observations (the cold start problem); (3) story attributes have high dimensionality, making it challenging to identify similar stories. To address the first challenge, we formulate the problem as predicting the percentage of a story a user will listen to; to address the remaining challenges, we propose several matrix factorization algorithms that cluster users, n-grams, and stories simultaneously, while optimizing prediction accuracy. We empirically evaluate our approach on a dataset of 50K users, 26K stories, and 975K interactions collected over a five month period. We find that while simple models work well for stories with many observations, our proposed approach performs best for stories with few ratings, which is critical for the real-world deployment of such an application.

Citation

@InProceedings{ehsan16domain,
  author =       {Ehsan Mohammady Ardehaly and Aron Culotta},
  title =        {Cold-Start Recommendations for Audio News Stories Using Matrix Factorization},
  booktitle = {25th International Joint Conference on Artificial Intelligence (IJCAI)},
  year =         2016
}

Domain Adaptation for Learning from Label Proportions Using Self-Training

2016-05-01T00:00:00+00:00

Abstract

Learning from Label Proportions (LLP) is a machine learning problem in which the training data consist of bags of instances, and only the class label distribution for each bag is known. In some domains label proportions are readily available; for example, by grouping social media users by location, one can use census statistics to build a classifier for user demographics. However, label proportions are unavailable in many domains, such as product review sites. The goal of this paper is to determine whether an LLP classifier fit in one domain can be modified to classify instances from another domain. To do so, we propose a domain adaptation algorithm that uses an LLP model fit on the source domain to generate label proportions for the target domain. A new LLP model is then fit on the target domain, and this self-training process is repeated to adapt the model from source to target. Our experiments on five diverse tasks indicate an 11% average absolute improvement in accuracy as compared to using LLP without domain adaptation. In contrast to existing domain adaptation algorithms, our approach requires only label proportions in the source domain, and the results suggest that the approach is effective even when the target domain is substantially different from the source domain.

Citation

@InProceedings{ehsan16domain,
  author =       {Ehsan Mohammady Ardehaly and Aron Culotta},
  title =        {Domain Adaptation for Learning from Label Proportions Using Self-Training},
  booktitle = {25th International Joint Conference on Artificial Intelligence (IJCAI)},
  year =         2016
}

Robust Text Classification in the Presence of Confounding Bias

2016-02-01T00:00:00+00:00

Abstract

As text classifiers become increasingly used in real-time applications, it is critical to consider not only their accuracy but also their robustness to changes in the data distribution. In this paper, we consider the case where there is a confounding variable $Z$ that influences both the text features $X$ and the class variable $Y$. For example, a classifier trained to predict the health status of a user based on their online communications may be confounded by socioeconomic variables. When the influence of $Z$ changes from training to testing data, we find that classifier accuracy can degrade rapidly. Our approach, based on Pearl’s back-door adjustment, estimates the underlying effect of a text variable on the class variable while controlling for the confounding variable. Although our goal is prediction, not causal inference, we find that such adjustments are essential to building text classifiers that are robust to confounding variables. On three diverse text classifications tasks, we find that covariate adjustment results in higher accuracy than competing baselines over a range of confounding relationships (e.g., in one setting, accuracy improves from 60% to 81%).

Citation

@InProceedings{virgile2016using,
  author =       {Virgile Landeiro and Aron Culotta},
  title =        {Robust Text Classification in the Presence of Confounding Bias},
  booktitle = {Thirtieth National Conference on Artificial Intelligence (AAAI)},
  year =         2016
}

Reducing confounding bias in observational studies that use text classification

2016-02-01T00:00:00+00:00

Abstract

As text classifiers become increasingly used in observational studies, it is critical to consider not only their accuracy but also their robustness to changes in the data distribution. In this paper, we consider the case where there is a confounding variable $Z$ that influences both the text features $W$ and the class variable $Y$. For example, a classifier trained to predict the health status of a user based on their online communications may be confounded by socioeconomic variables. When the influence of $Z$ changes from training to testing data, we find that classifier accuracy can degrade rapidly. Our approach, based on Pearl’s back-door adjustment, estimates the underlying effect of a text variable on the class variable while controlling for the confounding variable. We conduct an observational study to estimate the effect of location on dispositional affect, with gender as a confounder. We find that our adjustment results in more accurate estimates of effect sizes over a range of possible confounding strengths.

Citation

@InProceedings{virgile2016reducing,
  author =       {Virgile Landeiro and Aron Culotta},
  title =        {Reducing confounding bias in observational studies that use text classification},
  booktitle = {AAAI Spring Symposium on Observational Studies through Social Media and Other Human-Generated Content},
  year =         2016
}

Predicting Twitter User Demographics using Distant Supervision from Website Traffic Data

2016-02-01T00:00:00+00:00

Abstract

Understanding the demographics of users of online social networks has important applications for health, marketing, and public messaging. Whereas most prior approaches rely on a supervised learning approach, in which individual users are labeled with demographics for training, we instead create a distantly labeled dataset by collecting audience measurement data for 1,500 websites (e.g., 50% of visitors to gizmodo.com are estimated to have a bachelor’s degree). We then fit a regression model to predict these demographics from information about the followers of each website on Twitter. Using patterns derived both from textual content and the social network of each user, our final model produces an average held-out correlation of .77 across seven different variables (age, gender, education, ethnicity, income, parental status, and political preference). We then apply this model to classify individual Twitter users by ethnicity, gender, and political preference, finding performance that is surprisingly competitive with a fully supervised approach.

Citation

@Article{culotta16predicting,
	author =       {Aron Culotta and and Nirmal Kumar Ravi and Jennifer Cutler},
	title =        {Predicting Twitter User Demographics using Distant Supervision from Website Traffic Data},
	journal =      {Journal of Artificial Intelligence Research},
            volume =       {55},
	year =         2016,
            pages =        {389--408},
}

Mining brand perceptions from Twitter social networks

2016-02-01T00:00:00+00:00

Abstract

Consumer perceptions are important components of brand equity and therefore marketing strategy. Segmenting these perceptions into attributes such as eco-friendliness, nutrition, and luxury enable a fine-grained understanding of the brand’s strengths and weakness. Traditional approaches towards monitoring such perceptions (e.g., surveys) are costly and time-consuming, and their results may quickly become outdated. Extant data mining methods are not suitable for this goal, and generally require extensive hand-annotated data or context customization, which leads to many of the same limitations as direct elicitation. Here, we investigate a novel, general, and fully automated method for inferring attribute-specific brand perception ratings by mining the brand’s social connections on Twitter.

Using a set of over 200 brands and three perceptual attributes, we compare the method’s automatic ratings estimates with directly-elicited survey data, finding a consistently strong correlation. The approach provides a reliable, flexible, and scalable method for monitoring brand perceptions, and offers a foundation for future advances in understanding brand-consumer social media relationships.

Citation

@article{culotta2016mining,
  author =       {Aron Culotta and Jennifer Cutler},
  title =        {Mining brand perceptions from Twitter social networks},
  journal = {Marketing Science},
  year =         2016,
  note = {(to appear)}
}

Training a text classifier with a single word using Twitter Lists and domain adaptation

2016-02-01T00:00:00+00:00

Abstract

Annotating data is a common bottleneck in building text classifiers. This is particularly problematic in social media domains, where data drift requires frequent retraining to maintain high accuracy. In this paper, we propose and evaluate a text classification method for Twitter data whose only required human input is a single keyword per class. The algorithm proceeds by identifying exemplar Twitter accounts that are representative of each class by analyzing Twitter Lists (human-curated collections of related Twitter accounts). A classifier is then fit to the exemplar accounts and used to predict labels of new tweets and users. We develop domain adaptation methods to address the noise and selection bias inherent to this approach, which we find to be critical to classification accuracy. Across a diverse set of tasks (topic, gender, and political affiliation classification), we find that the resulting classifier is competitive with a fully supervised baseline, achieving superior accuracy on four of six datasets despite using no manually labeled data.

Citation

@article{culotta2016training,
  author =       {Aron Culotta},
  title =        {Training a text classifier with a single word using Twitter Lists and domain adaptation},
  journal = {Social Network Mining and Analysis},
  volume = 6,
  number = 1,
  pages = {1--15},
  year =         2016,
}

A demographic and sentiment analysis of e-cigarette messages on Twitter

2015-09-09T00:00:00+00:00

Abstract

Social media provide a potentially useful new data source to understand emerging public health behaviors. In this paper, we study messages about e-cigarettes posted to Twitter.com. We apply methods to classify messages by sentiment and to estimate the gender and age of users. We apply our approach to nearly one million messages about e-cigarettes posted from October 2012 to September 2013. We find that overall volume of e-cigarette tweets increased five-fold (from 30K per month to 150K per month); and that males and younger users were more likely to post positive messages about e-cigarettes. A qualitative analysis also reveals several trends, such as negative sentiment toward people who smoke in class; females giving e-cigarettes to relatives to help them quit smoking; and spikes in people using e-cigarettes to quit smoking in January.

Citation

@inproceedings{resende2015demographic,
  author =       {Elaine Cristina Resende and Aron Culotta},
  booktitle = {Workshop on Computational Health Science at the 6th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics},
  title = {A demographic and sentiment analysis of e-cigarette messages on Twitter},
  year = 2015
}

Finding truth in cause-related advertising: A lexical analysis of brands' health, environment, and social justice communications on Twitter

2015-09-01T00:00:00+00:00

Abstract

Consumers increasingly desire to make purchasing decisions based on factors such as health, the environment, and social justice. In response, there has been a commensurate rise in cause-related marketing to appeal to socially-conscious consumers. However, a lack of regulation and standardization makes it difficult for consumers to assess marketing claims; this is further complicated by social media, which firms use to cultivate a personality for their brand through frequent conversational messages. Yet, little empirical research has been done to explore the relationship between cause-related marketing messages on social media and the true cause alignment of brands. In this paper, we explore this by pairing the marketing messages from the Twitter accounts of over 1,000 brands with third-party ratings of each brand with respect to health, the environment, and social justice. Specifically, we perform text regression to predict each brand’s true rating in each dimension based on the lexical content of its tweets, and find significant held-out correlation on each task, suggesting that a brand’s alignment with a social cause can be somewhat reliably signaled through their Twitter communications—though the signal is weak in many cases. To aid in the identification of brands that engage in misleading cause-related communication as well as terms that more likely indicate insincerity, we propose a procedure to rank both brands and terms by their volume of “conflicting” communications (i.e., “greenwashing”). We further explore how cause-related terms are used differently by brands that are strong vs. weak in actual alignment with the cause. The results provide insight into current practices in cause-related marketing in social media, and provide a framework for identifying and monitoring misleading communications. Together, they can be used to promote transparency in cause-related marketing in social media, better enabling brands to communicate authentic values-based policy decisions, and consumers to make socially responsible purchase decisions.

Citation

@article{culotta2015finding,
  author =       {Aron Culotta and Jennifer Cutler and Junzhe Zheng},
  journal =        {Journal of Values-Based Leadership},
  title = {Finding truth in cause-related advertising: A lexical analysis of brands' health, environment, and social justice communications on Twitter},
  volume = 8,
  number = 2,
  year = 2015
}

Inferring latent attributes of Twitter users with label regularization

2015-05-31T00:00:00+00:00

Abstract

Inferring latent attributes of online users has many applications in public health, politics, and marketing. Most existing approaches rely on supervised learning algorithms, which require manual data annotation and therefore are costly to develop and adapt over time. In this paper, we propose a lightly supervised approach based on label regularization to infer the age, ethnicity, and political orientation of Twitter users. Our approach learns from a heterogeneous collection of soft constraints derived from Census demographics, trends in baby names, and Twitter accounts that are emblematic of class labels. To counteract the imprecision of such constraints, we compare several constraint selection algorithms that optimize classification accuracy on a tuning set. We find that using no user-annotated data, our approach is within 2% of a fully supervised baseline for three of four tasks. Using a small set of labeled data for tuning further improves accuracy on all tasks.

Citation

@InProceedings{ehsan2015inferring,
  author =       {Ehsan Mohammady Ardehaly and Aron Culotta},
  title =        {Inferring latent attributes of Twitter users with label regularization }, 
  booktitle = {NAACL/HLT}, 
  year =         2015
}

Using Matched Samples to Estimate the Effects of Exercise on Mental Health from Twitter

2015-01-01T00:00:00+00:00

Abstract

Recent work has demonstrated the value of social media monitoring for health surveillance (e.g., tracking influenza or depression rates). It is an open question whether such data can be used to make causal inferences (e.g., determining which activities lead to increased depression rates). Even in traditional, restricted domains, estimating causal effects from observational data is highly susceptible to confounding bias. In this work, we estimate the effect of exercise on mental health from Twitter, relying on statistical matching methods to reduce confounding bias. We train a text classifier to estimate the volume of a user’s tweets expressing anxiety, depression, or anger, then compare two groups: those who exercise regularly (identified by their use of physical activity trackers like Nike+), and a matched control group. We find that those who exercise regularly have significantly fewer tweets expressing depression or anxiety; there is no significant difference in rates of tweets expressing anger. We additionally perform a sensitivity analysis to investigate how the many experimental design choices in such a study impact the final conclusions, including the quality of the classifier and the construction of the control group.

Citation

@InProceedings{virgile2015using,
  author =       {Virgile Landeiro Dos Reis and Aron Culotta},
  title =        {Using Matched Samples to Estimate the Effects of Exercise\\ on Mental Health from Twitter},
  booktitle = {Twenty-ninth National Conference on Artificial Intelligence (AAAI)},
  year =         2015
}

Predicting the Demographics of Twitter Users from Website Traffic Data

2015-01-01T00:00:00+00:00

Abstract

Understanding the demographics of users of online social networks has important applications for health, marketing, and public messaging. In this paper, we predict the demographics of Twitter users based on whom they follow. Whereas most prior approaches rely on a supervised learning approach, in which individual users are labeled with demographics, we instead create a distantly labeled dataset by collecting audience measurement data for 1,500 websites (e.g., 50% of visitors to gizmodo.com are estimated to have a bachelor’s degree). We then fit a regression model to predict these demographics using information about the followers of each website on Twitter. The resulting average held-out correlation is .77 across six different variables (gender, age, ethnicity, education, income, and child status). We additionally validate the model on a smaller set of Twitter users labeled individually for ethnicity and gender, finding performance that is surprisingly competitive with a fully supervised approach.

Outstanding Paper, Honorable Mention

Citation

@InProceedings{culotta2015predicting,
  author =       {Aron Culotta and Nirmal Ravi Kumar and Jennifer Cutler},
  title =        {Predicting the Demographics of Twitter Users from Website Traffic Data},
  booktitle = {Twenty-ninth National Conference on Artificial Intelligence (AAAI)},
  year =         2015
}

Reducing Sampling Bias in Social Media Data for County Health Inference

2014-08-02T00:00:00+00:00

Abstract

A number of recent studies have demonstrated the utility of social media data for inferring societal attributes such as public opinion and health. A commonly declared limitation of this methodology is the selection bias inherent in this approach – social media users are a non-representative sample of the population. This is exacerbated by filtering steps that further limit the sample set in biased ways. Building on recent work in computational linguistics that infers demographic attributes of people based on their communications, we investigate methods to quantify and control for selection bias in social media studies. We present results estimating several county-level health statistics (e.g., obesity, diabetes, access to healthy foods) based on the Twitter activity of the top 100 counties in the U.S., and we compare strategies for reducing selection bias.

Citation

@InProceedings{culotta14reducing,
  author =       {Aron Culotta},
  title =        {Reducing Sampling Bias in Social Media Data for County Health Inference},
  booktitle = {JSM Proceedings},
  year =         2014
}

Anytime Active Learning

2014-07-01T00:00:00+00:00

Abstract

A common bottleneck in deploying supervised learning systems is collecting human-annotated examples. In many domains, annotators form an opinion about the label of an example incrementally — e.g., each additional word read from a document or each additional minute spent inspecting a video helps inform the annotation. In this paper, we investigate whether we can train learning systems more efficiently by requesting an annotation before inspection is fully complete — e.g., after reading only 25 words of a document. While doing so may reduce the overall annotation time, it also introduces the risk that the annotator might not be able to provide a label if interrupted too early. We propose an anytime active learning approach that optimizes the annotation time and response rate simultaneously. We conduct user studies on two document classification datasets and develop simulated annotators that mimic the users. Our simulated experiments show that anytime active learning outperforms several baselines on these two datasets. For example, with an annotation budget of one hour, training a classifier by annotating the first 25 words of each document reduces classification error by 17% over annotating the first 100 words of each document.

Citation

@InProceedings{ramirez14any,
  author =       {Maria E Ramirez-Loaiza and Aron Culotta and Mustafa Bilgic},
  title =        {Anytime Active Learning},
  booktitle = {AAAI},
  year =         2014
}

Using county demographics to infer attributes of Twitter users

2014-06-01T00:00:00+00:00

Abstract

Social media are increasingly being used to complement traditional survey methods in health, politics, and marketing. However, little has been done to adjust for the sampling bias inherent in this approach. Inferring demographic attributes of social media users is thus a critical step to improving the validity of such studies. While there have been a number of supervised machine learning approaches to this problem, these rely on a training set of users annotated with attributes, which can be difficult to obtain. We instead propose training a demographic attribute classifiers that uses county-level supervision. By pairing geolocated social media with county demographics, we build a regression model mapping text to demographics. We then adopt this model to make predictions at the user level. Our experiments using Twitter data show that this approach is surprisingly competitive with a fully supervised approach, estimating the race of a user with 80% accuracy.

Citation

@InProceedings{ehsan2014using,
  author =       {Ehsan Mohammady and Aron Culotta},
  title =        {Using county demographics to infer attributes of Twitter users},
  booktitle = {ACL Joint Workshop on Social Dynamics and Personal Attributes in Social Media},
  year =         2014
}

Tweedr: Mining Twitter to Inform Disaster Response

2014-05-01T00:00:00+00:00

Abstract

In this paper, we introduce Tweedr, a Twitter-mining tool that extracts actionable information for disaster relief workers during natural disasters. The Tweedr pipeline consists of three main parts: classification, clustering and extraction. In the classification phase, we use a variety of classification methods (sLDA, SVM, and logistic regression) to identify tweets reporting damage or casualties. In the clustering phase, we use filters to merge tweets that are similar to one another; and finally, in the extraction phase, we extract tokens and phrases that report specific information about different classes of infrastructure damage, damage types, and casualties. We empirically validate our approach with tweets collected from 12 different crises in the United States since 2006.

Citation

@InProceedings{ashktorab14tweedr
  author =       { Zahra Ashktorab and Christopher Brown and Manojit Nandi and Aron Culotta},
  title =        {Tweedr: {M}ining {T}witter to Inform Disaster Response},
  booktitle = {Proceedings of the 11th International ISCRAM Conference},
  year =         2014
}

Estimating County Health Statistics with Twitter

2014-04-26T00:00:00+00:00

Abstract

Understanding the relationships among environment, behavior, and health is a core concern of public health researchers. While a number of recent studies have investigated the use of social media to track infectious diseases such as influenza, little work has been done to determine if other health concerns can be inferred. In this paper, we present a large-scale study of 27 health-related statistics, including obesity, health insurance coverage, access to healthy foods, and teen birth rates. We perform a linguistic analysis of the Twitter activity in the top 100 most populous counties in the U.S., and find a significant correlation with 6 of the 27 health statistics. When compared to traditional models based on demographic variables alone, we find that augmenting models with Twitter-derived information improves predictive accuracy for 20 of 27 statistics, suggesting that this new methodology can complement existing approaches.

Citation

@InProceedings{culotta2014estimating,
  author =       {Aron Culotta},
  title =        {Estimating County Health Statistics with Twitter},
  booktitle = {Proceedings of the {SIGCHI} Conference on Human Factors in Computing Systems},
  year =         2014
}

Inferring the Origin Locations of Tweets with Quantitative Confidence

2014-02-15T00:00:00+00:00

Abstract

Social Internet content plays an increasingly critical role in many domains, including public health, disaster management, and politics. However, its utility is limited by missing geographic information; for example, fewer than 1.6% of Twitter messages (tweets) contain a geotag. We propose a scalable, content-based approach to estimate the location of tweets using a novel yet simple variant of gaussian mixture models. Further, because real-world applications depend on quantified uncertainty for such estimates, we propose novel metrics of accuracy, precision, and calibration, and we evaluate our approach accordingly. Experiments on 13 million global, comprehensively multi-lingual tweets show that our approach yields reliable, well-calibrated results competitive with previous computationally intensive methods. We also show that a relatively small number of training data are required for good estimates (roughly 30,000 tweets), and models are quite time-invariant (effective on tweets many weeks newer than the training set). Finally, we show that toponyms and languages with small geographic footprint provide the most useful location signals.

Citation

@InProceedings{priedhorsky14inferring,
  author    = {Reid Priedhorsky and Aron Culotta and Sara Y. Del Valle},
  title     = {Inferring the origin location of tweets with quantitative confidence},
  booktitle = {17th ACM Conference on Computer Supported Cooperative Work and Social Computing},
  year      = {2014}
}

Lightweight methods to estimate influenza rates and alcohol sales volume from Twitter messages

2013-01-01T00:00:00+00:00

Abstract

We analyze over 570 million Twitter messages from an eight month period and find that tracking a small number of keywords allows us to estimate influenza rates and alcohol sales volume with high accuracy. We validate our approach against government statistics and find strong correlations with influenza-like illnesses reported by the U.S. Centers for Disease Control and Prevention (r(14) = .964, p < .001) and with alcohol sales volume reported by the U.S. Census Bureau (r(5) = .932, p < .01). We analyze the robustness of this approach to spurious keyword matches, and we propose a document classification component to filter these misleading messages. We find that this document classifier can reduce error rates by over half in simulated false alarm experiments, though more research is needed to develop methods that are robust in cases of extremely high noise.

Citation

@article{culotta13lightweight,
  author  = Aron Culotta,
  title   = {Lightweight methods to estimate influenza rates and alcohol sales volume from {T}witter messages},
  journal = {Language Resources and Evaluation, Special Issue on Analysis of Short Texts on the Web},
  volume  = {47},
  number  = {1},
  pages   = {217--238},
  year    = 2013,
}

Too Neurotic, Not too Friendly: Structured Personality Classification on Textual Data

2013-01-01T00:00:00+00:00

Abstract

Personality plays a fundamental role in human interaction. With the increasing posting of text on the internet, automatic detection of a person’s personality based on the text she produces is an important step to labeling and analyzing human behavior at a large scale. To date, most approaches to personality classification have modeled feature representations of the text to produce output classifications. In this paper we use structured classification approaches that learn and model both feature representations of text and dependencies between output labels (i.e. personality traits). Our study finds that there seems to be a correlation between Agreeableness and Emotional Stability and that it may be helping boost accuracy for Agreeableness when compared to more traditional approaches for supervised classification.

Citation

@InProceedings{iacobelli13too,
  author = 	 {Francisco Iacobelli and {\bf Aron Culotta} },
  title = 	 {Too Neurotic, Not too Friendly: Structured Personality Classification on Textual Data },
  booktitle = {ICWSM Workshop on Personality Classification},
  mytype = {Refereed Workshop Publications},
  year = 	 2013
}

Towards Anytime Active Learning: Interrupting Experts to Reduce Annotation Costs

2013-01-01T00:00:00+00:00

Abstract

Many active learning methods use annotation cost or expert quality as part of their framework to select the best data for annotation. While these methods model expert quality, availability, or expertise, they have no direct influence on any of these elements. We present a novel framework built upon decision-theoretic active learning that allows the learner to directly control label quality by allocating a time budget to each annotation. We show that our method is able to improve performance efficiency of the active learner through an interruption mechanism trading off the induced error with the cost of annotation. Our simulation experiments on three document classification tasks show that some interruption is almost always better than none, but that the optimal interruption time varies by dataset.

Citation

@InProceedings{ramirez13towards,
  author = 	 {Maria E. Ramirez-Loaiza and {\bf Aron Culotta} and Mustafa Bilgic },
  title = 	 {Towards Anytime Active Learning: Interrupting Experts to Reduce Annotation Costs },
  booktitle = {KDD Workshop on Interactive Data Exploration and Analytics (IDEA)},
  year = 	 2013
}

A demographic analysis of online sentiment during Hurricane Irene

2012-06-03T00:00:00+00:00

Abstract

We examine the response to the recent natural disaster Hurricane Irene on Twitter.com. We collect over 65,000 Twitter messages relating to Hurricane Irene from August 18th to August 31st, 2011, and group them by location and gender. We train a sentiment classifier to categorize messages based on level of concern, and then use this classifier to investigate demographic differences. We report three principal findings: (1) the number of Twitter messages related to Hurricane Irene in directly affected regions peaks around the time the hurricane hits that region; (2) the level of concern in the days leading up to the hurricane’s arrival is dependent on region; and (3) the level of concern is dependent on gender, with females being more likely to express concern than males. Qualitative linguistic variations further support these differences. We conclude that social media analysis provides a viable, real-time complement to traditional survey methods for understanding public perception towards an impending disaster.

Citation

@inproceedings{mandel12demo,
  author = {Benjamin Mandel and Aron Culotta and John Boulahanis and Danielle Stark and Bonnie Lewis and Jeremy Rodrigue},
  title = {A demographic analysis of online sentiment during {H}urricane {I}rene},
  booktitle = {NAACL-HLT Workshop on Language in Social Media},
  year = {2012},
}

SampleRank: Training factor graphs with atomic gradients

2011-06-01T00:00:00+00:00

Abstract

We present SampleRank, an alternative to contrastive divergence (CD) for estimating parameters in complex graphical models. SampleRank harnesses a user-provided loss function to distribute stochastic gradients across an MCMC chain. As a result, parameter updates can be computed between arbitrary MCMC states. SampleRank is not only faster than CD, but also achieves better accuracy in practice (up to 23% error reduction on noun-phrase coreference).

Citation

@inproceedings{wick11sample,
  author = {Michael Wick and Khashayar Rohanimanesh and Kedar Bellare and Aron Culotta and Andrew McCallum},
  title = {SampleRank: Training factor graphs with atomic gradients},
  booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
  shortbooktitle = {ICML},
  year = {2011},
}

Detecting influenza epidemics by analyzing Twitter messages

2010-06-01T00:00:00+00:00

Abstract

We analyze over 500 million Twitter messages from an eight month period and find that tracking a small number of flu-related keywords allows us to forecast future influenza rates with high accuracy, obtaining a 95% correlation with national health statistics. We then analyze the robustness of this approach to spurious keyword matches, and we propose a document classification component to filter these misleading messages. We find that this document classifier can reduce error rates by over half in simulated false alarm experiments, though more research is needed to develop methods that are robust in cases of extremely high noise.

Citation

@techreport{culotta10detecting,
  author = {Aron Culotta},
  title = {Detecting influenza epidemics by analyzing {T}witter messages},
  howpublished = {arXiv:1007.4748v1 [cs.IR]},
  mytype = {Technical Reports},
  month = {July},
  year = {2010},
}

Towards detecting influenza epidemics by analyzing Twitter messages

2010-01-01T00:00:00+00:00

Abstract

Citation

@inproceedings{culotta10towards,
  author = {Aron Culotta},
  title = {Towards detecting influenza epidemics by analyzing {T}witter messages},
  booktitle = {KDD Workshop on Social Media Analytics},
  year = {2010},
}

SampleRank: Learning preferences from atomic gradients

2009-01-01T00:00:00+00:00

Abstract

Large templated factor graphs with complex structure that changes during inference have been shown to provide state-of-the-art experimental results on tasks such as identity uncertainty and information integration. However, learning parameters in these models is difficult because computing the gradients require expensive inference routines. In this paper we propose an online algorithm that instead learns preferences over hypotheses from the gradients between the atomic steps of inference. Although there are a combinatorial number of ranking constraints over the entire hypothesis space, a connection to the frameworks of sampled convex programs reveals a polynomial bound on the number of rankings that need to be satisfied in practice. We further apply ideas of passive aggressive algorithms to our update rules, enabling us to extend recent work in confidence-weighted classification to structured prediction problems. We compare our algorithm to structured perceptron, contrastive divergence, and persistent contrastive divergence, demonstrating substantial error reductions on two real-world problems (20% over contrastive divergence).

Citation

@inproceedings{wick09sample,
  author = {Michael Wick and Khashayar Rohanimanesh and {\bf Aron Culotta} and Andrew McCallum},
  title = {SampleRank: Learning preferences from atomic gradients},
  booktitle = {Neural Information Processing Systems (NIPS) Workshop on Advances in Ranking},
  shortbooktitle = {NIPS},
  year = {2009}
}

An entity-based model for coreference resolution

2009-01-01T00:00:00+00:00

Abstract

Recently, many advanced machine learning approaches have been proposed for coreference resolution; however, all of the discriminatively-trained models reason over mentions rather than entities. That is, they do not explicitly contain variables indicating the ‘canonical’ values for each attribute of an entity (e.g., name, venue, title, etc.). This canonicalization step is typically implemented as a post-processing routine to coreference resolution prior to adding the extracted entity to a database. In this paper, we propose a discriminatively-trained model that jointly performs coreference resolution and canonicalization, enabling features over hypothesized entities. We validate our approach on two different coreference problems: newswire anaphora resolution and research paper citation matching, demonstrating improvements in both tasks and achieving an error reduction of up to 62% when compared to a method that reasons about mentions only.

Citation

@inproceedings{wick09entity,
  	  author = {Michael Wick and Aron Culotta and Khashayar Rohanimanesh and Andrew McCallum},
  title = {An entity-based model for coreference resolution},
  booktitle = {SIAM International Conference on Data Mining},
  	  year = {2009},
}

Learning and inference in weighted logic with application to natural language processing

2008-01-01T00:00:00+00:00

Abstract

Over the past two decades, statistical machine learning approaches to natural language processing have largely replaced earlier logic-based systems. These probabilistic methods have proven to be well-suited to the ambiguity inherent in human communication. However, the shift to statistical modeling has mostly abandoned the representational advantages of logic-based approaches. For example, many language processing problems can be more meaningfully expressed in first-order logic rather than propositional logic. Unfortunately, most machine learning algorithms have been developed for propositional knowledge representations.
In recent years, there have been a number of attempts to combine logical and probabilistic approaches to artificial intelligence. However, their impact on real-world applications has been limited because of serious scalability issues that arise when algorithms designed for propositional representations are applied to first-order logic representations. In this thesis, we explore approximate learning and inference algorithms that are tailored for higher-order representations, and demonstrate that this synthesis of probability and logic can significantly improve the accuracy of several language processing systems.

Citation

@phdthesis{culotta08learning,
  author = {Aron Culotta},
  title = {Learning and inference in weighted logic with application to natural language processing},
  school = {University of Massachusetts},
  year = {2008},
  month = {May},
}

Sparse Message Passing Algorithms for Weighted Maximum Satisfiability

2007-01-01T00:00:00+00:00

Abstract

Weighted maximum satisfiability is a well-studied problem that has important applicability to artificial intelligence (for instance, MAP inference in Bayesian networks). General-purpose stochastic search algorithms have proven to be accurate and efficient for large problem instances; however, these algorithms largely ignore structural properties of the input. For example, many problems are highly clustered, in that they contain a collection of loosely coupled subproblems (e.g. pipelines of NLP tasks). In this paper, we propose a message passing algorithm to solve weighted maximum satisfiability problems that exhibit this clustering property. Our algorithm fuses local solutions to each subproblem into a global solution by iteratively passing summary information between clusters and recomputing local solutions. Because the size of these messages can become unwieldy for large problems, we explore several message compression techniques to transmit the most valuable information as compactly as possible. We empirically compare our algorithm against a state-of-the-art stochastic solver and show that for certain classes of problems our message passing algorithm finds significantly better solutions.

Citation

@inproceedings{culotta07sparse,
  author = {Aron Culotta and Andrew McCallum and Bart Selman and Ashish Sabharwal},
  title = {Sparse Message Passing Algorithms for Weighted Maximum Satisfiability},
  booktitle = {New England Student Colloquium on Artificial Intelligence (NESCAI)},
  address = {Ithaca, NY},
  year = {2007},
}

First-Order Probabilistic Models for Coreference Resolution

2007-01-01T00:00:00+00:00

Abstract

Traditional noun phrase coreference resolution systems represent features only of pairs of noun phrases. In this paper, we propose a machine learning method that enables features over sets of noun phrases, resulting in a first-order probabilistic model for coreference. We outline a set of approximations that make this approach practical, and apply our method to the ACE coreference dataset, achieving a 45% error reduction over a comparable method that only considers features of pairs of noun phrases. This result demonstrates an example of how a first-order logic representation can be incorporated into a probabilistic model and scaled efficiently.

Citation

@inproceedings{culotta07first,
  author = {Aron Culotta and Michael Wick and Robert Hall and Andrew McCallum},
  title = {First-Order Probabilistic Models for Coreference Resolution},
  booktitle = {Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (HLT/NAACL)},
  year = {2007},
  location = {Rochester, NY},
  pages = {81--88},
}

Canonicalization of Database Records using Adaptive Similarity Measures

2007-01-01T00:00:00+00:00

Abstract

It is becoming increasingly common to construct databases from information automatically culled from many heterogeneous sources. For example, a research publication database can be constructed by automatically extracting titles, authors, and conference information from papers and their references. A common difficulty in consolidating data from multiple sources is that records are referenced in a variety of ways (e.g. abbreviations, aliases, and misspellings). Therefore, it can be difficult to construct a single, standard representation to present to the user. We refer to the task of constructing this representation as canonicalization. Despite its importance, there is very little existing work on canonicalization. In this paper, we explore the use of edit distance measures to construct a canonical representation that is “central” in the sense that it is most similar to each of the disparate records. This approach reduces the impact of noisy records on the canonical representation. Furthermore, because the user may prefer different styles of canonicalization, we show how different edit distance costs can result in different forms of canonicalization. For example, reducing the cost of character deletions can result in representations that favor abbreviated forms over expanded forms (e.g. KDD versus Conference on Knowledge Discovery and Data Mining). We describe how to learn these costs from a small amount of manually annotated data using stochastic hill-climbing. Additionally, we investigate feature-based methods to learn ranking preferences over canonicalizations. We empirically evaluate our approach on a real-world publications database and show that our learning method results in a canonicalization solution that is robust to errors and easily customizable to user preferences.

Citation

@inproceedings{culotta07canonicalization,
  author = {Aron Culotta and Michael Wick and Robert Hall and Matthew Marzilli and Andrew McCallum},
  title = {Canonicalization of Database Records using Adaptive Similarity Measures},
  year = {2007},
  booktitle = {Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)},
  address = {San Jose, CA},
}

Author Disambiguation using Error-driven Machine Learning with a Ranking Loss Function

2007-01-01T00:00:00+00:00

Abstract

Author disambiguation is the problem of determining whether records in a publications database refer to the same person. A common supervised machine learning approach is to build a classifier to predict whether a pair of records is coreferent, followed by a clustering step to enforce transitivity. However, this approach ignores powerful evidence obtainable by examining sets (rather than pairs) of records, such as the number of publications or co-authors an author has. In this paper we propose a representation that enables these first-order features over sets of records. We then propose a training algorithm well-suited to this representation that is (1) error-driven in that training examples are generated from incorrect predictions on the training data, and (2) rank-based in that the classifier induces a ranking over candidate predictions. We evaluate our algorithms on three author disambiguation datasets and demonstrate error reductions of up to 60% over the standard binary classification approach.

Citation

@inproceedings{culotta07author,
  author = {Aron Culotta and Pallika Kanani and Robert Hall and Michael Wick and Andrew McCallum},
  title = {Author Disambiguation using Error-driven Machine Learning with a Ranking Loss Function},
  booktitle = {Sixth International Workshop on Information Integration on the Web (IIWeb-07)},
  	  year = {2007},
  address = {Vancouver, Canada},
}

Tractable Learning and Inference with High-Order Representations

2006-01-01T00:00:00+00:00

Abstract

Representing high-order interactions in data often results in large models with an intractable number of hidden variables. In these models, inference and learning must operate without instantiating the entire set of variables. This paper presents a Metropolis-Hastings sampling approach to address this issue, and proposes new methods to discriminatively estimate the proposal and target distribution of the sampler using a ranking function over configurations. We demonstrate our approach on the task of paper and author deduplication, showing that our method enables complex, advantageous representations of the data while maintaining tractable learning and inference procedures.

Citation

@inproceedings{culotta06tractable,
  author = {Aron Culotta and Andrew McCallum},
  title = {Tractable Learning and Inference with High-Order Representations},
  booktitle = {International Conference on Machine Learning Workshop on Open Problems in Statistical Relational Learning},
  address = {Pittsburgh, PA},
  year = {2006},
 	}

Practical Markov logic containing first-order quantifiers with application to identity uncertainty

2006-01-01T00:00:00+00:00

Abstract

Markov logic is a highly expressive language recently introduced to specify the connectivity of a Markov network using first-order logic. While Markov logic is capable of constructing arbitrary first-order formulae over the data, the complexity of these formulae is often limited in practice because of the size and connectivity of the resulting network. In this paper, we present approximate inference and estimation methods that incrementally instantiate portions of the network as needed to enable first-order existential and universal quantifiers in Markov logic networks. When applied to the problem of identity uncertainty, this approach results in a conditional probabilistic model that can reason about objects, combining the expressivity of recently introduced BLOG models with the predictive power of conditional training. We validate oualgorithms on the tasks of citation matching and author disambiguation.

Citation

@inproceedings{culotta06practical,
  author = {Aron Culotta and Andrew McCallum},
  title = {Practical Markov logic containing first-order quantifiers with application to identity uncertainty},
  booktitle = {Human Language Technology Workshop on Computationally Hard Problems and Joint Inference in Speech and Language Processing (HLT/NAACL)},
  year = {2006},
  month = {June},
}

Learning field compatibilities to extract database records from unstructured text

2006-01-01T00:00:00+00:00

Abstract

Named-entity recognition systems extract entities such as people, organizations, and locations from unstructured text. Rather than extract these mentions in isolation, this paper presents a record extraction system that assembles mentions into records (i.e. database tuples). We construct a probabilistic model of the compatibility between field values, then employ graph partitioning algorithms to cluster fields into cohesive records. We also investigate compatibility functions over sets of fields, rather than simply pairs of fields, to examine how higher representational power can impact performance. We apply our techniques to the task of extracting contact records from faculty and student homepages, demonstrating a 53% error reduction over baseline approaches.

Citation

@inproceedings{wick06learning,
  	  author = {Michael Wick and {\bf Aron Culotta} and Andrew McCallum},
  title = {Learning field compatibilities to extract database records from unstructured text},
  booktitle = {Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2006},
  address = {Sydney, Australia},
  pages = {603--611}
}

Integrating probabilistic extraction models and data mining to discover relations and patterns in text

2006-01-01T00:00:00+00:00

Abstract

In order for relation extraction systems to obtain human-level performance, they must be able to incorporate relational patterns inherent in the data (for example, that one’s sister is likely one’s mother’s daughter, or that children are likely to attend the same college as their parents). Hand-coding such knowledge can be time-consuming and inadequate. Additionally, there may exist many interesting, unknown relational patterns that both improve extraction performance and provide insight into text. We describe a probabilistic extraction model that provides mutual benefits to both “top-down” relational pattern discovery and “bottom-up” relation extraction.

Citation

@inproceedings{culotta06integrating,
  author = {Aron Culotta and Andrew McCallum and Jonathan Betz},
  title = {Integrating probabilistic extraction models and data mining to discover relations and patterns in text},
  shortbooktitle = {HLT-NAACL},
  booktitle = {Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (HLT/NAACL)},
  year = {2006},
  pages = {296--303},
  address = {New York, NY},
  month = {June},
}

Corrective Feedback and Persistent Learning for Information Extraction

2006-01-01T00:00:00+00:00

Abstract

To successfully embed statistical machine learning models in real world applications, two post-deployment capabilities must be provided: (1) the ability to solicit user corrections and (2) the ability to update the model from these corrections. We refer to the former capability as corrective feedback and the latter as persistent learning. While these capabilities have a natural implementation for simple classification tasks such as spam filtering, we argue that a more careful design is required for structured classification tasks. One example of a structured classification task is information extraction, in which raw text is analyzed to automatically populate a database. In this work, we augment a probabilistic information extraction system with corrective feedback and persistent learning components to assist the user in building, correcting, and updating the extraction model. We describe methods of guiding the user to incorrect predictions, suggesting the most informative fields to correct, and incorporating corrections into the inference algorithm. We also present an active learning framework that minimizes not only how many examples a user must label, but also how difficult each example is to label. We empirically validate each of the technical components in simulation and quantify the user effort saved. We conclude that more efficient corrective feedback mechanisms lead to more effective persistent learning. #### Citation

Citation

@article{culotta06corrective,
  author = {Aron Culotta and Trausti Kristjansson and Andrew McCallum and Paul Viola},
  title = {Corrective Feedback and Persistent Learning for Information Extraction},
  journal = {Artificial Intelligence},
  year = {2006},
  volume = {170},
  pages = {1101--1122},
}

Reducing labeling effort for structured prediction tasks

2005-01-01T00:00:00+00:00

Abstract

A common obstacle preventing the rapid deployment of supervised machine learning algorithms is the lack of labeled training data. This is particularly expensive to obtain for structured prediction tasks, where each training instance may have multiple, interacting labels, all of which must be correctly annotated for the instance to be of use to the learner. Traditional active learning addresses this problem by optimizing the order in which the examples are labeled to increase learning efficiency. However, this approach does not consider the difficulty of labeling each example, which can vary widely in structured prediction tasks. For example, the labeling predicted by a partially trained system may be easier to correct for some instances than for others. We propose a new active learning paradigm which reduces not only how many instances the annotator must label, but also how difficult each instance is to annotate. The system also leverages information from partially correct predictions to efficiently solicit annotations from the user. We validate this active learning framework in an interactive information extraction system, reducing the total number of annotation actions by 22%.

Citation

@inproceedings{culotta05reducing,
  author = {Aron Culotta and Andrew McCallum},
  title = {Reducing labeling effort for structured prediction tasks},
  shortbooktitle = {AAAI},
  booktitle = {The Twentieth National Conference on Artificial Intelligence (AAAI)},
  address = {Pittsburgh, PA},
  pages = {746--751},
  year = {2005},
}

Learning clusterwise similarity with first-order features

2005-01-01T00:00:00+00:00

Abstract

Many clustering problems can be reduced to the task of partitioning a weighted graph into highly-connected components. The weighted edges indicate pairwise similarity between two nodes and can often be estimated from training data. However, in many domains, there exist higher-order dependencies not captured by pairwise metrics. For example, there may exist soft constraints on aggregate features of an entire cluster, such as its size, mean or mode. We propose clusterwise similarity metrics to directly measure the cohesion of an entire cluster of points. We describe ways to learn a clusterwise metric from labeled data, using weighted, first-order features over clusters. Extending recent work equating graph partitioning with inference in graphical models, we frame this approach within a discriminatively-trained Markov network. The advantages of our approach are demonstrated on the task of coreference resolution.

Citation

@inproceedings{culotta05learning,
  author = {Aron Culotta and Andrew McCallum},
  title = {Learning clusterwise similarity with first-order features},
  booktitle = {Neural Information Processing Systems (NIPS) Workshop on the Theoretical Foundations of Clustering},
  year = {2005},
  address = {Whistler, B.C.},
  month = {December},
}

Joint deduplication of multiple record types in relational data

2005-01-01T00:00:00+00:00

Abstract

Record deduplication is the task of merging database records that refer to the same underlying entity. In relational databases, accurate deduplication for records of one type is often dependent on the merge decisions made for records of other types. Whereas nearly all previous approaches have merged records of different types independently, this work models these inter-dependencies explicitly to collectively deduplicate records of multiple types. We construct a conditional random field model of deduplication that captures these relational dependencies, and then employ a novel relational partitioning algorithm to jointly deduplicate records. We evaluate the system on two citation matching datasets, for which we deduplicate both papers and venues. We show that by collectively deduplicating paper and venue records, we obtain up to a 30% error reduction in venue deduplication, and up to a 20% error reduction in paper deduplication over competing methods.

Citation

@inproceedings{culotta05joint,
  author = {Aron Culotta and Andrew McCallum},
  title = {Joint deduplication of multiple record types in relational data},
  booktitle = {2005 ACM CIKM International Conference on Information and Knowledge Management},
  year = {2005},
  pages = {257--258},
}

Interactive information extraction with constrained conditional random fields

2004-01-01T00:00:00+00:00

Abstract

Information Extraction methods can be used to automatically “fill-in” database forms from unstructured data such as Web documents or email. State-of-the-art methods have achieved low error rates but invariably make a number of errors. The goal of an interactive information extraction system is to assist the user in filling in database fields while giving the user confidence in the integrity of the data. The user is presented with an interactive interface that allows both the rapid verification of automatic field assignments and the correction of errors. In cases where there are multiple errors, our system takes into account user corrections, and immediately propagates these constraints such that other fields are often corrected automatically. Linear-chain conditional random fields (CRFs) have been shown to perform well for information extraction and other language modelling tasks due to their ability to capture arbitrary, overlapping features of the input in a Markov model. We apply this framework with two extensions: a constrained Viterbi decoding which finds the optimal field assignments consistent with the fields explicitly specified or corrected by the user; and a mechanism for estimating the confidence of each extracted field, so that low-confidence extractions can be highlighted. Both of these mechanisms are incorporated in a novel user interface for form filling that is intuitive and speeds the entry of data—providing a 23% reduction in error due to automated corrections.

Best Paper Award (Honorable Mention)

Citation

@inproceedings{kristjannson04interactive,
  author = {Trausti Kristjannson and Aron Culotta and Paul Viola and Andrew McCallum},
  title = {Interactive information extraction with constrained conditional random fields},
  shortbooktitle = {AAAI},
  booktitle = {Nineteenth National Conference on Artificial Intelligence (AAAI)},
  address = {San Jose, CA},
  year = {2004},
}

Extracting social networks and contact information from email and the Web

2004-01-01T00:00:00+00:00

Abstract

We present an end-to-end system that extracts a user’s social network and its members’ contact information given the user’s email inbox. The system identifies unique people in email, finds their Web presence, and automatically fills the fields of a contact address book using conditional random fields—a type of probabilistic model well-suited for such information extraction tasks. By recursively calling itself on new people discovered on the Web, the system builds a social network with multiple degrees of separation from the user. Additionally, a set of expertise-describing keywords are extracted and associated with each person. We outline the collection of statistical and learning components that enable this system, and present experimental results on the real email of two users; we also present results with a simple method of learning transfer, and discuss the capabilities of the system for address-book population, expert-finding, and social network analysis.

Citation

@inproceedings{culotta04extracting,
  author = {Aron Culotta and Ron Bekkerman and Andrew McCallum},
  title = {Extracting social networks and contact information from email and the Web},
  booktitle = {First Conference on Email and Anti-Spam (CEAS)},
  address = {Mountain View, CA},
  year = {2004},
}

Dependency tree kernels for relation extraction

2004-01-01T00:00:00+00:00

Abstract

We extend previous work on tree kernels to estimate the similarity between the dependency trees of sentences. Using this kernel within a Support Vector Machine, we detect and classify relations between entities in the Automatic Content Extraction (ACE) corpus of news articles. We examine the utility of different features such as Wordnet hypernyms, parts of speech, and entity types, and find that the dependency tree kernel achieves a 20% F1 improvement over a “bag-of-words” kernel.

Citation

@inproceedings{culotta04dependency,
  author = {Aron Culotta and Jeffery Sorensen},
  title = {Dependency tree kernels for relation extraction},
  shortbooktitle = {ACL},
  booktitle = {42nd Annual Meeting of the Association for Computational Linguistics (ACL)},
  address = {Barcelona, Spain},
  year = {2004},
}

Confidence estimation for information extraction

2004-01-01T00:00:00+00:00

Abstract

Information extraction techniques automatically create structured databases from unstructured data sources, such as the Web or newswire documents. Despite the successes of these systems, accuracy will always be imperfect. For many reasons, it is highly desirable to accurately estimate the confidence the system has in the correctness of each extracted field. The information extraction system we evaluate is based on a linear-chain conditional random field (CRF), a probabilistic model which has performed well on information extraction tasks because of its ability to capture arbitrary, overlapping features of the input in a Markov model. We implement several techniques to estimate the confidence of both extracted fields and entire multi-field records, obtaining an average precision of 98% for retrieving correct fields and 87% for multi-field records.

Citation

@inproceedings{culotta04confidence,
  author = {Aron Culotta and Andrew McCallum},
  title = {Confidence estimation for information extraction},
  booktitle = {Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL)},
  address = {Boston, MA},
  year = {2004},
}