Applied Analytics in an Organizational Context
Module 12
November 21, 2024
Applied Analytics Program
Agenda
Human Resources - Talent Describe the ways in which analytics
Analytics is transforming HR
Evaluate the cultural impact of
talent analytics
In the News…
Experiments were designed by LinkedIn as part of the company’s
continuing efforts to improve the relevance of its “People You May Know”
algorithm, which suggests new connections to members.
As LinkedIn tried out different versions of their PYMK algorithms, it
created random variations in the prevalence of weak ties in the
professional networks of more than 20 million LinkedIn users over a 5-
year period, during which ∼2 billion new ties and 600,000 job changes
were recorded.
The researchers analyzed how LinkedIn’s algorithmic changes had
affected users’ job mobility. They found that moderately weak social ties
on LinkedIn proved twice as effective in securing employment as stronger
social ties.
Moderately weak contacts — people with whom LinkedIn members
shared only 10 mutual connections — proved much more productive for
job hunting than stronger contacts with whom users shared more than 20
mutual connections
https://www.nytimes.com/2022/09/24/business/linkedin-social-experiments.html
https://www-science-org.ezproxy.cul.columbia.edu/doi/epdf/10.1126/science.add0692
Talent Analytics
Link HR processes to organization’s financial outcomes
• Total human capital cost accounts for 60-70% of total companies’
expenses
• Analytics allow the talent management life cycle to be an actionable
combination of art (intuition and experience) and science (data
intelligence) for human capital management
• Business case for HR
Types of Talent Analytics
Descriptive – Reporting; retrospective
• How many people did we hire last year?
• What was our attrition rate over the past 3 years?
Diagnostic – ID causes
• Were trends different if we recruited talent from different
schools?
Predictive – future oriented
• What do we anticipate our attrition to be this year?
• Where should we search for key talent?
• How can we be prepared for leader succession?
Applications of Talent Analytics
• Optimizing the hiring process
• Improving diversity
• Identifying characteristics of employees
most likely to stay
• Finetuning performance measurement
• Developing promotion and retention
strategies
• Enhancing leadership effectiveness
Unconscious Bias
Unconscious bias - unconscious feelings towards other people that
play a strong part in influencing our judgement of certain people and
groups, away from being balanced or even-handed
Impact of biases:
• Perception – how we see people and perceive reality.
• Attitude – how we react towards certain people.
• Behaviors – how receptive/friendly we are towards certain people.
• Attention – which aspects of a person we pay most attention to.
• Listening Skills – how much we actively listen to what certain people say.
• Micro-affirmations – how much or how little we comfort certain people
in certain situations.
Unconscious Bias and Diversity
Common biases in recruitment:
• Conformity bias – group peer pressure
• Beauty bias – physical attributes
• Affinity bias – commonality
• Halo effect – carryover of one positive attribute
• Horns effect – carryover of one negative attribute
• Similarity bias – people who are like us
• Contrast effect – ranking of people
• Attribution bias – luck and personality as contributors to success and failure
• Confirmation bias – evidence to support our judgments
https://www.socialtalent.com/blog/recruitment/9-types-of-unconscious-bias
AI and Bias
By using machine learning algorithms that are trained to look for the skills, education and
work experience that match job requirements, companies can avoid many of the bias-
related problems that plague human-driven hiring decisions.
AI systems are able to select a more diverse candidate pool and avoid biases that stem
from hiring managers looking within their own immediate networks to find candidates for
positions.
Machine learning systems are only as good as their training data. Companies that leverage
AI for recruiting must make sure that they don't train their systems using biased data.
Hiring Process
One of the biggest challenges that companies face is filtering through large Amazon opened 250 new
amounts of resumes, online job profiles and inbound inquiries. fulfillment centers, air hubs,
• In especially high-turnover industries, e.g. retail, warehouse logistics and and delivery stations in
food services, companies face significant challenges when managing the 2021.
hiring process. • Hired 450,000 people
since the start of the
pandemic
Machine learning and AI methods automatically peruse online job boards,
• Looking to fill 125,000
analyze resumes and filter email communications from applicants.
jobs in fulfillment and
• Sort and analyze resumes and respond automatically to those that are transportation divisions,
not a fit for the organization; sort best candidates to the top of the pile and 40,000 corporate
• Create profiles of existing, high-performing employees in particular and tech jobs across 220
positions and search for candidates that have similar profiles locations in the U.S.
• Create and manage online job postings. Create job postings customized
for different platforms and handle the responses for each job site in
different ways. Craft job ads to optimize for the specifics of the position,
location, skill set, etc.
https://fortune.com/2021/09/14/amazon-hiring-125000-workers/
Candidate Evaluation Process
AI chatbots directly engage with candidates to schedule interview times and
ask standard interview questions prior to any meeting with company
management.
• AI system can rank and score the candidates based on those standard
questions in order to further help employers with the process.
AI chatbots can answer candidate questions on specifics of the job, such as
standard company hours, company information or other matters.
• Candidates can have their questions answered at any hour of the day
• Hiring managers don't have to get bogged down with responding to
multiple inquiries
Candidate Assessment
Pymetrics – platform to assess person's cognitive, social and behavioral abilities
• AI that measures traits like multi-tasking, altruism, and problem-solving
• Assess how likely a candidate is to succeed in a particular role by comparing
their results to a baseline of the company's best employees
• Gives applicants a chance to be assessed on potential, rather than only
school or prior work experience
Impact
• Client companies have more than doubled the percentage of candidates they hire out
of those they invite for in-person interviews
• One-year retention rates have increased by 30-60%
• Companies report that job performance has improved among newly hired candidates
• Increase in diversity - algorithms test for and remove ethnic or gender biases that arise
Schmidt, F.l. and Hunter, J.E. (1998) Psychological Bulletin, 124, pp 262-274
https://www.practiceaptitudetests.com/testing-publishers/pymetrics/
In the News…
Some times a small number of data points are extremely useful,
and that as data points are added they become increasingly less
useful.
A recent paper modeled the issue of statistical discrimination,
an economic theory that argues one reason for the persistence
of discrimination was that a company seeking to maximize
profit as its only goal would use all available information about
job candidates. A hiring manager, in pursuit of that goal, might
use race or gender, even if subconsciously, to make the best
prediction.
Race and gender are poor predictors of performance
compared with skill. But they might be more easily observed,
which might lead a hiring manager to overweight them. In this
case statistical discrimination, which recommends using both
types of information to get the best prediction, does the
https://www.wsj.com/articles/when-it-comes-to-data-sometimes-less-is-
more-11667554203
opposite.
https://pubsonline.informs.org/doi/full/10.1287/orsc.2022.1626
Candidate Interviews
Increasing number of companies are applying facial recognition technology and other
methods to perform online video screening of job candidates, usually as a first step in
screening
• Advantages from the perspective of efficiency and standardization
• Potential risk for bias when facial recognition models are inappropriately trained,
e.g.
- Qualified candidates not making it through to the next round of interviews because
the system weighted mannerisms and emotional responses from men more
favorably than those of women
- Systems might lack training data sets that include minority faces
HireVue
Artificial Intelligence: The Robots Are Now Hiring | Moving Upstream
Some Fortune 500 companies are using tools that deploy artificial intelligence to weed out job applicants. But is this practice fair? In this episode of Moving Upstream, WSJ's Jason Bellini investigates.
Watch for new episodes of Moving Upstream this fall.
To be notified of future episodes and updates on the series, sign up here: https://confirmsubscription.com/h/d/CABE7097D284D80F
AI platform to enable employers to
Don’t miss a WSJ video, subscribe here: http://bit.ly/14Q81Xy
More from the Wall Street Journal:
Visit WSJ.com: http://www.wsj.com
Visit the WSJ Video Center: https://wsj.com/video
On Facebook: https://www.facebook.com/pg/wsj/videos/
On Twitter: https://twitter.com/WSJ
On Snapchat: https://on.wsj.com/2ratjSM
interview job applicants on camera, using
rate videos of each candidate according
to verbal and nonverbal cues
• Used by > 700 companies globally
Discussion: System’s ratings reflect the
previous preferences of hiring managers.
• Is this good or bad?
• What are the risks?
• How could the risks be mitigated?
https://www.wsj.com/articles/artificial-intelligence-the-robots-are-now-hiring-moving-upstream-1537435820
https://youtu.be/8QEK7B9GUhM?si=f589EUrNegpLWBpc
Areas for Consideration and Concern
Understanding of meaningful difference between high performing and low
performing employees against attributes used for training
Privacy concerns with using social media data
Access to sufficiently large volumes of data—number of hires, performance
appraisals, etc. required to make accurate predictions.
Analytic approaches are retrospective, which may be different than future needs
Risk of unintended selection bias because of correlation between variables
Cappelli, P. (2019) Harvard Business Review 97, 48-58.
Considerations Error Rate
One widely used facial-recognition data set estimated to be more
than 75 percent male and more than 80 percent white 1%
African Americans were most likely to be singled out in face
recognition networks used by law enforcement because they are
disproportionately represented in mug-shot databases.
7%
How do we know that a qualified candidate, whose verbal and
nonverbal cues tied to age, gender, sexual orientation or race do
not fit the training data of example high performers, will not being
judged lower than a similar candidate who fits the in-group?
12%
Check system continuously to make sure it is not amplifying the
biases that informed previous in hiring decisions
35%
https://www.nytimes.com/2018/02/09/technology/facial-recognition-race-artificial-intelligence.html
https://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212
Bias in hiring
https://www.techtarget.com/searchhrsoftware/tip/AI-hiring-bias-Everything-you-need-to-
know#:~:text=The%20most%20common%20form%20of,biases%20hidden%20in%20the%20data.
In the News…
NYC law requires companies using A.I.
software in hiring to:
• Notify candidates that an automated
system is being used
• Have independent auditors check the
technology annually for bias
Candidates can request and be told what
data is being collected and analyzed.
Companies will be fined for violations.
https://www.nytimes.com/2023/05/25/technology/ai-hiring-law-new-york.html?searchResultPosition=1
Unilever Interview Process
Use internet to recruit beyond small number of college campuses.
Targeted advertisements on Facebook and career-advice sites. We tried the AI software companies like Goldman Sachs and Unilever use to analyze job applicants
We got to experience HireVue's AI technology that allows companies to sort and grade video job applicants.
--------------------------------------------------
Follow BI Video on Twitter: http://bit.ly/1oS68Zs
Follow BI on Facebook: http://bit.ly/1W9Lk0n
Read more: http://www.businessinsider.com/
--------------------------------------------------
Business Insider is the fastest growing business news site in the US. Our mission: to tell you all you need to know about the big world around you. The BI Video team focuses on technology, strategy and science with an emphasis on unique storytelling and data that appeals to the next generation of leaders – the digital generation.
Unilever pulls information from the candidate’s LinkedIn profile to fill
out the application. Algorithm scans applications to surface candidates
who meet a given role’s requirements. The software weeds out more
than half of the pool
Candidates play a set of Pymetric games designed to assess skills like
concentration under pressure and short-term memory. Pymetrics
score matches candidates to 7 internal divisions.
Top third are invited to submit video interviews, answering questions
about how they would respond to business challenges encountered on
the job.
Unilever Interview Process
AI filters 60% to 80% of candidates.When Unilever tested the system in late 2016, applicant
volume increased by 100%, to 275,000 candidates globally. At each step of the process,
algorithms narrowed that talent pool, eventually to just 300 final-round candidates in the U.S.
and Canada who were interviewed in-person
80% got offers
80% accepted
Average time to hire reduced from 4 months to 4 weeks
Increased diversity of hires
https://www.wsj.com/articles/in-unilevers-radical-hiring-experiment-resumes-are-out-algorithms-are-in-1498478400
http://www.businessinsider.com/unilever-artificial-intelligence-hiring-process-2017-6
https://www.fastcompany.com/90205539/moneyball-for-business-how-ai-is-changing-talent-management
Amazon AI-based Recruiting
2014-2015: 500 models focused on specific job functions and
locations
• Trained to vet applicants by observing patterns in resumes
submitted to the company over a 10-year period based on
50,000 terms
• Little significance to skills that were common across IT
applicants, such as the ability to write various computer
codes
• Favored candidates who described themselves using
verbs more commonly found on male engineers’
resumes, such as “executed” and “captured,” one person
said.
• Penalized resumes that included the word “women’s”
• Downgraded graduates of two all-women’s colleges
2017: Disbanded team and effort
https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-
against-women-idUSKCN1MK08G
Flight Risk Predictive Modeling
Predict when top performers are considering leaving
Correlate patterns of employees who have voluntarily left
the organization with those of existing workers to create
a flight risk score Flight Risk prediction model
• Engagement levels at work • The 40 percent of HP employees
assigned highest scores includes 75
• Time since last promotion
percent of those who will quit
• Absenteeism • Estimates $300 million in potential
• Changes in performance review ratings savings with respect to staff
• Activity on LinkedIn profiles replacement and productivity loss
globally.
https://www.shrm.org/resourcesandtools/hr-topics/technology/pages/dangers-using-predictive-analytics-employee-flight-risk.aspx
https://www.aihr.com/blog/predictive-analytics-human-resources/ /
Cisco HR Analytics (click link)
https://www.wsj.com/video/series/open-office/inside-ciscos-nyc-office-where-5000-data-points-are-being-
collected/A432D9D6-2F6E-436F-8624-C69F69E6797F
LinkedIn Chat Bot
LinkedIn is beta testing ChatGPT to personalize
a customer’s career actions
• Nudges
• Suggestions for skills to build
• Professionals in an individual’s network to
reach out to with questions
AI will
• Analyze an individual’s feed posts and reveal the key salient opportunities
• Provide expert articles and discussions on trending topics from LinkedIn and across the web.
• Assess if a particular job is a good fit and identify the best way to position for it
https://www.linkedin.com/pulse/celebrating-1-billion-members-our-new-ai-powered-linkedin-tomer-cohen-26vre/
HR Analytics – Confusion Matrix
predicted
ŷ=1 ŷ=0
Sensitivity = true positive rate
Sensitivity • When qualified, how often does model
Y = 1 True Positive (TP) False Negative (FN)
TP/(y=1) predict qualified?
actual
• Sensitivity = true positive/(true positive
+ false negative)
Specificity
Y = 0 False Positive (FP) True Negative (TN)
TN/ (y=0) Specificity = true negative rate
• When not qualified, how often does
model predict not qualified?
Precision Accuracy • Specificity = true negative/(true negative
TP/ (ŷ=1) (TP+TN)/ Total + false positive)
Precision = predictive value Accuracy = rate of correctly predicted
• When model predicts qualified, positives & negatives
how often is it correct? • Overall, how often is model correct?
• Precision = true Positive/(true • Accuracy = (true positive + true
positive + false positive) negative)/ (true positive + false positive +
true negative + false negative)
Additional Resource: https://mlu-explain.github.io/precision-recall/
Assignment 4 HR Analytics - Predictions
True Predictions:
True positive – predicted positive and actually positive
True negative – predicted negative and actually negative
False Predictions:
False positive – predicted positive but actually negative Which type of error is worse depends
False negative – predicted negative but actually positive on the consequences of the error
Note that you are judging the error of the prediction
Breakout questions
Predicted
Actual
Predicted
Actual
Predicted
Actual
Predicted
Actual
Next Class
Assignment 4 due on Dec 5 Thursday at 7:30pm
Corporate Social Responsibility
Pucker, K. P. (2021) Overselling Sustainability Reporting. Harvard Business Review 99, 134-143.
Serafeim, G. (2020). Social-Impact Efforts that Create Real Value. Harvard Business Review 98,
38-48.
Denning, S. (2017). Making sense of shareholder value: 'The World’s Dumbest Idea'. Forbes.
