© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

AIM226-S

SPONSORED BY DATAROBOT

Sony Interactive Entertainment:

Generative and predictive AI on
AWS
Lisa Aguilar Simon Nicoud Francisco Fatore
she/her he/him he/him
VP Industry Field CTOs, Sr. ML Engineering Manager Sr. ML Engineer
and Product Solutions Sony Interactive Sony Interactive
DataRobot Entertainment Entertainment

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Speakers

Lisa Aguilar Simon Nicoud Francisco Fatore

she/her he/him he/him
VP Industry Field CTOs, Sr. ML Engineering Manager Sr. ML Engineer
and Product Solutions

Confidential. ©2023 DataRobot, Inc. – All rights reserved

Confidential. ©2022 DataRobot, Inc. – All rights reserved
3
3
About DataRobot

BUILT FOR LEADERS, STRATEGIC TECHNOLOGY

EXPERTISE
DATA TEAMS & DEVELOPERS PARTNER OF CHOICE

Pure-Play AI Lifecycle
Management Platform
Generative and Predictive AI 500+ Engineers &
data scientists

Predictions created Years of AI R/D

1T using DataRobot 10 with a focus on value

RECOGNIZED BY OUR
AI projects delivered Engineering hours CUSTOMERS & THE MARKET
1M using DataRobot 1.6M to build the platform

Models in production & Machine learning patents

15k monitored in a single client 80+ & innovations Data Science
and Machine
Learning Platform

Confidential. ©2023 DataRobot, Inc. – All rights reserved

Confidential. ©2023 DataRobot, Inc. – All rights reserved 4
 PLACEHOLDER For NEW DIAGRAM

Confidential. ©2023 DataRobot, Inc. – All rights reserved

Confidential. ©2022 DataRobot, Inc. – All rights reserved
5
5
AI is Evolving at a Pace Faster LLaMA

Than Teams Can Apply it

GPT-3.5

GPT-4

GPT-3.5 fine-
tuning
Cerebras-GPT
Stable Beluga
LaMDA Falcon 40B

GPT-NeoX Mosaic

Chinchilla Amazon Bedrock

PaLM Dolly
GPT-Neo
OPT PanGu-Sigma
GPT-J
YaLM BloombergGPT
Megatron-TuringNLG Minerva
OpenAssistant
Ernie 3.0 Titan BLOOM
Claude Galactica Jurassic-2
GPT-2 GLAM AlexaTM PaLM-2
BERT XLNET Da-Vinci Gopher GPT-3 LLaMA-2

2018 2019 2020 Q1 2023 Q2 2023 Q3 2023

Confidential. ©2023 DataRobot, Inc. – All rights reserved 6
AI Sprawl Is Accelerating & Silos Are Increasing
254
335
Average number of
New gen AI startups software applications

80%
within an enterprise
142+ developing applications

Open source LLMs 17 Of organizations

use multiple cloud
providers

Hosted vector
DB options

Confidential. ©2023 DataRobot, Inc. – All rights reserved

Confidential. ©2022 DataRobot, Inc. – All rights reserved
7
7
 The Canonical GenAI Stack Governance
Human Feedback

Playground/Experimentation Grounding Data Governance/Access Control

Grounding Data
Embedding
Data Pipeline
Model Embedding
Groundin Data Pipeline Data OAuth/IAM Approval Flows
Model
Assessment Data Vector Prompt g Access
Database Strategy Strategy
Vector Document Model/Plugin
Assessment Model/Plugins
Database Access Access
Use Case Framing Models
LLM

Retrieval

Orchestration
Generation Logic Champion/Challenger
Prompt Grounding Data
Strategy Context
Model
Materialized Prompt
Agent Logic & Model/Plugin Access
State Output
User Query

Application Monitoring/Observability LLM APIs and Hosting

Query
Webapps/ Guardrail Response Open Source
Validation Proprietary API
Streamlit Model API
Output Audited
Response
Chatbots/ Feedback Opinionated
Audit Model Logging Cloud Provider
Slack Capture Cloud
Feedback
Feedback

Confidential. ©2023 DataRobot, Inc. – All rights reserved 8

 Build
Governance
Human Feedback

Playground/Experimentation Grounding Data Governance/Access Control

Grounding Data
Embedding
Data Pipeline
Model Embedding
Groundin Data Pipeline Data OAuth/IAM Approval Flows
Model
Assessment Data Vector Prompt g Access
Database Strategy Strategy
Vector Document Model/Plugin
Assessment Model/Plugins
Database Access Access
Use Case Framing Models
LLM

Retrieval

Orchestration
Generation Logic Champion/Challenger
Prompt Grounding Data
Strategy Context
Model
Materialized Prompt
Agent Logic & Model/Plugin Access
State Output
User Query

Application Monitoring/Observability LLM APIs and Hosting

Query
Webapps/ Guardrail Response Open Source
Validation Proprietary API
Streamlit Model API
Output Audited
Response
Chatbots/ Feedback Opinionated
Audit Model Logging Cloud Provider
Slack Capture Cloud
Feedback
Feedback

Confidential. ©2023 DataRobot, Inc. – All rights reserved 9

 Operate
Governance
Human Feedback

Playground/Experimentation Grounding Data Governance/Access Control

Grounding Data
Embedding
Data Pipeline
Model Embedding
Groundin Data Pipeline Data OAuth/IAM Approval Flows
Model
Assessment Data Vector Prompt g Access
Database Strategy Strategy
Vector Document Model/Plugin
Assessment Model/Plugins
Database Access Access
Use Case Framing Models
LLM

Retrieval

Orchestration
Generation Logic Champion/Challenger
Prompt Grounding Data
Strategy Context
Model
Materialized Prompt
Agent Logic & Model/Plugin Access
State Output
User Query

Application Monitoring/Observability LLM APIs and Hosting

Query
Webapps/ Guardrail Response Open Source
Validation Proprietary API
Streamlit Model API
Output Audited
Response
Chatbots/ Feedback Opinionated
Audit Model Logging Cloud Provider
Slack Capture Cloud
Feedback
Feedback

Confidential. ©2023 DataRobot, Inc. – All rights reserved 10

 Govern
Governance
Human Feedback

Playground/Experimentation Grounding Data Governance/Access Control

Grounding Data
Embedding
Data Pipeline
Model Embedding
Groundin Data Pipeline Data OAuth/IAM Approval Flows
Model
Assessment Data Vector Prompt g Access
Database Strategy Strategy
Vector Document Model/Plugin
Assessment Model/Plugins
Database Access Access
Use Case Framing Models
LLM

Retrieval

Orchestration
Generation Logic Champion/Challenger
Prompt Grounding Data
Strategy Context
Model
Materialized Prompt
Agent Logic & Model/Plugin Access
State Output
User Query

Application Monitoring/Observability LLM APIs and Hosting

Query
Webapps/ Guardrail Response Open Source
Validation Proprietary API
Streamlit Model API
Output Audited
Response
Chatbots/ Feedback Opinionated
Audit Model Logging Cloud Provider
Slack Capture Cloud
Feedback
Feedback

Confidential. ©2023 DataRobot, Inc. – All rights reserved 11

 Simon Nicoud Francisco Fatore
he/him he/him
Sr. ML Engineering Manager Sr. ML Engineer

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
 Our ML pipeline journey
From rules to generative AI, on premises to AWS

• Our managed services philosophy

• Where does DataRobot fit in, and

organizational challenges to gain its adoption

• Power of automation/reusability

©2023 Sony Interactive Entertainment LLC

 Predictive AI use cases vs. generative AI

• Current predictive use cases for our ML pipeline

• Generative AI use cases fall into 4 different

categories, and we are exploring areas in each:

• Chat

• Search

• Content generation

• Associative reasoning

• Ways to harness interest in generative AI to

get started

©2023 Sony Interactive Entertainment LLC

 KPIs for AI projects: How to prove their worth?

• Cost savings (productivity/accuracy/quality improvements)

• Business impact (increased revenue, increased employee retention)

• Customer satisfaction

©2023 Sony Interactive Entertainment LLC

 AI operations plan

Things you need to think about before operationalizing a generative AI model:

• Infrastructure – Is it appropriately scalable for the use case?

• Quality/Effectiveness – Measure against KPIs

• Guardrails – Do we have controls in place against unpredicted model output?

• Audits/Compliance – Is customer data secure? Are model outputs within guardrails?

• Upgrade Process – How are model improvements deployed/compared against current?

• Monitoring/Alerting – How do we know if something is going wrong and what do we do about it?

©2023 Sony Interactive Entertainment LLC

ML
MLPIPELINE
Pipeline AT SONY INTERACTIVE ENTERTAINMENT

Amazon
DynamoDB
Data Flink Amazon Kinesis
Sources List Manager
Data Streams

Aggregations
Service

Transactions

Amazon S3 Amazon ECR Amazon EKS

Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact CI/CD Scoring Service
(JAR)

Gameplay

Batch Evaluation Leaderboard Promotion Inline

Shared Data Scores Metrics Decision Decisioning
Sources

AWS Lambda Amazon Kinesis

©2023 Sony Interactive Entertainment LLC

DATA SOURCES
ML Pipeline

Data
Sources

Transactions

Fraud Data

Gameplay

Shared Data
Sources

©2023 Sony Interactive Entertainment LLC

DATA PREP
ML Pipeline

Data
Sources

Transactions

Modelling
Fraud Data Dataset

Gameplay

Shared Data
Sources

©2023 Sony Interactive Entertainment LLC

MODELING
ML Pipeline

Data
Sources

Transactions

Modelling
Fraud Data Dataset DataRobot
AI Platform

Gameplay

Shared Data
Sources

©2023 Sony Interactive Entertainment LLC

MODELING
ML Pipeline

Data
Sources

Transactions

Modelling
Fraud Data Dataset DataRobot
AI Platform

Gameplay

Shared Data
Sources

©2023 Sony Interactive Entertainment LLC

MODELING
ML Pipeline

Data
Sources

Transactions

Modelling
Fraud Data Dataset DataRobot
AI Platform

Gameplay

Shared Data
Sources

©2023 Sony Interactive Entertainment LLC

MODEL SELECTION
ML Pipeline

Data
Sources

Transactions

Amazon S3
Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact
(JAR)

Gameplay

Shared Data
Sources

©2023 Sony Interactive Entertainment LLC

MODEL SELECTION
ML Pipeline

Data
Sources

Transactions

Amazon S3
Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact
(JAR)

Gameplay

Shared Data
Sources

©2023 Sony Interactive Entertainment LLC

MODEL SELECTION
ML Pipeline

Data
Sources

Transactions

Amazon S3
Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact
(JAR)

Gameplay

Shared Data
Sources

©2023 Sony Interactive Entertainment LLC

BATCH SCORING
ML Pipeline

Data
Sources

Transactions

Amazon S3
Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact
(JAR)

Gameplay

Batch
Shared Data Scores
Sources

©2023 Sony Interactive Entertainment LLC

MODEL EVALUATION
ML Pipeline

Data
Sources

Transactions

Amazon S3
Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact
(JAR)

Gameplay

Batch Evaluation Leaderboard

Shared Data Scores Metrics
Sources

©2023 Sony Interactive Entertainment LLC

MODEL EVALUATION
ML Pipeline

Data
Sources

Transactions

Amazon S3
Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact
(JAR)

Gameplay

Batch Evaluation Leaderboard Promotion

Shared Data Scores Metrics Decision
Sources

©2023 Sony Interactive Entertainment LLC

MODEL DEPLOYMENT
ML Pipeline

Data
Sources

Transactions

Amazon S3 Amazon ECR Amazon EKS

Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact CI/CD Scoring Service
(JAR)

Gameplay

Batch Evaluation Leaderboard Promotion

Shared Data Scores Metrics Decision
Sources

©2023 Sony Interactive Entertainment LLC

MODEL INFERENCE
ML Pipeline

Data
Sources

Transactions

Amazon S3 Amazon ECR Amazon EKS

Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact CI/CD Scoring Service
(JAR)

Gameplay

Batch Evaluation Leaderboard Promotion Inline

Shared Data Scores Metrics Decision Decisioning
Sources

©2023 Sony Interactive Entertainment LLC

REAL-TIME
ML Pipeline DATA

Amazon
DynamoDB
Data
Sources List Manager

Transactions

Amazon S3 Amazon ECR Amazon EKS

Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact CI/CD Scoring Service
(JAR)

Gameplay

Batch Evaluation Leaderboard Promotion Inline

Shared Data Scores Metrics Decision Decisioning
Sources

©2023 Sony Interactive Entertainment LLC

REAL-TIME
ML Pipeline DATA

Amazon
DynamoDB
Data Flink Amazon Kinesis
Sources List Manager
Data Streams

Aggregations
Service

Transactions

Amazon S3 Amazon ECR Amazon EKS

Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact CI/CD Scoring Service
(JAR)

Gameplay

Batch Evaluation Leaderboard Promotion Inline

Shared Data Scores Metrics Decision Decisioning
Sources

©2023 Sony Interactive Entertainment LLC

MODEL INFERENCE
ML Pipeline

Amazon
DynamoDB
Data Flink Amazon Kinesis
Sources List Manager
Data Streams

Aggregations
Service

Transactions

Amazon S3 Amazon ECR Amazon EKS

Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact CI/CD Scoring Service
(JAR)

Gameplay

Batch Evaluation Leaderboard Promotion Inline

Shared Data Scores Metrics Decision Decisioning
Sources

©2023 Sony Interactive Entertainment LLC

ML
MLPIPELINE
Pipeline AT SONY INTERACTIVE ENTERTAINMENT

Amazon
DynamoDB
Data Flink Amazon Kinesis
Sources List Manager
Data Streams

Aggregations
Service

Transactions

Amazon S3 Amazon ECR Amazon EKS

Modelling
Fraud Data Dataset DataRobot
AI Platform
Model Artifact CI/CD Scoring Service
(JAR)

Gameplay

Batch Evaluation Leaderboard Promotion Inline

Shared Data Scores Metrics Decision Decisioning
Sources

AWS Lambda Amazon Kinesis

©2023 Sony Interactive Entertainment LLC

Final takeaways +

Technical Debt is Prepare for New Cost, ROI and Value

a Real Threat; Operations will Take on a New
Optionality and and Observability Meaning and New
Flexibility are Key Process Accountability

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you! Please complete the session
survey in the mobile app

© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.