(Part B below)

More information here: homework.md

• Amazon Web Service (AWS):
  • Kinesis: Streams (Producer & Consumer)
  • Lambda: Serving API
  • S3 Bucket: Model artifacts
  • ECR: Image Registry

Setting up a stream-based pipeline infrastructure in AWS, using Terraform

1. Video 1: Terraform - Introduction

   • Introduction
   • Setup & Pre-Reqs
   • Concepts of Terraform and IaC (reference material from previous courses)

2. Video 2: Terraform - Modules and Outputs variables

   • What are they?
   • Creating a Kinesis module

3. Video 3: Build an e2e workflow for Ride Predictions

   • TF resources for ECR, Lambda, S3

4. Video 4: Test the pipeline e2e

   • Demo: apply TF to our use-case, manually deploy data dependencies & test (This video is TBD)

Installation:

• aws-cli (both versions are fine)
• terraform client

Configuration:

1. If you've already created an AWS account, head to the IAM section, generate your secret-key, and download it locally. Instructions

2. Configure aws-cli with your downloaded AWS secret keys:

      $ aws configure
      AWS Access Key ID [None]: xxx
      AWS Secret Access Key [None]: xxx
      Default region name [None]: eu-west-1
      Default output format [None]:

3. Verify aws config:

     $ aws sts get-caller-identity

4. (Optional) Configuring with aws profile: here and here

(refer to previous videos, links in README)

1. For an introduction to Terraform and IaC concepts, please refer to this video (from the DE Zoomcamp), especially the sections in the time-codes:

   • 00:00 Introduction
   • 00:35 What is Terraform?
   • 01:10 What is IaC?
   • 01:43 Advantages of IaC
   • 14:48 Installing Terraform
   • 02:28 More on Installing Terraform

2. For a quickstart tutorial, and understanding the main components of a basic Terraform script, please refer to this video (from the DE Zoomcamp). Please note that this example uses GCP as a cloud provider, while for MLOps Zoomcamp we are using AWS.

   • 00:00 Introduction
   • 00:20 .terraform-version
   • 01:04 main.tf
   • 01:23 terraform declaration
   • 03:25 provider plugins
   • 04:00 resource example - google_storage_bucket
   • 05:42 provider credentials
   • 06:34 variables.tf
   • 10:54 overview of terraform commands
   • 13:35 running terraform commands
   • 18:08 recap

CI/CD w/ GitHub Actions

• What are GitHub workflows?

• test-pr-pipeline.yml

  • Automate sections from tests: Env setup, Unit test, Integration test, Terraform plan
  • Create a CI for on-pull-request to develop branch
  • Execute demo

• deploy-pipeline.yml

  • Automate sections from tests: Terraform plan, Terraform apply, Docker build & ECR push, Update Lambda config
  • Create a CI for on-push-to to develop branch
  • Execute demo

Other material:

Deploying Serverless Machine Learning with AWS (one of my previous videos explaining Lambda integration with Docker images): https://www.youtube.com/watch?v=79B8AOKkpho&t=689s

Did you take notes? Add them here:

• Send a PR, add your notes above this line