(Part B below)

More information here: homework.md

Did you take notes? Add them here:

• Week 6a Notes by M. Ayoub C.
• Send a PR, add your notes above this line

with Terraform

Setting up a stream-based pipeline infrastructure in AWS, using Terraform

• Amazon Web Service (AWS):
  • Kinesis: Streams (Producer & Consumer)
  • Lambda: Serving API
  • S3 Bucket: Model artifacts
  • ECR: Image Registry

Further info here: Concepts of IaC and Terraform

https://www.youtube.com/watch?v=zRcLgT7Qnio&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=48

• Introduction
• Setup & Pre-Reqs
• Concepts of Terraform and IaC (reference material from previous courses)

https://www.youtube.com/watch?v=-6scXrFcPNk&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=49

• What are they?
• Creating a Kinesis module

https://www.youtube.com/watch?v=JVydd1K6R7M&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=50

• TF resources for ECR, Lambda, S3

https://www.youtube.com/watch?v=YWao0rnqVoI&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=51

• Demo: apply TF to our use-case, manually deploy data dependencies & test
• Recap: IaC, Terraform, next steps

Installation:

• aws-cli (both versions are fine)
• terraform client

Configuration:

1. If you've already created an AWS account, head to the IAM section, generate your secret-key, and download it locally. Instructions

2. Configure aws-cli with your downloaded AWS secret keys:

      $ aws configure
      AWS Access Key ID [None]: xxx
      AWS Secret Access Key [None]: xxx
      Default region name [None]: eu-west-1
      Default output format [None]:

3. Verify aws config:

     $ aws sts get-caller-identity

4. (Optional) Configuring with aws profile: here and here

Additional material on understanding Terraform concepts here: Reference Material

with GitHub Actions

• Automate a complete CI/CD pipeline using GitHub Actions to automatically trigger jobs to build, test, and deploy our service to Lambda for every new commit/code change to our repository.
• The goal of our CI/CD pipeline is to execute tests, build and push container image to a registry, and update our lambda service for every commit to the GitHub repository.

Further info here: Concepts of CI/CD and GitHub Actions

https://www.youtube.com/watch?v=OMwwZ0Z_cdk&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=52

• Architecture (Ride Predictions)
• What are GitHub Workflows?

https://www.youtube.com/watch?v=xkTWF9c33mU&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=53

• ci-tests.yml
  • Automate sections from tests: Env setup, Unit test, Integration test, Terraform plan
  • Create a CI workflow for on-pull-request to develop branch
  • Execute demo

https://www.youtube.com/watch?v=jCNxqXCKh2s&list=PL3MmuxUbc_hIUISrluw_A7wDSmfOhErJK&index=54

• cd-deploy.yml
  • Automate sections from tests: Terraform plan, Terraform apply, Docker build & ECR push, Update Lambda config
  • Create a CD workflow for on-push-to to develop branch
  • Execute demo

Did you take notes? Add them here:

• Send a PR, add your notes above this line