Download as PDF, PPTX
Uploaded bySuman Debnath
EFS_Integration.pdf
The document discusses Amazon EFS (Elastic File System) and how it can be used with serverless and containerized applications. Some key points discussed include: - Stateful applications benefit from durable shared storage provided by Amazon EFS for storing state. - Distributed applications like web serving and machine learning inference benefit from a shared storage layer. - Modern data-intensive applications require fast access to large volumes of data stored in Amazon EFS. - Amazon EFS can be used to migrate existing applications to the cloud or modernize applications into containerized/serverless architectures for increased scalability and efficiency. - Amazon EFS provides a shared file system that can be mounted by containers,
Technologyā¦
EFS_Integration.pdf
- 1. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. Suman Debnath Developer Advocate, Amazon EFS Amazon EFS Serverless and Container Integration
- 2. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Stateful applications need durable shared storage Availability and durability Data sharing Scalable ā¦ Containers and serverless functions are transient in nature; long-running applications can benefit from keeping state in a durable storage Distributed applications like web serving, machine learning inference, and continuous integration and delivery benefit from shared storage layer Modern data-intensive applications like analytics require fast access to large volumes of data
- 3. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Lift & shift today, accelerate modernization Migrate Lift & shift applications to the cloud Amazon EKS Amazon ECS AWS Fargate AWS Lambda Amazon SageMaker Amazon EFS Amazon EC2
- 4. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Lift & shift today, accelerate modernization Migrate Lift & shift applications to the cloud Amazon EKS Amazon ECS AWS Fargate AWS Lambda Amazon SageMaker Amazon EFS Amazon EC2 Modernize Convert existing applications and build new applications with microservices like containers and serverless
- 5. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Availability zone Availability zone AWS Lambda Mount Target Mount Target Amazon EFS file system Amazon EFS and AWS Lambda
- 6. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Availability zone Availability zone AWS Lambda Mount Target Mount Target Amazon EFS file system Elastic, shared, serverless storage Petabyte-scale elastic storage for Lambda functions Build stateful applications Low-latency Pay for what you use Serverless storage Share data across 1000s of function invocations Amazon EFS and AWS Lambda
- 7. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon EFS and AWS Lambda : How it works Create an Amazon EFS file system
- 8. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create an Amazon EFS file system Create an Access Point Amazon EFS and AWS Lambda : How it works
- 9. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Configuring the Access Point Amazon EFS and AWS Lambda : How it works
- 10. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Configuring the Access Point Creating the Lambda Function Amazon EFS and AWS Lambda : How it works
- 11. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Adding the file system Amazon EFS and AWS Lambda : How it works
- 12. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Adding the file system Select the file system Amazon EFS and AWS Lambda : How it works
- 13. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL What we are going to build ? User Request Prediction
- 14. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Letās focus on thisā¦ User Request Prediction
- 15. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Client Application Architecture
- 16. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application User Request Architecture
- 17. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application AWS Lambda User Request Invoke Architecture
- 18. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application AWS Lambda User Request Invoke Architecture Amazon EFS Loading Models
- 19. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application AWS Lambda User Request Invoke Architecture Amazon EFS Loading Models Amazon S3 Uploading Models ML Engineer
- 20. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application AWS Lambda User Request Invoke Architecture Amazon EFS Loading Models Amazon S3 Uploading Models ML Engineer AWS Lambda Trigger
- 21. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application AWS Lambda User Request Invoke Architecture Amazon EFS Loading Models Amazon S3 Uploading Models ML Engineer AWS Lambda Trigger Response Prediction
- 22. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application AWS Lambda User Request Invoke Architecture Amazon EFS Loading Models Amazon S3 Uploading Models ML Engineer AWS Lambda Trigger Response Prediction
- 23. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application AWS Lambda User Request Invoke Architecture Amazon EFS Loading Models Amazon S3 Uploading Models ML Engineer AWS Lambda Trigger Response Prediction
- 24. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon API Gateway Client Application AWS Lambda User Request Invoke Architecture Amazon EFS Loading Models Amazon S3 Uploading Models ML Engineer AWS Lambda Trigger Response Prediction app1(s3-efs) app2(ml-inference)
- 25. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Demo @ aws-samples/amazon-efs-developer-zone
- 26. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon Elastic File System (Amazon EFS) EC2 instance or on-premises server Application code Task Container 1 Container 2 Amazon ECS Amazon EC2 Pod Container 1 Container 2 Amazon EKS Amazon EC2 CSI driver AWS Fargate AWS Fargate Elastic, shared, highly-available storage Persist storage across containers Pay for what you use Serverless storage Application level security Amazon ECS and AWS Container Services
- 27. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon ECS and Amazon EFS: How it works
- 28. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create Amazon ECS Cluster Amazon ECS and Amazon EFS: How it works
- 29. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create Amazon ECS Cluster Create Amazon ECS and Amazon EFS: How it works
- 30. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create Amazon ECS Cluster Create a security group for Amazon EFS Create an Amazon EFS file system Amazon ECS and Amazon EFS: How it works
- 31. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create Amazon ECS and Amazon EFS: How it works
- 32. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create a task definition Mount Point definition Amazon ECS and Amazon EFS: How it works
- 33. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create Amazon EKS Cluster Amazon EFS CSI Driver for Kubernetes
- 34. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create Amazon EKS Cluster Create a security group for Amazon EFS Amazon EFS CSI Driver for Kubernetes
- 35. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create Amazon EKS Cluster Create a security group for Amazon EFS Create an Amazon EFS file system Amazon EFS CSI Driver for Kubernetes
- 36. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Create Amazon EKS Cluster Create a security group for Amazon EFS Create an Amazon EFS file system Deploy EFS CSI Driver (Open Source) Amazon EFS CSI Driver for Kubernetes
- 37. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Defining Storage class Amazon EFS CSI Driver for Kubernetes
- 38. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Defining Storage class Persistent Volume Claim Mounting on the container/pod Amazon EFS CSI Driver for Kubernetes
- 39. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Kubeflow with Amazon EKS and Amazon EFS Amazon EKS Training Container 2 Training Container n Amazon EFS CSI driver Pod Pod Amazon ECR Pod Training Container 1 Build the container image for training and push it to ECR Save the training dataset on Amazon EFS
- 40. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon EFS Developer Zone
- 41. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. Suman Debnath Developer Advocate, Amazon EFS Thank You
- 42. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Backup
- 43. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Goals for security and identity 1. File systems should only be mountable by applications that need them 2. Applications that mount file systems should only have access to data they need Amaz on EFS $ cat /my_app/data ### SUCCESS THIS IS MY FILE ### $ cat /someone_elses_app/data cat: /someone_elses_app/data : Permission denied
- 44. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Using IAM for file system access { āStatementā : { āEffectā : āallowā, āActionā : āelasticfilesystem:Client*ā, āPrincipalā : { āAWSā: āarn:..Lambdaā } } } Amazon EFS IAM { āStatementā : { āEffectā : āallowā, āActionā : āelasticfilesystem:Client*ā, "Resource": āfs-feedfeed" } } Task or execution role Function Container ECS EKS Lambda
- 45. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Understanding container identity ECS task Task identity (IAM role) IAM Container image Application identity User: Root Group: Root $ ls āl /efs/home drwx------ bob . BobHome drwx------ maria . MariaHome drwxrwx--- . biusers BI_Shared By default, POSIX identity comes from the container image, not the task/pod runtime
- 46. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Understanding function identity Lambda function IAM $ ls āl /efs/home drwx------ bob . BobHome drwx------ maria . MariaHome drwxrwx--- . biusers BI_Shared By default, Lambda functions have no predetermined identity Task identity (IAM role)
- 47. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL Amazon EFS access points { āNameā: āMyAppā, "FileSystemId": āfs-feedfeed", āPosixUserā: { āUidā: 123 āGidā: 123, āSecondaryGidsā: [100, 200, 300] }, āRootDirectoryā: { āPathā: ā/apps/myappā, āCreationInfoā: { āOwnerUidā: 123, āOwnerGidā: 123, āPermissionsā: ā0700ā } } } Creates application-specific directory and permissions No Amazon EC2 instance required Applications only see the data they need Enforces file system identity Root containers cannot escalate access Arbitrary users are not locked out ECS EKS Lambda
- 48. Ā© 2022, AmazonWeb Services, Inc. or its Affiliates. CONFIDENTIAL { āNameā: āMyAppā, āPosixUserā: { āUidā: 123 āGidā: 123, āSecondaryGidsā: [100, 200, 300] }, āRootDirectoryā: { āPathā: ā/apps/myappā, āCreationInfoā: { āOwnerUidā: 123, āOwnerGidā: 123, āPermissionsā: ā0700ā } } } How Amazon EFS access points work Amazon EFS with POSIX permissions āEffectā : āallowā, āActionā : āelasticfilesystem:Client*ā, āPrincipalā : { āAWSā: āapproleā }, āConditionā* : {āaccessPointArnā : āfsap-1234ā