Uploaded byJon Todd
ECS and Docker at Okta
The document details Jon Todd's insights on leveraging Okta's services with a focus on ECS and Docker for managing microservices. It outlines the architecture, DevOps practices, security measures, and operational procedures needed for deploying applications with reliability and efficiency. Key takeaways include the importance of automation, scaling, and maintaining zero downtime during deployments.
Technologyā¦
In this document
Powered by AI
Jon Todd introduces himself as the Chief Architect at Okta, discussing ECS and Docker on August 2, 2016.
Okta is a popular service used by millions of individuals and thousands of enterprises, notably for Adobeās Creative Cloud.
Okta serves a wide range of industries including Finance, Technology, Health, and more, indicating its adaptability.
Describes various Okta features including Mobility Management, SSO, MFA, Directory Integration, providing secure access for all applications.
Okta is presented as the most reliable IDaaS, with global data centers, highlighting its redundancy and scalability.
Discussion of the challenges faced in defining micro-services and the need for development and operational stability.
Key requirements for deployments dwell on immutability, 0-downtime deployments, and compliance necessary for a robust system.
The selection of effective technology components such as container frameworks and continuous integration tools in DevOps.
Docker and ECS address issues like repeatability, stability, and compliance in managing applications efficiently.
Explains ECS concepts including task definitions, services, clusters, and continuous integration workflows for development.
Discusses dynamic scaling with ECS, cost savings using spot instances, and various termination policies for resource management.
Highlights ECS efficiency in operations with a cluster size of over 150 instances, emphasizing deployment strategies.
Focus on micro-services architecture with ECS and methods for 0-downtime testing to ensure smooth operations.
Technical parameters including ECS configuration, cluster setups, and performance metrics for operation evaluations.
Details the workflow structure involving deployment configurations and definitions provided by developers for ECS tasks.
Security measures and conventions for managing container updates, versions, and integration tests for reliable operations.
Emphasizes the importance of logging, monitoring metrics, and highlights feature requests to improve ECS functionalities.
Transition from challenges to an effective automated pipeline with a final opportunity for audience questions.
More Related Content
Similar to ECS and Docker at Okta
ECS and Docker at Okta
- 1. Jon Todd, ChiefArchitect ECS and Docker @Okta August 2, 2016 @JonToddDotCom
- 2.
- 3. Millions of peopleuse Okta every dayMillions of people use Okta every day
- 4. Thousands of enterprisesuse Okta to connect to Adobeās Creative Cloud jim@designer.com
- 5. Thousands of EnterpriseCustomers Ed, Gov, Non-Profit Services Media ConsumerTechnology Manufacturing, Energy FinanceCloudHealth
- 6. Okta Application Network MobilityManagementSingle Sign On Adaptive MFA Provisioning Universal Directory Extensible Profiles, Attribute Transformations, Directory Integration and AD Password Management Secure SSO for All Your Web Apps, On-prem and Cloud, with Flexible Policy, from Any Device Contextual Access Policies, Modern Factors, Adaptive Authentication, Integrations for Apps and VPNs Lifecycle Management, Cloud & On-prem App Integration, Mastering from Apps, Directory Provisioning, Rules, Workflow, Reporting Tight User Identity Integration, Device Based Contextual Access, Light-weight Management Okta IT & Platform products
- 7. The most reliableIDaaS available Never taken offline for upgrades Redundant and scalable A B C A B C DC2 DC1 okta.com/trust A Platform Architecture For Scale DATA TIER A B C LOAD BALANCERS APP SERVERS
- 8.
- 9.
- 10.
- 11. Defining a patternfor micro-services https://www.pinterest.com/pin/205828645447534387/ http://www.bennysbaker.com/poop-emoji-cupcakes/
- 12. DevOps abstraction layer Inspiredby: http://dev2ops.org/2010/02/what-is-devops/ Dev OpsWall of turmoil Dev Ops I want stabilityI want change Domain boundary
- 13. Repeatability through immutability ā¢Same runtime environment dev / test / prod ā¢ Runtime versioned w/ code ā¢ Easy reproducibility ā¢ All changes use same release process
- 14. Additional requirements ā¢ 0-downtimedeployments ā¢ Support for our multi-az & multi-region architecture ā¢ Compliance ā SOC2 type 2, HIPAA, ISO 27001 ā¢ Separation of duties ā a.k.a. no developer access to production hosts ā¢ Push button deployment ā¢ Rollback and canary support
- 15.
- 16. Building blocks Dev Ops Iwant stabilityI want changeContainer frameworks Cluster schedulerDev Ops Continuous integration
- 17. Options Container frameworks Clusterschedulers Amazon EC2 Container Service LXC
- 18. Our problems solved ā¢Repeatability ā¢ Declarative & composable Dockerfile ā¢ Images are immutable ā¢ Stability ā¢ Massive community with production adoption ā¢ Initial release > 3 years ago ā¢ Compliant ā¢ ECS isnāt in flow, EC2 is already compliant ā¢ DevOps Abstraction ā¢ Hosts and underlying resources abstracted away ā¢ Task Definition allows developers to schedule deploys ā¢ Stability ā¢ 0-downtime services ā¢ Fully managed! ā¢ Works with existing AWS tooling Docker EC2 Container Service
- 19.
- 20. Source: All ThingsDistributed ā a.k.a Werner Vogels
- 21. Additional concepts ā¢ TaskDefinitions define one or more containers to run. ā¢ Services define a long running task and run inside a cluster ā¢ Clusters define a set of EC2 resources that can be shared by more than one service ā¢ Auto scaling groups can be used to define size and launch configuration of a cluster
- 22. CI with ECSTasks
- 23. CI Workflow Artifactory (Maven, NPM,Docker, YUM) Topic builds ā topic repo Promoted buildsā release repo
- 24.
- 25. Why ECS āIsolation & Versioning
- 26. 1. Lambda: Taskwhich scales cluster based on queue 2. Lambda: Inspect running tasks an bin pack new tasks where possible ā¢ This is one of the changes we had to make in order to use ECS for long running tasks, rather than long running services spread across many stateless instances ā¢ Disconnects unneeded nodes from cluster allowing themselves to self terminate when they are idle Why ECS - Dynamic worker scaling VS
- 27.
- 28. Cost Savings WithSpot Instances
- 29. Feature Requests ā¢ Abilityto have spot and on-demand in same Auto Scaling Group (ASG) ā¢ Built-in bin packing scheduler ā¢ Give ASG a termination policy based on ECS status ā¢ i.e. prefer instances with no running tasks
- 30. Termination policy ā¢ OldestInstance.Auto Scaling terminates the oldest instance in the group. This option is useful when you're upgrading the instances in the Auto Scaling group to a new EC2 instance type, so you can gradually replace instances of the old type with instances of the new type. ā¢ NewestInstance. Auto Scaling terminates the newest instance in the group. This policy is useful when you're testing a new launch configuration but don't want to keep it in production. ā¢ OldestLaunchConfiguration. Auto Scaling terminates instances that have the oldest launch configuration. This policy is useful when you're updating a group and phasing out the instances from a previous configuration. ā¢ ClosestToNextInstanceHour. Auto Scaling terminates instances that are closest to the next billing hour. This policy helps you maximize the use of your instances and manage costs. ā¢ Default. Auto Scaling uses its default termination policy. This policy is useful when you have more than one scaling policy associated with the group.
- 31. Takeaways ā¢ ECS isrunning well for us in a 150+ instance cluster ā¢ Bake AMI with large files and common images into host machines ā¢ Spot instances give 2 min warning. Keeps jobs short
- 32.
- 33.
- 34.
- 36. Test Assumptions ā¢ ECSconfig ā¢ Agent version 1.11.0 ā¢ Docker version 1.11.2 ā¢ Cluster config ā¢ 8 instances backed by ASG ā¢ ASG config ā¢ 8 instances across 3 AZs ā¢ Default termination policy ā¢ 5 min health check grace period ā¢ ELB ā¢ Timeout 4s ā¢ Interval 5s ā¢ Unhealthy threshold 2 ā¢ Healthy threshold 10 ā¢ Enable connection draining 300s timeout ā¢ Load generation ā¢ 16 threads ā¢ Throughput ā¢ Interactive ļØ 490 r/s ā¢ 10s long poll ļØ 1.5 r/s
- 37. Operation Interactive Errors (~70mslatency, 490rps) Long Poll Errors (~10s latency, 1.5rps) Upsize ECS service 4 ļ 8 0 0 Downsize ECS service 8 ļ 4 0 0 Deploy ECS service ā 50% min healthy 0 0 Stop task* 0 0 Downsize Auto Scaling Group (ASG) 0 0 Terminate EC2 instance 0 0 Stop Docker daemon (service docker stop)* 0 0 Stop EC2 instance** 0 0 Kill Docker Container (docker kill <containerId>)* 2 2 Fail health check 450 5 * No intention of running operation in practice ** Caused inconsistent state
- 38.
- 39. Workflow Auto Scaling Group LaunchConfig EC2 ECS Cluster ECS Service ECS Canary Service Application YAML Docker Registry (Artifactory) ELB Images pulled when tasks start Conductor (Bastion ECS Controller) CI Pipeline Git Repo Promoted artifactsDockerfile docker_compose.yml Test / Preview / ProductionDev Deploy new version
- 40. Application definition ā¢ Developersdefine YAML for their application ā¢ Deploy time configuration is supplied to the ECS task definition ā¢ Secrets are pulled by the application at startup
- 41. Security conventions ā¢ Containerrepository ā¢ Only allow containers from internal repository ā¢ IAM separation per service ā¢ Either service per cluster or use new IAM for ECS functionality ā¢ Security scanning of containers - JFrog Xray ā¢ Process monitoring on docker host ā cAdvisor from google ā¢ Secrets or any form of config NEVER baked in containers ā¢ Start from minimal, audited base OS ā¢ Run container as non-privileged user w/ user namespaces Docker 1.10+ ā¢ Monitor alas.aws.amazon.com for critical updates
- 42. Source Conventions ā¢ 3categories of container definitions 1. āLibraryā definitions used as the basis for building other images 2. Third-party service definitions e.g. Zookeeper or Elasticsearch 3. Internal service definitions ā¢ Repo per internal service ā¢ Dockerfile in same repo => image versioned with code ā¢ Docker compose for running dependent services ā¢ Pegged versions (no builds) ā¢ Single repo for library and third-party service definitions
- 43. Build Conventions ā¢ Integrationtests run against code running in container ā¢ Build owns creating immutable version and publishing to artifact server ā¢ Strict rules around āFROMā clause ā¢ Must point at internal artifact server ā¢ Must be tagged following SEMVER-SHORT_SHA convention ā¢ Never allow missing or use of ālatestā tag for repeatable builds
- 44. Logging and monitoring ā¢Logging ā¢ All output streams pipe to STDOUT/STDERR of the running process ā¢ Log forwarding is provided by underlying host ā¢ Log entries contain ā¢ Host ā¢ Container Id ā¢ Image name & version ā¢ Request Id ā¢ Metrics ā¢ Host level, generic container metrics provided by host ā¢ App level metrics published directly to well defined endpoints
- 45. Feature requests ā¢ ELB ā¢Dynamic port mapping to containers ā¢ Fail health based on HTTP return code ā¢ Different health endpoint for adding vs removing ā¢ Service level security groups ā¢ Service discovery w/o ELB ā¢ Ability to mark container instances as un-schedulable ā¢ Remove sharp edges around the stopped state ā¢ Give ASG ability to set EC2 āshutdown behaviorā ā¢ Periodic cleanup process in ECS to deregister stopped instances
- 46. Takeaways ā¢ /etc/ecs/ecs.config ā¢ ECS_ENGINE_TASK_CLEANUP_WAIT_DURATIONfor forensics (default 1hr) ā¢ ECS_LOGLEVEL=debug ā¢ Beware of running services in same cluster that use the same ports ā¢ Tune ELB health check ā¢ Docker 1.10 for security enhancements ā¢ Canary & Blue/Green separate service attached to same ELB ā¢ Rollback is trivial ā¢ ECS is incredibly easy to get up and running ā¢ The ecosystem is changing quickly, we are moving cautiously ā¢ ECS team has made a lot of improvements
- 47. Dev OpsWall ofturmoil Automated pipeline of awesomeness
- 48.
- 49. Thank You Follow me@JonToddDotCom Join us @Okta - www.okta.com/company/careers/
Editor's Notes
- #3Ā How many have heard of Okta? Used it?
- #7Ā This is the full set of Okta IT and Platform products, 100-cloud based and integrated. Each of these are full featured products you could use to replace CA, RSA, or Airwatch.
- #10Ā Java backend JS Front end Entirely hosted in the cloud in AWS In general we like using and giving back to open source
- #14Ā Same environment dev / test / prod Environment should be versioned with code Problems with chef mutating production with bad or incorrect version of config Easy reproducibility Security audit can be done on artifact and then just monitor runtime for correct version
- #19Ā All together we get a PATTERN FOR MICROSERVICES
- #21Ā - We run on ecs optimized - Reduced packages - Upgrade is easier
- #24Ā Developers can run all CI test on any topic branch Master locked down, Bacon is the gate keeper Jenkins used for job definition and lifecycle Slave pool is ECS! ECS run as short lived tasks Each day we get between 100 & 150 containers at peak load
- #25Ā This is bacon
- #26Ā From before: main goal repeatability and immutability Not only is the artifact and itās runtime immutable but the container which builds the artifact for testing is containerized Solves classic problem: changes to environment in CI
- #27Ā Who has the knowledge about sizing?
- #29Ā We presently respond to Spot price termination notices( you get 2 minutes warning) by placing tasks running on a node to be terminated back into the queue to immediately get picked up by another node. Currently working on recognition of spot price instance pool cascades, so we can switch to on demand. No ability to have both spot and on demand in same ASG Something to worry about. If the prices spike, and cause large outages, what is the availability of on demand instances?
- #32Ā We auto scale daily to around 150 instances and back down to under 20 daily. Preload Maven, NPM, & git repositories. This saved us about 4 minutes on container start time
- #40Ā The integration point between CI and Deployment is artifactory. Any sign off or approval happens there Autoscaling groups control pool of EC2 instances Launch config sets environment variables for ECS config like cluster and ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION ECS cluster per service due to IAM issues, looking forward to using new feature 1 or more services registered to a service ELB supports canary Conductor is bastion service. Allows non-operators to perform deployments
- #41Ā Software is grouped into applications which may have multiple components YAML defines all components. In this case we have an application with a single backend running in ECS
- #45Ā 2016-07-26T18:56:08Z [INFO] Redundant container state change for task op1-sage:15 arn:aws:ecs:us-east-1:011750033084:task/8f9920cf-a289-44bb-ac43-e436d6fb84d7, Status: (RUNNING->RUNNING) Containers: [op1-sage-app (RUNNING->RUNNING),]: op1-sage-app(docker.aue1d.saasure.com/okta-sage:1_1_0_029796_ec67fd3) (RUNNING->RUNNING) to RUNNING, but already RUNNING