AWS Cost Optimization
Techniques and Best Practices
Sana Mahajan
Nov 2023
Agenda
• Cost Optimization Importance – AWS
• Cost Optimization Framework
• Best Practices & Techniques – AWS Cost Saving
Right Sizing
On-demand, Spot and Reserved Instance Usage
Tagging and Resource Grouping
Optimize Data storage cost
Use AWS Trusted Advisor
Use Auto-Scaling
• Cost Optimization Success Stories
• Q&A
Cost Optimization Importance - AWS
Cost Efficiency: AWS offers a wide range of services with varying pricing models, and it's easy for costs to escalate if resources are not
managed properly. Cost optimization helps to get the most value from AWS spending by reducing unnecessary expenses.
Scalability: AWS allows us to scale resources up or down as needed. Optimizing costs ensures to avoid over-provisioning of resources.
Resource Allocation: Helps identify underutilized or unused resources, which helps to use unutilized resources in a better manner.
Data-Driven Decision-Making: With this we can make data-driven decisions about our AWS resources and identify services or resources
which are driving costs, and adjust the usage accordingly.
Cost Visibility: AWS provides various tools and services to help track and analyze cloud spending in an account and have clear visibility
into spending, which is crucial for informed decision-making.
Budget Control: Enables to set and maintain budgets, making it easier to manage and control spending. This is essential for financial
planning and preventing unexpected overruns.
Compliance: Helps to ensure compliance with budget and financial goals and is much important for organizations with regulatory or
financial constraints.
Sustainability: By reducing the amount of resources used, we can contribute to a more sustainable and environmentally friendly cloud
operation and have a positive impact on the carbon footprint.
Cost Optimization Framework
• Set of guidelines and practices for managing and reducing the costs associated with cloud computing
• Systematic way to analyze, control, and optimize cloud spending to ensure performance, security, and business objectives are
met
Reserved
Cost Resource Cost Educating
Rightsizing Instances
Assessment Analysis Allocation Teams
Optimization
Spot Cost
Tagging and
Instances Lifecycle Monitoring Governance Reporting
Cost
and Savings Management and Alerting and Policies and
Allocation
Plans Visualization
Best Practices &
Techniques –
AWS Cost Saving
Right Sizing
• Choosing the most appropriate and cost-effective instance type or resource configurations for
the workloads.
• Match resource capacity to actual workload requirements, neither over-provision
nor under-provision.
• Helps in cost reduction, improved performance, and resource efficiency.
• Example- if smaller instance type is chosen for workload requiring more memory,
the job will fail which will cause wastage of time and cost for unsuccessful runs. On contrary,
if larger instance is chosen than required workload, it will lead to more cost and underutilization of resources.
Usage of on-demand, spot and reserved instance
On-demand Instances
• For priority • For long-term • Bid for unused
Reserved Instances
workloads that commitments capacity with
Spot Instances
can’t be based on capacity heavy discounts
interrupted requirement • Good for flexible,
• Pay for compute • Significant fault tolerant or
capacity on per discount for transient
hour basis with no commitment workloads
long term utilization and
commitments capacity
reservation
Tagging and Resource Grouping
• Add metadata to AWS resources in the form of key-value pairs.
• Tags provide information about resources and helps to manage, search for, and
categorize them for better tracking.
• Example- Each project can tag their resources with unique key value pair to get insights
on overall cost utilized by the specific project.
Optimize Data Storage Costs
10 TB S3 Standard Storage Class 10 TB S3 Deep Archive Storage
• Choosing the right storage class to save cost on s3. Cost (Monthly USD) Class Cost (Monthly USD)
• Different storage classes available and one must chose the class as per 235.52 10.27
the use case to minimize the cost.
• For instance- Auditing data accessed once in 3 years should be stored ~96% cost savings by using Deep Archive storage
in S3 Deep Archive to save ~96% cost as compared to storage on S3
Standard.
• Implement lifecycle policies to automatically transition objects to a
less expensive storage class after a specific period.
• Review and delete objects that are no longer needed.
• For instance- A lifecycle can be set to transition all objects in Standard
storage to Glacier storage after 90 days which reduces manual
intervention.
• Setup alarms on CloudWatch to receive notifications when storage
usage exceeds certain thresholds. This helps to detect unexpected
spikes in storage costs.
• Configure S3 Inventory to receive scheduled reports on the insights of
data stored in s3 buckets.
• Helps to understand storage usage and make decisions accordingly on
the stored data by applying lifecycle policies effectively.
• Elastic Block Storage (EBS) Snapshots periodical review is imperative
to check for unwanted snapshots which can be deleted.
• Usage of incremental snapshots should be considered as it is much
cheaper than historical snapshots.
• Data transfer cost is another important factor which can lead to spike
in overall AWS Billing.
• Whenever the data is moved between AWS services in different
regions and from on-prem to cloud or vice-versa is always charged on
the amount of data transferred- out.
• Cost Explorer tool usage to review the detailed cost associated with
each service and identify potential resources that be cost optimized.
• The report can also give detailed insights into instance reservations
and savings plan.
Use AWS Trusted Advisor
• Trusted Advisor provides automated checks and recommendations to help identify and act on opportunities to reduce the AWS costs.
• Checks on areas like EC2 instances, RDS instances, EBS volumes, S3 buckets and more.
• Recommends which EC2 instance type should be used based on the past data and also provides reserved/ savings plan recommendation to
further bring down the cost.
• Recommends for ElasticCache, Redshift, Elasticsearch, RDS reserved instance optimization.
• Has the capability to flag an EC2 instance if it is under utilized and notify the end users accordingly.
Use Auto Scaling
• Auto Scaling policies help to automatically add or remove instances
based on demand.
• Ensures that we have enough capacity to handle traffic during peak
periods and scales down during periods of lower demand.
• Scheduled scaling helps to adjust capacity based on predictable traffic
patterns and thus improves cost further.
• Can create auto scaling group based on minimum capacity, desired
capacity and maximum size to maintain a balance between
performance of the application with the cost implications.
Cost
Optimization
Success Stories
• I got an opportunity to analyze, report and optimize the cost of different AWS services used across the organization.
• Was able to bring down the monthly cost by ~USD 25,000 just across one vertical using detailed monitoring and analysis.
• This was a major milestone of the project and helped the team to understand the importance of cost optimization in cloud and
how it can benefit the overall revenue of the organization.
Below techniques were used for this optimization-
• Creation of s3 buckets with standardized naming convention like orgname-envr-teamname e.g exlproddataengg,
exldevdataanalytics.
• This helped to categorize the buckets as per the team and environment.
• Relevant teams were reached out for information on data stored in their buckets and the time period requirement of the data.
This also helped to segregate important prod data from dev data.
Folder name Data size
• Setup of bucket inventories to get 3-4 level deep analysis of data present exlproddataengg/code/lib/data/ 18 TB
in any of the buckets.
• It helped to get folder wise data storage in a bucket and high storage exlproddataengg/models/algo/ 50 TB
folder were targeted for data cleanup as described in the table-
• Usage of lifecycle policies to move the data for which no owner was identified to lower storage classes before permanent
deletion.
• Example- Data from unidentified owner folder with standard storage class was first moved to glacier class before permanent
deletion.
• It is always important to weigh in the pricing associated with moving of data between different storage classes.
• Reviewed and deleted the older versions to save up on storage and cost on versioned buckets.
• Guidelines were set to have tags attached to each and every resource launched in AWS.
• The tags included- ProjectName, ReqNo_NoOfHours, Environment. For instance, an EC2 launched by Prod team for GenAI work
for 24 hours had following tags- ProjectName- GenAI, ReqNo_NoOfHours- REQ000000_24, Environment- Prod.
• Reports were setup to fetch running EC2 instance each hour and compare the running time with requested number of hours in
the resource tags to audit any over utilized EC2 instances.
• Analysis was done for EBS volumes and snapshots and any unused resources were deleted with owner’s permissions.
• A dashboard was created specifically for launching EMR’s and Ec2’s.
• This tool helped in estimating the cost of running these resources upfront based on number of hours and instances requested
and their types used.
• There was also a provision to allocate daily budget to each project/team. The tool had the feature to trigger an alert to the
owner directly if any instance was over used.
• During cost monitoring, major spike for cost was observed for data transfer out.
• On investigation using AWS Cost Explorer and AWS CloudTrail for data downloads API calls, found that end users were
downloading same set of files 20 times.
• For instance- 200 files were present in a folder, so user X was making 4000 downloads as compared to 200 times.
• In multiple cases, we implemented the feature of Requester pays wherein the end users were charged for the files downloaded.
This discovery helped in saving future costs for such kind of download requests.
• Developer’s were launching resources across regions without knowing the implications on the overall cost it has on cross region
data transfers. The teams were notified about such scenarios to prevent unwanted cost.
• Redshift cluster usage was analyzed and multiple instances were put for reservation due to long run commitment of the cluster.
Questions
Thank You!!
