Open In App

Caching – System Design Concept

Last Updated : 12 Feb, 2024
Summarize
Comments
Improve
Suggest changes
Like Article
Like
Save
Share
Report
News Follow

Caching is a system design concept that involves storing frequently accessed data in a location that is easily and quickly accessible. The purpose of caching is to improve the performance and efficiency of a system by reducing the amount of time it takes to access frequently accessed data.

caching-beginner

1. What is Caching

what-is-caching-in-system-design-(1)

Imagine a library where books are stored on shelves. Retrieving a book from a shelf takes time, so a librarian decides to keep a small table near the entrance. This table is like a cache, where the librarian places the most popular or recently borrowed books.

Now, when someone asks for a frequently requested book, the librarian checks the table first. If the book is there, it’s quickly provided. This saves time compared to going to the shelves each time. The table acts as a cache, making popular books easily accessible.

  • The same things happen in the system. In a system accessing data from primary memory (RAM) is faster than accessing data from secondary memory (disk). 
  • Caching acts as the local store for the data and retrieving the data from this local or temporary storage is easier and faster than retrieving it from the database.
  • Consider it as a short-term memory that has limited space but is faster and contains the most recently accessed items.
  • So If you need to rely on a certain piece of data often then cache the data and retrieve it faster from the memory rather than the disk.

As you know there are many benefits of the cache but that doesn’t mean we will store all the information in your cache memory for faster access, we can’t do this for multiple reasons, such as:

  • Hardware of the cache which is much more expensive than a normal database.
  • Also, the search time will increase if you store tons of data in your cache.
  • So in short a cache needs to have the most relevant information according to the request which is going to come in the future.

2. How Does Cache Work?

Cache-Working

Typically, web application stores data in a database. When a client requests some data, it is fetched from the database and then it is returned to the user. Reading data from the database needs network calls and I/O operation which is a time-consuming process. Cache reduces the network call to the database and speeds up the performance of the system.

Lets understand how cache work with the help of an example:

Twitter: when a tweet becomes viral, a huge number of clients request the same tweet. Twitter is a gigantic website that has millions of users. It is inefficient to read data from the disks for this large volume of user requests.

Here is how using cache helps to resolve this problem:

  • To reduce the number of calls to the database, we can use cache and the tweets can be provided much faster.
  • In a typical web application, we can add an application server cache, and an in-memory store like Redis alongside our application server.
  • When the first time a request is made a call will have to be made to the database to process the query. This is known as a cache miss.
  • Before giving back the result to the user, the result will be saved in the cache.
  • When the second time a user makes the same request, the application will check your cache first to see if the result for that request is cached or not.
  • If it is then the result will be returned from the in-memory store. This is known as a cache hit.
  • The response time for the second time request will be a lot less than the first time. 

3. Where Cache Can be Added?

Caching is used in almost every layer of computing.

  • In hardware, for example, you have various layers of cache memory.
  • You have layer 1 cache memory which is the CPU cache memory, then you have layer 2 cache memory and finally, you would have the regular RAM (random access memory).
  • You also have to cache in the operating systems such as caching various kernel extensions or application files.
  • You also have caching in a web browser to decrease the load time of the website.

4. Key points to understand Caching

Caching can be used in a variety of different systems, including web applications, databases, and operating systems. In each case, caching works by storing data that is frequently accessed in a location that is closer to the user or application. This can include storing data in memory or on a local hard drive.

  • How it works:
    • When data is requested, the system first checks if the data is stored in the cache.
    • If it is, the system retrieves the data from the cache rather than from the original source.
    • This can significantly reduce the time it takes to access the data.
  • Types of caching:
    • There are several types of caching, including in-memory caching, disk caching, and distributed caching.
    • In-memory caching stores data in memory, while disk caching stores data on a local hard drive.
    • Distributed caching involves storing data across multiple systems to improve availability and performance.
  • Cache eviction:
    • Caches can become full over time, which can cause performance issues.
    • To prevent this, caches are typically designed to automatically evict older or less frequently accessed data to make room for new data.
  • Cache consistency:
    • Caching can introduce issues with data consistency, particularly in systems where multiple users or applications are accessing the same data.
    • To prevent this, systems may use cache invalidation techniques or implement a cache consistency protocol to ensure that data remains consistent across all users and applications.

5. Types of Cache

In common there are four types of Cache…

5.1. Application Server Cache:

In the “How does Cache work?” section we discussed how application server cache can be added to a web application.

  • A cache can be added in in-memory alongside the application server.
  • The user’s request will be stored in this cache and whenever the same request comes again, it will be returned from the cache.
  • For a new request, data will be fetched from the disk and then it will be returned.
  • Once the new request will be returned from the disk, it will be stored in the same cache for the next time request from the user.

Note: When you place your cache in memory ,the amount of memory in the server is going to be used up by the cache. If the number of results you are working with is really small then you can keep the cache in memory. 

Application-Server-Cache-(1)

Drawbacks of Application Server Cache:

  • The problem arises when you need to scale your system. You add multiple servers in your web application (because one node can not handle a large volume of requests) and you have a load balancer that sends requests to any node.
  • In this scenario, you’ll end up with a lot of cache misses because each node will be unaware of the already cached request.
  • This is not great and to overcome this problem we have two choices: Distribute Cache and Global Cache. Let’s discuss that…

5.2. Distributed Cache:

In the distributed cache, each node will have a part of the whole cache space, and then using the consistent hashing function each request can be routed to where the cache request could be found. Let’s suppose we have 10 nodes in a distributed system, and we are using a load balancer to route the request then…

  • Each of its nodes will have a small part of the cached data.
  • To identify which node has which request the cache is divided up using a consistent hashing function each request can be routed to where the cached request could be found. If a requesting node is looking for a certain piece of data, it can quickly know where to look within the distributed cache to check if the data is available.
  • We can easily increase the cache memory by simply adding the new node to the request pool.

Distributed-Cache

5.3. Global Cache:

As the name suggests, you will have a single cache space and all the nodes use this single space. Every request will go to this single cache space. There are two kinds of the global cache

  • First, when a cache request is not found in the global cache, it’s the responsibility of the cache to find out the missing piece of data from anywhere underlying the store (database, disk, etc).
  • Second, if the request comes and the cache doesn’t find the data then the requesting node will directly communicate with the DB or the server to fetch the requested data.

Global-Cache

5.4. CDN (Content Distribution Network)

A CDN is essentially a group of servers that are strategically placed across the globe with the purpose of accelerating the delivery of web content. A CDN-

  • Manages servers that are geographically distributed over different locations.
  • Stores the web content in its servers.
  • Attempts to direct each user to a server that is part of the CDN so as to deliver content quickly.

CDN is used where a large amount of static content is served by the website. This can be an HTML file, CSS file, JavaScript file, pictures, videos, etc. First, request ask the CDN for data, if it exists then the data will be returned. If not, the CDN will query the backend servers and then cache it locally.

CDN-new

6. Applications of Caching

Facebook, Instagram, Amazon, Flipkart….these applications are the favorite applications for a lot of people and most probably these are the most frequently visited websites on your list.

Have you ever noticed that these websites take less time to load than brand-new websites? And have you noticed ever that on a slow internet connection when you browse a website, texts are loaded before any high-quality image? Why does this happen?

The answer is Caching.

  • If you check your Instagram page on a slow internet connection you will notice that the images keep loading but the text is displayed. For any kind of business, these things matter a lot.
  • A better customer/user experience is the most important thing and you may lose a lot of customers due to the poor user experience with your website.
  • A user immediately switches to another website if they find that the current website is taking more time to load or display the results.

You can take the example of watching your favorite series on any video streaming application. How would you feel if the video keeps buffering all the time? Chances are higher that you won’t stick to that service and you discontinue the subscription. All the above problems can be solved by improving retention and engagement on your website and by delivering the best user experience. And one of the best solutions is Caching.

7. What are the Advantages of using Caching?

Caching optimizes resource usage, reduces server loads, and enhances overall scalability, making it a valuable tool in software development.

  • Improved performance: Caching can significantly reduce the time it takes to access frequently accessed data, which can improve system performance and responsiveness.
  • Reduced load on the original source: By reducing the amount of data that needs to be accessed from the original source, caching can reduce the load on the source and improve its scalability and reliability.
  • Cost savings: Caching can reduce the need for expensive hardware or infrastructure upgrades by improving the efficiency of existing resources.

8. What are the Disadvantages of using Caching?

Despite its advantages, caching comes with drawbacks also and some of them are:

  • Data inconsistency: If cache consistency is not maintained properly, caching can introduce issues with data consistency.
  • Cache eviction issues: If cache eviction policies are not designed properly, caching can result in performance issues or data loss.
  • Additional complexity: Caching can add additional complexity to a system, which can make it more difficult to design, implement, and maintain.

Overall, caching is a powerful system design concept that can significantly improve the performance and efficiency of a system. By understanding the key principles of caching and the potential advantages and disadvantages, developers can make informed decisions about when and how to use caching in their systems.

9. Cache Invalidation Strategies

Cache invalidation is crucial in systems that use caching to enhance performance. When data is cached, it’s stored temporarily for quicker access. However, if the original data changes, the cached version becomes outdated. Cache invalidation mechanisms ensure that outdated entries are refreshed or removed, guaranteeing that users receive up-to-date information.

  • Common strategies include time-based expiration, where cached data is discarded after a certain time, and event-driven invalidation, triggered by changes to the underlying data.
  • Proper cache invalidation optimizes performance and avoids serving users with obsolete or inaccurate content from the cache.

10. Eviction Policies of Caching

Eviction policies are crucial in caching systems to manage limited cache space efficiently. When the cache is full and a new item needs to be stored, an eviction policy determines which existing item to remove.

  • One common approach is the Least Recently Used (LRU) policy, which discards the least recently accessed item. This assumes that recently used items are more likely to be used again soon.
  • Another method is the Least Frequently Used (LFU) policy, removing the least frequently accessed items.
  • Alternatively, there’s the First-In-First-Out (FIFO) policy, evicting the oldest cached item.

Each policy has its trade-offs in terms of computational complexity and adaptability to access patterns. Choosing the right eviction policy depends on the specific requirements and usage patterns of the application, balancing the need for efficient cache utilization with the goal of minimizing cache misses and improving overall performance.

11. Conclusion

  • Caching is becoming more common nowadays because it helps make things faster and saves resources.
  • The internet is witnessing an exponential growth in content, including web pages, images, videos, and more.
  • Caching helps reduce the load on servers by storing frequently accessed content closer to the users, leading to faster load times.
  • Real-time applications, such as online gaming, video streaming, and collaborative tools, demand low-latency interactions.
  • Caching helps in delivering content quickly by storing and serving frequently accessed data without the need to fetch it from the original source every time.



Previous Article
Next Article

Similar Reads

Differences between Edge Caching and Centralized Caching in System Design
In system design, caching plays a critical role in enhancing performance and reducing latency. Two common caching strategies are edge caching and centralized caching, each suited for different architectural needs. Edge caching involves placing data closer to the user at the network edge, improving access speed, and reducing the load on central serv
4 min read
Server-side Caching and Client-side Caching
Caching is a temporary technique that stores frequently accessed data for faster retrieval. There are two main types of caching in web development: server-side caching and client-side caching. Server-side caching stores data on the server to reduce load times and server strain. Client-side caching stores data on the user's device, improving speed a
8 min read
Negative Caching - System Design
Negative caching refers to storing failed results or errors to avoid redundant requests. It plays a major role in enhancing system performance by preventing repeated processing of known failures. By caching these negative responses, systems save resources and improve response times. Unlike positive caching, which stores successful results, negative
11 min read
Edge Caching - System Design
Edge caching in system design refers to storing data or content closer to the user at the network's edge like in local servers or devices rather than in a central location. This approach reduces the distance data needs to travel, speeding up access and reducing latency. Edge caching is commonly used in content delivery networks CDNs to improve perf
13 min read
Caching Design Pattern
In today's digital world, speed and efficiency matter a lot. When we use apps and websites, we want things to happen quickly. But making applications run fast is a bit tricky. That's where the caching design pattern comes in. Imagine caching as a fast storage trick. It stores important data from apps in a special place so that the app doesn't need
10 min read
Asynchronous Caching Mechanisms to Overcome Cache Stampede Problem
Caching is a critical component in modern software systems, serving as an effective means to reduce latency and improve system performance. Asynchronous caching is a caching mechanism that allows data to be stored and retrieved in a non-blocking, asynchronous manner. This means that when data is requested, the system doesn't wait for the cache to p
4 min read
What is Pre-Caching?
Pre-caching is like getting ready for something before it happens. Imagine you're going on a trip and you pack your bag the night before so you're all set to go in the morning. That's pre-caching! In the digital world, it's when your device stores information ahead of time, like loading a webpage before you even click on it. This helps things run s
13 min read
Caching Strategies for API
The article explains how to improve the performance and efficiency of APIs using caching. Caching is a technique where frequently accessed data is stored temporarily to reduce the time and resources needed for future requests. The article discusses different methods and strategies for implementing caching in APIs, highlighting the benefits and best
10 min read
Why Caching Does not Always Improve Performance?
Caching is widely regarded as a key technique for enhancing the performance of systems by reducing data retrieval times and alleviating the load on backend resources. However, despite its potential benefits, caching does not always lead to performance improvements and can sometimes even degrade system performance. This article delves into the compl
8 min read
Design a system that counts the number of clicks on YouTube videos | System Design
Designing a Click Tracking System for YouTube Videos involves architecting a comprehensive and efficient solution to monitor and analyze user interactions with videos on the YouTube platform. Important Topics for Counting the number of Clicks on Youtube Videos System DesignRequirementsCapacity EstimationLow-Level Design (LLD)Database Design-High-Le
15+ min read
Design Restaurant Management System | System Design
In the modern restaurant industry, delivering exceptional dining experiences requires more than just good cuisine. Restaurant Management Systems have emerged as the secret ingredient for efficient operations and customer satisfaction. In this article, we are going to design a restaurant management system where users can effortlessly discover nearby
15 min read
Design a Picture-Sharing System - System Design
In the present day, the need for good tools to exchange and organize images has never been this much higher. As for social networking, e-shopping, personal branding, and many other related purposes, users anticipate that the sites they visit provide them with fast and dependable environments with or without loads of images that enable them to inter
11 min read
Difference between System Design and System Architecture
When it comes to software development and engineering there can be some confusion, between the two terms "System Design" and "System Architecture." Although they are related they actually refer to stages and responsibilities in creating scalable and efficient systems. It is important to grasp the differences, between System Design and System Archit
3 min read
Distributed Messaging System | System Design
In our fast-paced world, how we share information matters more than ever. Old-school messaging setups sometimes struggle to keep up with today's tech demands. That's where distributed messaging systems step in. They're like a breath of fresh air, changing the game and making sure our messages get where they need to go, no matter what. Important Top
8 min read
System Analysis | System Design
In the areas of science, information technology, and knowledge, the difficulty of systems is of much importance. As systems became more complicated, the traditional method of problem-solving became inefficient. System analysis is to examine a business problem, identify its objectives and requirements, and then design the most optimal solution to fu
6 min read
Designing Authentication System | System Design
Keeping your important digital information safe is like building a strong foundation, and the key to that is a good security system. This article will help you understand how to make a robust security system step by step. We'll start with figuring out what you want to protect and what you want to achieve. Then, we'll talk about the detailed design
10 min read
Designing Parking Lot (Garage) System | System Design
Parking garage plays an important role in today's world, as the number of vehicles is increasing every day, so it has become a major concern to manage them. The parking garage provides the solution to this problem of limited space. It offers to book a Parking garage using any method whether online or offline. In this article, we will explore all th
11 min read
System Design | Online Banking System
In the digital era, the layout of a Online Banking System is a critical element of any commercial enterprise, it not only affects user experience but also the growth of a service-providing company. This article explores the comprehensive guide to designing a Online Banking System for its efficient use. From practical and non-useful requirements to
12 min read
Resilient System - System Design
Imagine you're building a castle out of blocks. If you design it so that removing one block doesn't make the whole castle collapse, you've made something resilient. hen we talk about creating a resilient system, we're essentially doing the same thing but with computer systems. These systems are designed to handle problems like errors, crashes, or e
10 min read
Blob vs. File System in System Design
In system design, choosing the right method for storing and managing data is crucial for performance, scalability, and maintenance. Two common approaches are using Blobs (Binary Large Objects) and traditional file systems. Each has its own set of advantages and challenges, making them suitable for different use cases. This article explores the key
7 min read
Differences between System Analysis and System Design
System Analysis and System Design are two stages of the software development life cycle. System Analysis is a process of collecting and analyzing the requirements of the system whereas System Design is a process of creating a design for the system to meet the requirements. Both are important stages as it helps to create an effective system with all
4 min read
System Development Life Cycle vs. System Design Life Cycle
The concepts of System Development Life Cycle (SDLC) and System Design Life Cycle (SDLC) are fundamental in the field of software engineering and system architecture. Both frameworks play crucial roles in the creation and implementation of complex systems, but they address different aspects of system creation and management. Important Topics for Sy
6 min read
System Design of Uber App | Uber System Architecture
Getting the hassle-free transportation service(Uber, Ola) is very easy but is it also simple to build these gigantic applications that have hundreds of software engineers working on them for a decade? surely not. These systems have much more complex architecture and there are a lot of components joined together internally to provide riding services
13 min read
System Design - Design Google Calendar
Google Calendar is like a special calendar that we can use on our computer or phone. We can use it to remember important things, like birthdays, appointments, and events. We can also set reminders to help us remember when something is coming up soon. We can also share our calendars with our family or friends. It's like having our own special schedu
8 min read
How to Design a Rate Limiter API | Learn System Design
A Rate Limiter API is a tool that developers can use to define rules that specify how many requests can be made in a given time period and what actions should be taken when these limits are exceeded. Rate limiting is an essential technique used in software systems to control the rate of incoming requests. It helps to prevent the overloading of serv
11 min read
System Design - Design a Sequencer
Pre-requisites: Logic gates Sequencer is a device that generates a sequence of unique identifiers or numbers. In the context of distributed systems, a sequencer is often used to generate unique IDs and objects across multiple nodes or servers. In this article, we will discuss how to design a sequencer for use in a distributed system. The sequencer
5 min read
Design Principles in System Design
Design Principles in System Design are a set of considerations that form the basis of any good System. But the question now arises why use Design Principles in System Design? Design Principles help teams with decision-making, and is a multi-disciplinary field that involves trade-off analysis, balancing conflicting needs, and making decisions about
5 min read
Design Notification Services | System Design
If we are building an e-commerce application or a booking system or anything of that sort, we will always have a notification service that will be used to notify your consumers. Let us now look at the requirements to build a notification service. Topics for Designing Notification ServicesRequirementsComponentsHow does our system tackle if we want t
8 min read
Design Facebook's live update of comments on posts | System Design
A live comment system, like Facebook's, works by employing real-time communication technologies, such as WebSockets, to establish a persistent connection between users and the server. When a user posts a comment, it's immediately sent to the server, which then broadcasts the comment to all users viewing the same post in real-time. This allows users
15+ min read
Design a webpage that can show the status of 10M+ users including: name, photo, badge and points | System Design
We've got this huge community—over 10 million strong—and you want to build a webpage where everyone's details, like their names, photos, those cool badges they've earned, and their points, can all be seen. That's a massive load of information to handle. Achieving this goal necessitates an efficient and scalable system architecture capable of handli
15+ min read
three90RightbarBannerImg