Download as PDF, PPTX
Uploaded byDori Waldman
Memcached
ElasticCache is a caching service that uses Memcached. Memcached is an in-memory key-value store that provides no persistence or replication. It is fast and preferable for caching relatively small static data. At a certain point, implementation knowledge is needed to ensure Memcached is behaving as expected. Production issues can occur if objects do not fit properly into Memcached slabs, which allocate fixed-size chunks of memory. Monitoring tools like "stats slabs" help analyze slab allocation and object eviction patterns.
More Related Content
Memcached
- 1. ElasticCache Use Case Dori Waldman- Big Data Lead
- 2. AT A GLANCE 1.2M TRANSACTIONS/S 8T RAW DATA/d NEW YORK LONDON SAN FRANCISCO TEL AVIV BEIJING 500-1000 SERVERS 2007 FOUNDED 100EMPLOYEES 800M UNIQUE USERS InnerActive in numbers
- 3. OUR MISSION Ad-Request RTB Non RtB Im pressions / Clicks 1 3 2 Highlevel architecture: fully scale & elastic
- 4. Main DB Stack Aggregations Configuration Analytic -Cube Analytic KV - Cache SSOT Q
- 5.
- 6. Memcached Simple in-memory Key-Valuestore ~ HashMap No persistence (redis is also in memory but use disk for persistence) No replication No complex objects (List/Map...) , just strings,Int,byte LRU eviction policy Fast preferable when caching relatively small and static data (html code) Scale (default 20 nodes in AWS) Simple
- 7.
- 8. Problem: 3 month agowe faced production issue , lots of eviction although there is enough memory in the cluster, we tried to increase cluster size , change machines , add memory , play with slab size … (We are using 16 nodes (r3.xlarge), verison 1.4.33) Solution: ... Production Issue The Memcached API is very simple, straightforward. But at a certain point, you need implementation knowledge to make sure things are behaving as expected.The Memcached API is very simple, straightforward. But at a certain point, you need implementation knowledge to make sure things are behaving as expected.The Memcached API is very simple, straightforward. But at a certain point, you need implementation knowledge to make sure things are behaving as expected. Memcached API is very simple, But at a certain point, you need implementation knowledge to make sure things are behaving as expected.
- 9. How it works Grow factorof 1.25 : round(104*1.25)=136 ... P1 - Waste of resources: If you have objects that are all 720,000 bytes long, they have to fit in class 42 -- which means a guaranteed internal wastage of ~30% (remain 1048576-72000) There is no way to use this memory! The chunks in each slab have the same size so there cannot be 2 different sized chunks inside the same page.
- 10. How it works P2 –different input sizes/Eviction Slab 1 might have 800 pages and slab 3 might have 200 pages. Everything is good, all memory is used , as long as we keep getting another small item “slab class 1” the LRU eviction of that slab class will handle it, we might evict item from slab1 although slab3 has older items – but its not that critical like Once we get medium item, slab 2 has no pages to use, more accurate it has default of one page, Meaning slab2 eviction will not behave as expected (pages from slab class 1/3 cant re-assign to class 2) Memcache will initially create 1 page per slab-class and the rest of the pages will be free. There can be multiple pages for each slab-class, but as soon as a page is assigned a slab-class (and thus, split up in chunks), it cannot be changed to another slab-class.
- 11. Production Issue What we saw: lets say we have 10G available memory Our input data should fit to one slab class only (4k size) Total memory in use is 4M , still we see eviction
- 12. Monitoring use the “statsslabs” command to see where memcached has allocated its memory. If you watch the various get/set statistics on each slab, you can get an idea on which slabs are hot
- 13. Auto Discovery -application does not need to manually connect to individual cache nodes, instead, your application connects to one Memcached node and retrieves the list of nodes. From that list your application is aware of the rest of the nodes in the cluster and can connect to any of them. You do not need to hard code the individual cache node endpoints in your application. → we are not using it Consist hashing : Regular hash: $server_id = hashfunc($key) % $servercount; Without CH in case we add/remove node , the keys are not relevant , we need to restart the cluster, with this feature we can look for the key in the neighbor node → we are not using it Features
- 14.
- 15.