Tangential, but what I appreciate about Elixir/Erlang is that you have ETS out of the box that is basically a key-value store on a lightweight Erlang process (without socket access or network calls) with flexible 'types', and then DETS brings persistence, mnesia adds schemas, queries, transactions, replication and other DBMS-like features... all this and you can use Erlang's actor model to cluster and horizontally scale, or just use it like a simple L1 cache, as described in the article
Most people use "order of magnitude" to mean "by a factor 10". I've always thought this was inappropriate. The origin of this expression is that the scientific use of magnitude implies a logarithmic scale. But "magnitude" was first used in astronomy with a scale that had a 2.5 base, not a 10 base.
Caching follows Pareto principle - users will get 80% of the benefit just by using either tool effectively, doesn't matter which.
That makes me more interested in comparing the developer experience.
From a practical perspective, Redis is probably a better cache choice for a typical web developer who just wants to get things done and doesn't have major scaling/performance concerns.
It has two main advantages:
(1) Much better tooling. Memcached doesn't have a lot of support to figure out what's going on, measure hit ratio, run functions on cache data, etc. Redis has better built-in diagnostics and third-party tools and libraries.
(2) It's ideal for smaller teams to reduce infrastructure dependencies. Since Redis has many capabilities beyond in-memory caching, there's a good chance you'll be able to take advantage of it for more than just caching in the future (while it's still not really any more complicated to install/maintain than memcached).
1) memcached has excellent support for checking cache hit ratio. Just telnet to it and execute "stats". Most client libraries have support for the stats command, and there is a Prometheus exporter for memcached that works really well.
2) A problem I've seen with Redis is that it is _stateful_. While, "it's still not really any more complicated to install/maintain than memcached" is true for running it locally, I keep seeing over and over again how engineers
i) don't think about backup/restore at all.
ii) expect it to behave like a disk-backed database and don't consider it's limited to the amount of memory you have.
iii) don't consider that it (usually) is a secondary data store that can be out of sync with primary datastore (distributed commits are hard).
iv) don't consider that a proper Redis setup likely needs support for point-in-time recovery and the complexities around that and redundancy/failover.
While Redis is easy to setup as a cache and likely doesn't need the above, I keep seeing them organically grow into being used for other use cases where the above things are not in place, yielding very high operational risks sometimes not acknowledged anywhere.
Thanks. On (2) I'm thinking only of Redis as a memory cache, in that case I think it's fairly simple, as with Memcached.
Once you start doing persistence, then yes, it becomes a bit more complicated with a lot of other things to worry about, especially that you need RAM >= disk space. While it's another advantage of Redis that persistent caching is possible (and can be bolted on at a later date), taking advantage of it definitely adds some complexity and room for error.
2) III - Distributed commits aren't just hard they don't exist. Redis and your primary dB can never hold the same data - at most you can guarantee only for IMMUTABLE data that what is in DB is also in redis.
I've seen others and myself included try to sync them when solving some problems. Just because it is so natural to believe it possible to do. Then you find out you are running circles.
Honestly for the use cases I've met Redis has only made sense to be used as a cache for performance or message bus. Your primary data store is gonna give you the other guarantees of backups, consistency etc...
Unless somebody usew it as their primary store, but everybody chooses their own religion I guess.
There is still the misconception that if you just want to cache, you don't need data types. One of the biggest Redis use cases, Twitter (I didn't check if they still use it in the last two years, I guess so, and anyway used it for many years), beg to differ. But in general almost all the serious caching scenarios using Redis represent part of the stored data, in memory, with an organization that allows updating the cache and selectively retrieving from the cache like if it was the primary (oh, not just the primary, specified queries you could do against the primary), so everything is incremental and there is no need to cache multiple "views" of the same thing.
I switched from Memcached to Redis 10 years ago because I had a situation where I needed to cache lookups of (x, y) but needed to invalidate (x, *), something that is trivial with Redis hashes, but extremely difficult in memcached.
Redis is probably my favorite software of the past decade. I abuse it liberally and it always takes it with resounding uptime and performance. In the stack that I utilize it's always among the things I have to worry the least about.
Agreed, Redis is truly wonderful, I've used it extensively as both a cache lookalike and also as primary datasource (as long as your data size/growth is known and manageable on RAM, which is pretty common for many projects and current hosts) and it's always super fast, reliable and easy to work with.
Not tested recently, but 3 years ago it was said that Redis was slightly faster than memcached for PHP sessions -for example- and I benchmarked it to decide to change. Add persistence to that formula and Redis was a winner for me (put your app into maintenance mode, reboot server(s), enable app again and your users can just resume operations...)
Looks like this article got one bit of updated information but missed everything else... I'll address some things point by point:
data structures: - yes, fewer datastructures, if any. The point isn't that they have the features or not, but that memcached is a distributed system _first_, so any feature has to make sense in that context.
"Redis is better supported, updated more often. (or maybe memcached is "finished" or has a narrower scope?)" - I've been cutting monthly releases for like 5 years now (mind the pandemic gap). Sigh.
Memory organization: This is mostly accurate but missing some major points. The sizes of the slab classes doesn't change, but slabs pages can and do get re-assigned automatically. If you assign all memory to the 1MB page class, then empty that class, memory will go back to a global pool to get re-assigned. There are edge cases but it isn't static and hasn't been for ten years.
Item size limit: The max slab size has actually been 512k internally for a long time now, despite the item limit being 1mb. Why? Because "large" items are stitched together from smaller slab chunks. Setting a 2mb or 10mb limit is fine in most use cases, but again there are edge cases, especially for very small memory limits. Usually large items aren't combined with small memory limits.
You can also _reduce the slab class overhead_ (which doesn't typically exceed 5-10%), by lowering the "slab_chunk_max" option, which puts the slab classes closer together at the expense of stitching items larger than this class. IE; if all of your objects are 16kb or less, you can freely set this limit to 16kb and reduce your slab class overhead. I'd love to make this automatic or at least reduce the defaults.
LRU: looks like the author did notice the blog post (https://memcached.org/blog/modern-lru/) - I'll add that the LRU bumping (mutex contention) is completely removed from the _access path_. This is why it scales to 48 threads. The LRU crawler is not necessary to expire items, there is also a specific thread that does the LRU balancing.
The LRU crawler is used to proactively expire items. It is highly efficient since it independently scans slab classes; the more memory an object uses the fewer neighbors it has, and it schedules when to run on each slab class, so it can "Focus" on areas with higest return.
Most of the thread scalability is pretty old; not just since 2020.
Also worth noting memcached has an efficient flash backed storage system: https://memcached.org/blog/nvm-caching/ - requires RAM to keep track of keys, but can put value data on disk. With this tradeoff we can use flash devices without burning them out, as non-get/non-set operations do not touch the SSD (ie; delete removes from memory, but doesn't cause a write). Many very huge installations of this exist.
I've also been working on an internal proxy which is nearing production-readiness for an early featureset: https://github.com/memcached/memcached/issues/827 - scriptable in lua, will have lots of useful features.
For people who don't know and didn't realize this from the comment: dormando is the principal maintainer of memcached and has been for years (e.g., bradfitz was much less involved after he joined Google).
I’m sorry for asking a rookie question, but you seem to know memcached really well and I couldn’t find an answer online.
Is there a way to obtain/monitor the time stamp of LTU evictions?
I want to get a sense of how memory constrained my memcached server is, and it seems intuitive to me to monitor the “last used” date of recent evictions. Like, if The server is evicting values that haven’t been accessed in 3 months; great. But if the server is evicting values that were last used < 24 hours ago; I have concerns.
There are stats in "stats items" / "stats slabs". Last access time for most recent eviction per slab class, etc. (see doc/protocol.txt from the tarball).
"watch evictions" command will also show a stream of details for items being evicted.
Not sure if there's a better place to ask? But ill just try here. Curious about a design decision in the extstore. It seems to include a lot of extra stuff around managing writes and whats in memory and whats on disk. Why do you think this is better then just mmap-ing and letting the OS decide whats in memory using the fs cache and what pages are still disk?
That's an excellent question; it turns out there are a _lot_ of semantics that the OS is covering up for you when using mmap. For instance (this may be fixed by now), but any process doing certain mmap syscalls locked access to any open mmap's in an OS. So some random cronjob firing could clock your mmap'ed app pretty solidly.
There are also wild bugs; if you google my threads on the LKML you'll find me trying to hunt down a few in the past.
Mainly what I'm doing with extstore is maintaining a clear line between what I want the OS doing and what I want the app doing: a hard rule that the memcached worker threads _cannot_ be blocked for any reason. When they submit work to extstore, they submit to background threads then return to dequeueing network traffic. If the flash disk hiccups for any reason it means some queue's can bloat but other ops may still succeed.
Further, by controlling when we defrag or drop pages we can be more careful with where writes to flash happen.
TLDR: for predictable performance. Extstore is also a lot simpler than it may sound; it's a handful of short functions built on a lot of design decisions instead of a lot of code building up an algorithm.
Everything that interacts with the disk is extstore.c, most of the wrapper code that glues memcached with extstore is storage.c. extstore.c has barely changed since I first wrote it; so there's not much maintenance overhead vs mmap anyway.
It was an algorithmic/lock scaling limit. Originally it was single threaded, then when it was first multi-threaded it scaled up to 4 threads. Then I split up some locks and it scaled to 8 threads (depending). Then I rewrote the LRU and now reads mostly scale linearly and writes don't. If there's enough interest we'll make writes scale better.
Partly this is because the software is so old that the thread scalability tends to track how many CPU's people actually have.
Memcached's consistent hashing is really its killer feature nowadays, but to put this into play you need large clusters of hosts where you expect regular rotations in and out of the pool (hardware failure etc.). One of the links at the bottom of the article is a good overview: https://holmeshe.me/understanding-memcached-source-code-X-co...
> In a benchmark I made for the company I work for, Redis was only 30 - 40% faster than Postgres for our use cases because of the network trips.
Obviously I'm not familiarized with your use case, but there are a couple of things to keep in mind when performing these kind of analysis; The first one is connection cost & limit - Postgres client connection cost is quite high when compared to Redis (thats why tools like PGPool exist). Also, the upper bound connection limit in Postgres is quite low, compared to what you can achieve with Redis.
Then there's the caching issue - it seems, from your description - you were actually benchmarking Postgres & filesystem cache. This happens if your dataset is small enough and your query is simple enough, but also indicates the database is idle - no other queries are running that will compete for cache resources or trigger disk I/O or buffer invalidation. If this is not your production scenario, the numbers you measured are meaningless.
Following the cache issue, by the description (and how well it fitted in cache), it seems you're not analyzing throughput. Reading 10 items is different from reading 10^6 items, and you will quickly find out that the throughput of your database server is several times lower than what you can achieve with Redis, specially considering you can easily store your data compressed. It may or may not be relevant to your particular case, but its always good to also check this.
Finally, there is the predictability aspect of the design. Having data available with predictable performance cost on a system that will happily handle tens of thousands of "simultaneous" operations, with a very low connection cost, instead of hammering your complex, multi purpose database is often a godsend, even if the performance difference was 0.
Another use case to consider is pods. If you’re using Docker/Kubernetes and you have 10 pods, then you technically have 10 in memory caches. The chance of any one of them getting hit is random. Using redis centralizes this.
I quite like Redis as a cache or as an ad hoc locking mechanism accessible from across a cluster, but as a database I will never fully trust it. RDB and AOF seem far too fragile to trust in production, especially in a distributed setting.
There was a multi threaded fork somewhere but I can't really see a real life scenario where this could be beneficial. If your setup is done well, your gains would probably be in the low double digit milliseconds. Surely the least significant bottleneck.
In my experience working on performance with redis (and memcache) latency doesn't slowly creep up with more load. They're fast until they're not. Some of the multi threaded forks exist because during profiling when it finally breaks the slowest parts is all the request and connection handling.
I also think that having threads helps separate points that can fail. Separate connection handling means that can slow and get jammed up but established connections can still get requests through. Slow requests don't slow down everything. Since redis does everything in the one thread stuff like evictions all stop the request handling. Its like having a gc pause in a way. also end to end encryption seems to be becoming popular in data centers and establishing those connections can be slow.
I don't think your wrong though, probably for 95% of developers redis with single thread is fine. It wouldn't be popular if it wasn't. But to be fair to myself I was asking for perfection :) At that point I dont think their would be any significant reason to pick memcache over redis.
I feel like this article is frustrating. They talk about some design choices of each, but never really connect it to use-cases or trade-offs as they apply in practise. The title of the article is "more different than you expect", but i came away feeling like there was even less differences (in ways that matter) than i expected.
I guess it felt like the article didn't really deliver on what it promised and instead gave something else. Which would have been fine if the topic was more clearly introduced.
Redis is not a cache, it is in-memory database. Aside of key-value storage with optional key TTL it has complex data structures with complex operations over them, it has horizontal and vertical clustering, atomic operations, disk persistence, and it can disable data eviction.
Having worked with both, to me an untalked about advantage of memcached is it's a simpler tool - there are less ways to shoot yourself in the foot with it. From a code point of view the only real choice is whether you let LRU exclusively decide when expiry happens or whether you're attaching expiring times to values.
Meanwhile Redis - used in a large codebase - seems to be tempting for developers to use it in different ways, sometimes as a cache, sometimes as a database. That in turn makes problems for devops who have to start worrying about how different redis instances are being used.
The beginning of article acknowledges that Redis can do a bunch more things and explicitly scopes things down to caching use case:
> None of the the above are going to be covered in any detail. If you do need any of the above functionality, then the answer is use Redis, don't read any further and go for a walk outside.
The only reason of using Redis as a dumb cache I can justify is an unwillingness to introduce another tech into tight stack. In my brief experience memcache also has better performance at scale.
I see this mistake all the time, even in print! It's millis -> micros -> nanos, so...that's a lot more than a single order of magnitude.