其他
浅析Redis 4.0新特性之LazyFree
1.Redis作者
Redis uses a standard practice for its versioning: major.minor.patchlevel. An even minor marks a stable release, like 1.2, 2.0, 2.2, 2.4, 2.6, 2.8. Odd minors are used for unstable releases, for example 2.9.x releases are the unstable versions of what will be Redis 3.0 once stable. redis.io官方说明
版本号第二位如果是奇数，则为非稳定版本 如2.7、2.9、3.1
版本号第二位如果是偶数，则为稳定版本 如2.6、2.8、3.0、3.2
当前奇数版本就是下一个稳定版本的开发版本，如2.9版本是3.0版本的开发版本
3.Redis版本迭代简介
4.Redis4.0版本新功能
4.0版本官方feature说明
Redis 4.0 was released as GA in July 2017, newcomers should use Redis 5, but Redis 4 is currently the most production-proven release and will be updated for the next year until Redis 6 will be out. It contains several big improvements: ---------------------------------------------1.a modules system, 2.much better replication (PSYNC2), 3.improvements to eviction policies, 4.threaded DEL/FLUSH, 5.mixed RDB+AOF format, 6.Raspberry Pi support as primary platform, 7.the new MEMORY command, 8.Redis Cluster support for Nat/Docker, 9.active memory defragmentation, 10.memory usage and performance improvements, 11.much faster Redis Cluster key creation, 12.many other smaller features and a number of behavior fixed. 13.others-----------------------------------------------------------新功能详细说明
模块系统(modules system)
4.0版本允许根据自己需求独立开发modules去扩展Redis的额外功能，模块系统的接口与Redis内核完全分离，有些类似Nginx的lua插件。
部分重传PSYNCv2
解决了旧版本从机器重启必须与主机器重新进行全量复制的问题以及从机器在failover后成为新的主节点，在旧版中其他从节点在复制新主时就必须进行全量复制的问题，上述两种情况在新版中可使用部分复制。
缓存淘汰改进
添加了LFU 缓存淘汰策略，对已有的缓存策略进行了优化。
多线程异步删除
混合持久化功能
内存统计命令
5.异步删除LazyFree
删除机制
单线程删除阻塞问题
我们生产Redis Cluster大集群，业务缓慢地写入一个带有TTL的2000多万个字段的Hash键，当这个键过期时，redis开始被动清理它时，导致redis被阻塞20多秒，当前分片主节点因20多秒不能处理请求，并发生主库故障切换。
https://www.jianshu.com/p/e927e99e650d
异步删除命令
6.Redis的BIO线程
创建BIO线程
BIO_CLOSE_FILE对应的关闭文件描述符线程
BIO_AOF_FSYNC对应的aof持久化冲刷磁盘线程
BIO_LAZY_FREE对应的异步惰性删除线程
http://download.redis.io/redis-stable/src/bio.c
http://download.redis.io/redis-stable/src/bio.h
redis bio相关源码
//4.0版本增加了BIO_LAZY_FREE 之前并没有这个宏/* Background job opcodes */#define BIO_CLOSE_FILE 0 /* Deferred close(2) syscall. */#define BIO_AOF_FSYNC 1 /* Deferred AOF fsync. */#define BIO_LAZY_FREE 2 /* Deferred objects freeing. */#define BIO_NUM_OPS 3//主线程启动多个辅助后台线程void bioInit(void) { pthread_attr_t attr; pthread_t thread; size_t stacksize; int j;
// 初始化各任务类型的锁和条件变量, BIO_NUM_OPS 个 for (j = 0; j < BIO_NUM_OPS; j++) { pthread_mutex_init(&bio_mutex[j],NULL); pthread_cond_init(&bio_condvar[j],NULL); bio_jobs[j] = listCreate(); bio_pending[j] = 0; }
//设置 stack 大小 pthread_attr_init(&attr); pthread_attr_getstacksize(&attr,&stacksize); if (!stacksize) stacksize = 1; /* The world is full of Solaris Fixes */ while (stacksize < REDIS_THREAD_STACK_SIZE) stacksize *= 2; pthread_attr_setstacksize(&attr, stacksize);
// 创建线程 for (j = 0; j < BIO_NUM_OPS; j++) { void *arg = (void*)(unsigned long) j; if (pthread_create(&thread,&attr,bioProcessBackgroundJobs,arg) != 0) { serverLog(LL_WARNING,"Fatal: Can't initialize Background Jobs."); exit(1); } bio_threads[j] = thread; }}主线程和BIO线程的交互
主要流程：
创建3个线程.这个三个线程的功能互不影响 相互独立
每个线程都有一个工作队列，主线程生产任务放到任务队里，三个线程消费队列中的任务
主线程和BIO线程作为生产者和消费者，从队列添加和消费任务时都得加锁防止竞争状态
BIO使用条件变量来等待任务，以及通知，典型的Pro-Con模型
//主线程和BIO线程之间的工作队列 使用双链表实现static list *bio_jobs[REDIS_BIO_NUM_OPS];//任务结构struct bio_job { time_t time; /* Time at which the job was created. */ /* Job specific arguments pointers. If we need to pass more than three * arguments we can just pass a pointer to a structure or alike. */ void *arg1, *arg2, *arg3;};//主线程向队列中增加任务void bioCreateBackgroundJob(int type, void *arg1, void *arg2, void *arg3) { struct bio_job *job = zmalloc(sizeof(*job)); job->time = time(NULL); job->arg1 = arg1; job->arg2 = arg2; job->arg3 = arg3; pthread_mutex_lock(&bio_mutex[type]); listAddNodeTail(bio_jobs[type],job); bio_pending[type]++; pthread_cond_signal(&bio_condvar[type]); pthread_mutex_unlock(&bio_mutex[type]);}//BIO线程从工作队列中获取任务并处理void *bioProcessBackgroundJobs(void *arg) { struct bio_job *job; unsigned long type = (unsigned long) arg;
pthread_detach(pthread_self()); pthread_mutex_lock(&bio_mutex[type]); while(1) { listNode *ln;
/* The loop always starts with the lock hold. */ if (listLength(bio_jobs[type]) == 0) { pthread_cond_wait(&bio_condvar[type],&bio_mutex[type]); continue; } /* Pop the job from the queue. */ ln = listFirst(bio_jobs[type]); job = ln->value; /* It is now possible to unlock the background system as we know have * a stand alone job structure to process.*/ pthread_mutex_unlock(&bio_mutex[type]);
/* Process the job accordingly to its type. */ if (type == REDIS_BIO_CLOSE_FILE) { close((long)job->arg1); } else if (type == REDIS_BIO_AOF_FSYNC) { aof_fsync((long)job->arg1); } else { redisPanic("Wrong job type in bioProcessBackgroundJobs()."); } zfree(job);
/* Lock again before reiterating the loop, if there are no longer * jobs to process we'll block again in pthread_cond_wait(). */ pthread_mutex_lock(&bio_mutex[type]); listDelNode(bio_jobs[type],ln); bio_pending[type]--; }}使用双链表作为工作队列
bio_job任务结构
主线程生产函数
BIO线程消费函数
一些插曲
后来使用了异步线程方案，主线程只需要将需要删除的对象放入队列，异步线程从队列里取出对象来释放逻辑，说起来容易Antirez在实际处理多种结构时也遇到了许多问题，比如共享对象等，这时就需要做权衡，Antirez选择了lazyfree而全部解决旧版本中的共享对象问题，也就是为了支持新功能解决痛点修改了之前的一些基础数据结构，可见4.0相比3.x确实是有很多改变。
https://zhuanlan.zhihu.com/p/41754417
https://www.jianshu.com/p/d39a213362bd
https://www.cnblogs.com/liuhao/archive/2012/05/17/2506810.html