There are too many advantages in doing this, RDB is faster to persist,
more compact, much faster to load back. The main issues here are that
the code is less tested because this was not the old default (so we are
enabling it for the new 5.0 release), and that the AOF is no longer a
trivially parsable format from now on. However the non-preamble mode
will be supported in the future as well, if new data types will be
added.
This commit, in some parts derived from PR #3041 which is no longer
possible to merge (because the user deleted the original branch),
implements the ability of slaves to have a special configuration
preventing that they try to start a failover when the master is failing.
There are multiple reasons for wanting this, and the feautre was
requested in issue #3021 time ago.
The differences between this patch and the original PR are the
following:
1. The flag is saved/loaded on the nodes configuration.
2. The 'myself' node is now flag-aware, the flag is updated as needed
when the configuration is changed via CONFIG SET.
3. The flag name uses NOFAILOVER instead of NO_FAILOVER to be consistent
with existing NOADDR.
4. The redis.conf documentation was rewritten.
Thanks to @deep011 for the original patch.
- big keys are not defragged in one go from within the dict scan
instead they are scanned in parts after the main dict hash bucket is done.
- add latency monitor sample for defrag
- change default active-defrag-cycle-min to induce lower latency
- make active defrag start a new scan right away if needed, so it's easier
(for the test suite) to detect when it's done
- make active defrag quick the current cycle after each db / big key
- defrag some non key long term global allocations
- some refactoring for smaller functions and more reusable code
- during dict rehashing, one scan iteration of the dict, can end up scanning
one bucket in the smaller dict and many many buckets in the larger dict.
so waiting for 16 scan iterations before checking the time, may be much too long.
You can still force the logo in the normal logs.
For motivations, check issue #3112. For me the reason is that actually
the logo is nice to have in interactive sessions, but inside the logs
kinda loses its usefulness, but for the ability of users to recognize
restarts easily: for this reason the new startup sequence shows a one
liner ASCII "wave" so that there is still a bit of visual clue.
Startup logging was modified in order to log events in more obvious
ways, and to log more events. Also certain important informations are
now more easy to parse/grep since they are printed in field=value style.
The option --always-show-logo in redis.conf was added, defaulting to no.
The gist of the changes is that now, partial resynchronizations between
slaves and masters (without the need of a full resync with RDB transfer
and so forth), work in a number of cases when it was impossible
in the past. For instance:
1. When a slave is promoted to mastrer, the slaves of the old master can
partially resynchronize with the new master.
2. Chained slalves (slaves of slaves) can be moved to replicate to other
slaves or the master itsef, without requiring a full resync.
3. The master itself, after being turned into a slave, is able to
partially resynchronize with the new master, when it joins replication
again.
In order to obtain this, the following main changes were operated:
* Slaves also take a replication backlog, not just masters.
* Same stream replication for all the slaves and sub slaves. The
replication stream is identical from the top level master to its slaves
and is also the same from the slaves to their sub-slaves and so forth.
This means that if a slave is later promoted to master, it has the
same replication backlong, and can partially resynchronize with its
slaves (that were previously slaves of the old master).
* A given replication history is no longer identified by the `runid` of
a Redis node. There is instead a `replication ID` which changes every
time the instance has a new history no longer coherent with the past
one. So, for example, slaves publish the same replication history of
their master, however when they are turned into masters, they publish
a new replication ID, but still remember the old ID, so that they are
able to partially resynchronize with slaves of the old master (up to a
given offset).
* The replication protocol was slightly modified so that a new extended
+CONTINUE reply from the master is able to inform the slave of a
replication ID change.
* REPLCONF CAPA is used in order to notify masters that a slave is able
to understand the new +CONTINUE reply.
* The RDB file was extended with an auxiliary field that is able to
select a given DB after loading in the slave, so that the slave can
continue receiving the replication stream from the point it was
disconnected without requiring the master to insert "SELECT" statements.
This is useful in order to guarantee the "same stream" property, because
the slave must be able to accumulate an identical backlog.
* Slave pings to sub-slaves are now sent in a special form, when the
top-level master is disconnected, in order to don't interfer with the
replication stream. We just use out of band "\n" bytes as in other parts
of the Redis protocol.
An old design document is available here:
https://gist.github.com/antirez/ae068f95c0d084891305
However the implementation is not identical to the description because
during the work to implement it, different changes were needed in order
to make things working well.
This feature is useful, especially in deployments using Sentinel in
order to setup Redis HA, where the slave is executed with NAT or port
forwarding, so that the auto-detected port/ip addresses, as listed in
the "INFO replication" output of the master, or as provided by the
"ROLE" command, don't match the real addresses at which the slave is
reachable for connections.
An exposed Redis instance on the internet can be cause of serious
issues. Since Redis, by default, binds to all the interfaces, it is easy
to forget an instance without any protection layer, for error.
Protected mode try to address this feature in a soft way, providing a
layer of protection, but giving clues to Redis users about why the
server is not accepting connections.
When protected mode is enabeld (the default), and if there are no
minumum hints about the fact the server is properly configured (no
"bind" directive is used in order to restrict the server to certain
interfaces, nor a password is set), clients connecting from external
intefaces are refused with an error explaining what to do in order to
fix the issue.
Clients connecting from the IPv4 and IPv6 lookback interfaces are still
accepted normally, similarly Unix domain socket connections are not
restricted in any way.
It's hard to pick a good approach here. A few arguments:
1) There are many exposed instances on the internet.
2) Changing the default when "bind" is not given is very dangerous,
after an upgrade the server changes a fundamental behavior.
3) Usually Redis, when used in a proper way, will be protected *and*
accessed often from other computers, so this new default is likely
not what most people want.
4) However if users end with this default, they are using the example
redis.conf: likely they are reading what is inside, and they'll see
the warning.
Adds configuration option 'supervised [no | upstart | systemd | auto]'
Also removed 'bzero' from the previous implementation because it's 2015.
(We could actually statically initialize those structs, but clang
throws an invalid warning when we try, so it looks bad even though it
isn't bad.)
Fixes#2264
This removes:
- list-max-ziplist-entries
- list-max-ziplist-value
This adds:
- list-max-ziplist-size
- list-compress-depth
Also updates config file with new sections and updates
tests to use quicklist settings instead of old list settings.
- Remove trailing newlines from redis.conf
- Fix comment misspelling
- Clarifies zipEncodeLength usage and a C API mention (#1243, #1242)
- Fix cluster typos (inspired by @papanikge #1507)
- Fix rewite -> rewrite in a few places (inspired by #682)
Closes#1243, #1242, #1507
According to unix manuals, "Connecting to the socket object requires
read/write permission." -- mode 755 is useless for anybody
other than the owner.
Fixes#1696
Because of output buffer limits Redis internals had this idea of type of
clients: normal, pubsub, slave. It is possible to set different output
buffer limits for the three kinds of clients.
However all the macros and API were named after output buffer limit
classes, while the idea of a client type is a generic one that can be
reused.
This commit does two things:
1) Rename the API and defines with more general names.
2) Change the class of clients executing the MONITOR command from "slave"
to "normal".
"2" is a good idea because you want to have very special settings for
slaves, that are not a good idea for MONITOR clients that are instead
normal clients even if they are conceptually slave-alike (since it is a
push protocol).
The backward-compatibility breakage resulting from "2" is considered to
be minimal to care, since MONITOR is a debugging command, and because
anyway this change is not going to break the format or the behavior, but
just when a connection is closed on big output buffer issues.
This is safer as by default maxmemory should just set a memory limit
without any key to be deleted, unless the policy is set to something
more relaxed.
In high RPS environments, the default listen backlog is not sufficient, so
giving users the power to configure it is the right approach, especially
since it requires only minor modifications to the code.
IPv6 support is not going to use IPv6 specific options, just it will be
possible to specify all the ipv4 / ipv6 addresses of the interfaces to
bind, otherwise connections will be accepted from all the interfaces in
both IPv4 and IPv6 addresses.
This reverts commit 93570e179e.
This feature allows the user to specify the minimum number of
connected replicas having a lag less or equal than the specified
amount of seconds for writes to be accepted.
Also the logfile option was modified to always have an explicit value
and to log to stdout when an empty string is used as log file.
Previously there was special handling of the string "stdout" that set
the logfile to NULL, this always required some special handling.
When keyspace events are enabled, the overhead is not sever but
noticeable, so this commit introduces the ability to select subclasses
of events in order to avoid to generate events the user is not
interested in.
The events can be selected using redis.conf or CONFIG SET / GET.
REDIS_HZ is the frequency our serverCron() function is called with.
A more frequent call to this function results into less latency when the
server is trying to handle very expansive background operations like
mass expires of a lot of keys at the same time.
Redis 2.4 used to have an HZ of 10. This was good enough with almost
every setup, but the incremental key expiration algorithm was working a
bit better under *extreme* pressure when HZ was set to 100 for Redis
2.6.
However for most users a latency spike of 30 milliseconds when million
of keys are expiring at the same time is acceptable, on the other hand a
default HZ of 100 in Redis 2.6 was causing idle instances to use some
CPU time compared to Redis 2.4. The CPU usage was in the order of 0.3%
for an idle instance, however this is a shame as more energy is consumed
by the server, if not important resources.
This commit introduces HZ as a runtime parameter, that can be queried by
INFO or CONFIG GET, and can be modified with CONFIG SET. At the same
time the default frequency is set back to 10.
In this way we default to a sane value of 10, but allows users to
easily switch to values up to 500 for near real-time applications if
needed and if they are willing to pay this small CPU usage penalty.
Two limits are added:
1) Up to SLOWLOG_ENTRY_MAX_ARGV arguments are logged.
2) Up to SLOWLOG_ENTRY_MAX_STRING bytes per argument are logged.
3) slowlog-max-len is set to 128 by default (was 1024).
The number of remaining arguments / bytes is logged in the entry
so that the user can understand better the nature of the logged command.
This commit introduces support for read only slaves via redis.conf and CONFIG GET/SET commands. Also various semantical fixes are implemented here:
1) MULTI/EXEC with only read commands now work where the server is into a state where writes (or commands increasing memory usage) are not allowed. Before this patch everything inside a transaction would fail in this conditions.
2) Scripts just calling read-only commands will work against read only
slaves, when the server is out of memory, or when persistence is into an
error condition. Before the patch EVAL always failed in this condition.
Added a configuration directive to allow a user to specify the
permissions to be granted to the Unix socket file. I followed
the format Pieter and Salvatore discusses in issue #85 (
https://github.com/antirez/redis/issues/85).
$ make
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g adlist.c
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g ae.c
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g anet.c
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g dict.c
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g redis.c
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g sds.c
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g zmalloc.c
cc -o redis-server -O2 -Wall -W -DSDS_ABORT_ON_OOM -g adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o
Hint: To run the test-redis.tcl script is a good idea.
Launch the redis server with ./redis-server, then in another
terminal window enter this directory and run 'make test'.
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g benchmark.c
cc -o redis-benchmark -O2 -Wall -W -DSDS_ABORT_ON_OOM -g ae.o anet.o benchmark.o sds.o adlist.o zmalloc.o
cc -c -O2 -Wall -W -DSDS_ABORT_ON_OOM -g redis-cli.c
cc -o redis-cli -O2 -Wall -W -DSDS_ABORT_ON_OOM -g anet.o sds.o adlist.o redis-cli.o zmalloc.o
$ tail -1 redis.conf
$ ./redis-server redis.conf &
[1] 52494
$ - Server started, Redis version 0.08
- DB loaded from disk
- The server is now ready to accept connections on port 6379
. 0 clients connected (0 slaves), 1932 bytes in use
$ make test
tclsh test-redis.tcl
DEL all keys to start with a clean DB . Accepted 127.0.0.1:54490
PASSED
SET and GET an item PASSED
DEL against a single item PASSED
KEYS with pattern PASSED
KEYS to get all keys PASSED
DBSIZE PASSED
DEL all keys PASSED
Very big payload in GET/SET PASSED
SET 10000 numeric keys and access all them in reverse order . DB 0: 10001 keys in 16384 slots HT.
. 1 clients connected (0 slaves), 4885707 bytes in use
PASSED
DBSIZE should be 10001 now PASSED
INCR against non existing key PASSED
INCR against key created by incr itself PASSED
INCR against key originally set with SET PASSED
SETNX target key missing PASSED
SETNX target key exists PASSED
EXISTS PASSED
Zero length value in key. SET/GET/EXISTS PASSED
Commands pipelining PASSED
Non existing command PASSED
Basic LPUSH, RPUSH, LLENGTH, LINDEX PASSED
DEL a list PASSED
Create a long list and check every single element with LINDEX PASSED
Test elements with LINDEX in random access PASSED
LLEN against non-list value error PASSED
LINDEX against non-list value error PASSED
LPUSH against non-list value error PASSED
RPUSH against non-list value error PASSED
RENAME basic usage PASSED
RENAME source key should no longer exist PASSED
RENAME against already existing key PASSED
RENAMENX basic usage PASSED
RENAMENX against already existing key PASSED
RENAMENX against already existing key (2) PASSED
RENAME against non existing source key PASSED
RENAME where source and dest key is the same PASSED
DEL all keys again (DB 0) PASSED
DEL all keys again (DB 1) PASSED
MOVE basic usage PASSED
MOVE against key existing in the target DB PASSED
SET/GET keys in different DBs PASSED
Basic LPOP/RPOP PASSED
LPOP/RPOP against empty list PASSED
LPOP against non list value PASSED
Mass LPUSH/LPOP PASSED
LRANGE basics PASSED
LRANGE inverted indexes PASSED
LRANGE out of range indexes including the full list PASSED
LRANGE against non existing key PASSED
LTRIM basics PASSED
LSET PASSED
LSET out of range index PASSED
LSET against non existing key PASSED
LSET against non list value PASSED
SADD, SCARD, SISMEMBER, SMEMBERS basics PASSED
SADD adding the same element multiple times PASSED
SADD against non set PASSED
SREM basics PASSED
Mass SADD and SINTER with two sets PASSED
SINTERSTORE with two sets PASSED
SINTER against three sets PASSED
SINTERSTORE with three sets PASSED
SAVE - make sure there are all the types as values - DB saved on disk
PASSED
Create a random list . DB 0: 6057 keys in 16384 slots HT.
. DB 1: 3 keys in 16 slots HT.
. 1 clients connected (0 slaves), 1045147 bytes in use
PASSED
SORT with BY against the newly created list PASSED
SORT direct, numeric, against the newly created list PASSED
SORT decreasing sort PASSED
SORT speed, sorting 10000 elements list using BY, 100 times
Average time to sort: 5.28 milliseconds PASSED
SORT speed, sorting 10000 elements list directly, 100 times
Average time to sort: 1.53 milliseconds PASSED
SORT speed, pseudo-sorting 10000 elements list, BY <const>, 100 times
Average time to sort: 0.47 milliseconds PASSED
SORT regression for issue #19, sorting floats - DB saved on disk
PASSED
LREM, remove all the occurrences - DB saved on disk
PASSED
LREM, remove the first occurrence PASSED
LREM, remove non existing element PASSED
LREM, starting from tail with negative count - DB saved on disk
PASSED
LREM, starting from tail with negative count (2) PASSED
MGET - DB saved on disk
PASSED
MGET against non existing key PASSED
MGET against non-string key PASSED
FLUSHALL - DB saved on disk
PASSED
79 tests, 79 passed, 0 failed
. Client closed connection
. 0 clients connected (0 slaves), 1347660 bytes in use
$ kill -15 %1
[1]+ Terminated ./redis-server redis.conf
$ vim redis.conf
$ tail -1 redis.conf
requirepass foobared
$ ./redis-server redis.conf &
[1] 52518
$ - Server started, Redis version 0.08
- DB loaded from disk
- The server is now ready to accept connections on port 6379
. 0 clients connected (0 slaves), 1945 bytes in use
$ make test
tclsh test-redis.tcl
. Accepted 127.0.0.1:54492
DEL all keys to start with a clean DB
ERR operation not permitted
while executing
"::redis::redis_read_reply $fd"
(procedure "::redis::__dispatch__" line 13)
invoked from within
"$r keys *"
("uplevel" body line 2)
invoked from within
"uplevel 1 $code"
(procedure "test" line 4)
invoked from within
"test {DEL all keys to start with a clean DB} {
foreach key [$r keys *] {$r del $key}
$r dbsize
} {0}"
(procedure "main" line 13)
invoked from within
"main 127.0.0.1 6379"
invoked from within
"if {[llength $argv] == 0} {
main 127.0.0.1 6379
} elseif {[llength $argv] == 1 && [lindex $argv 0] eq {stress}} {
stress
} else {
main [li..."
(file "test-redis.tcl" line 630)
. Client closed connection
make: *** [test] Error 1
. 0 clients connected (0 slaves), 2041 bytes in use
. 0 clients connected (0 slaves), 2041 bytes in use
$ kill -15 %1
[1]+ Terminated ./redis-server redis.conf
$ vim test-redis.tcl
$ git diff test-redis.tcl
Jack Drogon