redis/src
Meir Shpilraien (Spielrein) aa856b39f2
Sort out the mess around Lua error messages and error stats (#10329)
This PR fix 2 issues on Lua scripting:
* Server error reply statistics (some errors were counted twice).
* Error code and error strings returning from scripts (error code was missing / misplaced).

## Statistics
a Lua script user is considered part of the user application, a sophisticated transaction,
so we want to count an error even if handled silently by the script, but when it is
propagated outwards from the script we don't wanna count it twice. on the other hand,
if the script decides to throw an error on its own (using `redis.error_reply`), we wanna
count that too.
Besides, we do count the `calls` in command statistics for the commands the script calls,
we we should certainly also count `failed_calls`.
So when a simple `eval "return redis.call('set','x','y')" 0` fails, it should count the failed call
to both SET and EVAL, but the `errorstats` and `total_error_replies` should be counted only once.

The PR changes the error object that is raised on errors. Instead of raising a simple Lua
string, Redis will raise a Lua table in the following format:

```
{
    err='<error message (including error code)>',
    source='<User source file name>',
    line='<line where the error happned>',
    ignore_error_stats_update=true/false,
}
```

The `luaPushError` function was modified to construct the new error table as describe above.
The `luaRaiseError` was renamed to `luaError` and is now simply called `lua_error` to raise
the table on the top of the Lua stack as the error object.
The reason is that since its functionality is changed, in case some Redis branch / fork uses it,
it's better to have a compilation error than a bug.

The `source` and `line` fields are enriched by the error handler (if possible) and the
`ignore_error_stats_update` is optional and if its not present then the default value is `false`.
If `ignore_error_stats_update` is true, the error will not be counted on the error stats.

When parsing Redis call reply, each error is translated to a Lua table on the format describe
above and the `ignore_error_stats_update` field is set to `true` so we will not count errors
twice (we counted this error when we invoke the command).

The changes in this PR might have been considered as a breaking change for users that used
Lua `pcall` function. Before, the error was a string and now its a table. To keep backward
comparability the PR override the `pcall` implementation and extract the error message from
the error table and return it.

Example of the error stats update:

```
127.0.0.1:6379> lpush l 1
(integer) 2
127.0.0.1:6379> eval "return redis.call('get', 'l')" 0
(error) WRONGTYPE Operation against a key holding the wrong kind of value. script: e471b73f1ef44774987ab00bdf51f21fd9f7974a, on @user_script:1.

127.0.0.1:6379> info Errorstats
# Errorstats
errorstat_WRONGTYPE:count=1

127.0.0.1:6379> info commandstats
# Commandstats
cmdstat_eval:calls=1,usec=341,usec_per_call=341.00,rejected_calls=0,failed_calls=1
cmdstat_info:calls=1,usec=35,usec_per_call=35.00,rejected_calls=0,failed_calls=0
cmdstat_lpush:calls=1,usec=14,usec_per_call=14.00,rejected_calls=0,failed_calls=0
cmdstat_get:calls=1,usec=10,usec_per_call=10.00,rejected_calls=0,failed_calls=1
```

## error message
We can now construct the error message (sent as a reply to the user) from the error table,
so this solves issues where the error message was malformed and the error code appeared
in the middle of the error message:

```diff
127.0.0.1:6379> eval "return redis.call('set','x','y')" 0
-(error) ERR Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479): @user_script:1: OOM command not allowed when used memory > 'maxmemory'.
+(error) OOM command not allowed when used memory > 'maxmemory' @user_script:1. Error running script (call to 71e6319f97b0fe8bdfa1c5df3ce4489946dda479)
```

```diff
127.0.0.1:6379> eval "redis.call('get', 'l')" 0
-(error) ERR Error running script (call to f_8a705cfb9fb09515bfe57ca2bd84a5caee2cbbd1): @user_script:1: WRONGTYPE Operation against a key holding the wrong kind of value
+(error) WRONGTYPE Operation against a key holding the wrong kind of value script: 8a705cfb9fb09515bfe57ca2bd84a5caee2cbbd1, on @user_script:1.
```

Notica that `redis.pcall` was not change:
```
127.0.0.1:6379> eval "return redis.pcall('get', 'l')" 0
(error) WRONGTYPE Operation against a key holding the wrong kind of value
```


## other notes
Notice that Some commands (like GEOADD) changes the cmd variable on the client stats so we
can not count on it to update the command stats. In order to be able to update those stats correctly
we needed to promote `realcmd` variable to be located on the client struct.

Tests was added and modified to verify the changes.

Related PR's: #10279, #10218, #10278, #10309

Co-authored-by: Oran Agra <oran@redislabs.com>
2022-02-27 13:40:57 +02:00
..
commands Add stream consumer group lag tracking and reporting (#9127) 2022-02-23 22:34:58 +02:00
modules Add 'Available since' to module API function docs (#10229) 2022-02-03 10:25:37 +02:00
.gitignore
Makefile Build commands.c in Makefile (#10039) 2022-01-06 17:59:37 +02:00
acl.c Implemented module getchannels api and renamed channel keyspec (#10299) 2022-02-22 11:00:03 +02:00
adlist.c Free value if dup succeed but listAddNodeTail failed. (#8901) 2021-05-03 13:39:07 +03:00
adlist.h Speedup: unblock clients on keys in O(1). 2020-04-08 12:55:57 +02:00
ae.c Add event loop support to the module API (#10001) 2022-01-18 13:10:07 +02:00
ae.h Add event loop support to the module API (#10001) 2022-01-18 13:10:07 +02:00
ae_epoll.c Fail fast when systemic error occurs in poll (#8749) 2021-04-26 15:52:06 +03:00
ae_evport.c Fix cluster bus extensions backwards compatibility (#10206) 2022-01-30 19:43:37 +02:00
ae_kqueue.c Fix the timing of read and write events under kqueue (#9416) 2021-09-02 11:07:51 +03:00
ae_select.c Fail fast when systemic error occurs in poll (#8749) 2021-04-26 15:52:06 +03:00
anet.c Fix some nonsense came from LGTM (#9962) 2021-12-19 17:52:23 +02:00
anet.h Implement anetPipe() to combine creating pipe and setting flags (#9511) 2021-10-06 16:08:13 +03:00
aof.c Add stream consumer group lag tracking and reporting (#9127) 2022-02-23 22:34:58 +02:00
asciilogo.h Changes http to https in texts (#8495) 2021-03-10 19:11:16 +02:00
atomicvar.h Several (mostly Solaris-related) cleanups (#8171) 2020-12-13 17:09:54 +02:00
bio.c Delete unused 'time' fields from struct bio_job (#9622) 2021-10-10 08:17:54 +03:00
bio.h Delete some unimplemented prototype. (#8882) 2021-04-29 08:25:10 +03:00
bitops.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
blocked.c Fix error stats and failed command stats for blocked clients (#10309) 2022-02-21 11:20:41 +02:00
call_reply.c Fix and improve module error reply statistics (#10278) 2022-02-13 18:37:32 +02:00
call_reply.h Fix and improve module error reply statistics (#10278) 2022-02-13 18:37:32 +02:00
childinfo.c Implement anetPipe() to combine creating pipe and setting flags (#9511) 2021-10-06 16:08:13 +03:00
cli_common.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
cli_common.h Added URI support to redis-benchmark (cli and benchmark share the same uri-parsing methods) (#9314) 2021-09-14 19:45:06 +03:00
cluster.c Add stream consumer group lag tracking and reporting (#9127) 2022-02-23 22:34:58 +02:00
cluster.h Show publishshard_sent stat in cluster info (#10314) 2022-02-19 21:11:20 -08:00
commands.c Add stream consumer group lag tracking and reporting (#9127) 2022-02-23 22:34:58 +02:00
config.c Set default channel permission to resetchannels for 7.0 (#10181) 2022-01-30 12:02:55 +02:00
config.h Check somaxconn system settings on macOS, FreeBSD and OpenBSD. (#9972) 2021-12-28 09:20:10 +02:00
connection.c Reduce system calls of write for client->reply by introducing writev (#9934) 2022-02-22 14:00:37 +02:00
connection.h Reduce system calls of write for client->reply by introducing writev (#9934) 2022-02-22 14:00:37 +02:00
connhelpers.h Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
crc16.c
crc16_slottable.h Added basic support for clusters to redis-benchmark. 2019-03-01 17:53:14 +01:00
crc64.c Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
crc64.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
crcspeed.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
crcspeed.h Added crcspeed library 2020-04-24 17:11:21 -07:00
db.c Delete key doesn't dirty client who watched stale key (#10256) 2022-02-22 12:09:46 +02:00
debug.c introduce dynamic client reply buffer size - save memory on idle clients (#9822) 2022-02-22 11:19:38 +02:00
debugmacro.h Supplement define guards to prevent multiple inclusion (#10246) 2022-02-06 20:13:34 -08:00
defrag.c Fix script active defrag test (#10318) 2022-02-21 09:37:25 +02:00
dict.c Fix some nonsense came from LGTM (#9962) 2021-12-19 17:52:23 +02:00
dict.h Fix possible int overflow when hashing an sds. (#9916) 2021-12-13 21:16:25 +02:00
endianconv.c Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
endianconv.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
eval.c Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
evict.c use startEvictionTimeProc() in config set maxmemory (#10019) 2022-01-04 13:08:10 +02:00
expire.c sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127) 2022-01-23 10:05:06 +02:00
fmacros.h Fix failed tests on Linux Alpine and add a CI job. (#8532) 2021-02-23 12:57:45 +02:00
function_lua.c Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
functions.c update function help message, changed DESC->DESCRIPTION. (#10228) 2022-02-02 15:50:41 +02:00
functions.h Function Flags support (no-writes, no-cluster, allow-state, allow-oom) (#10066) 2022-01-14 14:02:02 +02:00
geo.c Fix some nonsense came from LGTM (#9962) 2021-12-19 17:52:23 +02:00
geo.h
geohash.c FIX: truncate max/min longitude,latitude related geo_point (ex: {180, 85.05112878} ) 2020-04-09 17:48:29 +08:00
geohash.h Remove duplicate header file include (#10264) 2022-02-08 16:49:47 +02:00
geohash_helper.c Fix geo search bounding box check causing missing results (#10018) 2022-02-21 08:06:58 +02:00
geohash_helper.h Delete some unimplemented prototype. (#8882) 2021-04-29 08:25:10 +03:00
help.h Update FCALL/FCALL_RO summary and complexity (#10240) 2022-02-05 10:35:07 +02:00
hyperloglog.c Sort out the mess around writable replicas and lookupKeyRead/Write (#9572) 2021-11-28 11:26:28 +02:00
intset.c Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
intset.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
latency.c sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127) 2022-01-23 10:05:06 +02:00
latency.h Disable THP if enabled (#7381) 2020-10-27 15:04:18 +02:00
lazyfree.c Redis Function Libraries (#10004) 2022-01-06 13:39:38 +02:00
listpack.c Santize dump payload: fix invalid listpack entry start with EOF (#9889) 2021-12-04 16:43:08 +02:00
listpack.h Replace ziplist with listpack in quicklist (#9740) 2021-11-24 13:34:13 +02:00
listpack_malloc.h Optimize listpack for stream usage to avoid repeated reallocs (#6281) 2021-02-16 16:17:38 +02:00
localtime.c update leap year comment when div by 400 2018-09-18 17:04:00 +08:00
lolwut.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lolwut.h add include guard for lolwut.h 2020-05-05 23:35:08 -04:00
lolwut5.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lolwut6.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
lzf.h Change lzf to handle values larger than UINT32_MAX (#9776) 2021-11-16 13:12:25 +02:00
lzfP.h Change lzf to handle values larger than UINT32_MAX (#9776) 2021-11-16 13:12:25 +02:00
lzf_c.c Change lzf to handle values larger than UINT32_MAX (#9776) 2021-11-16 13:12:25 +02:00
lzf_d.c Change lzf to handle values larger than UINT32_MAX (#9776) 2021-11-16 13:12:25 +02:00
memtest.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
mkreleasehdr.sh Update mkreleasehdr.sh 2019-11-19 17:23:47 +08:00
module.c Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
monotonic.c Use H/W Monotonic clock and updates to AE (#7644) 2020-08-28 11:54:10 +03:00
monotonic.h Use H/W Monotonic clock and updates to AE (#7644) 2020-08-28 11:54:10 +03:00
mt19937-64.c Fix random element selection for large hash tables. (#8133) 2020-12-23 15:52:07 +02:00
mt19937-64.h Fix random element selection for large hash tables. (#8133) 2020-12-23 15:52:07 +02:00
multi.c Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
networking.c Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
notify.c Fixed some typos, add a spell check ci and others minor fix (#8890) 2021-06-10 15:39:33 +03:00
object.c Implement Multi Part AOF mechanism to avoid AOFRW overheads. (#9788) 2022-01-03 19:14:13 +02:00
pqsort.c
pqsort.h
pubsub.c Sharded pubsub implementation (#8621) 2022-01-02 16:54:47 -08:00
quicklist.c Fix quicklist node not being recompressed correctly after inserting a new node before or after it (#10120) 2022-01-16 08:54:40 +02:00
quicklist.h Fix recent daily CI test failures (#9966) 2021-12-20 12:31:13 +02:00
rand.c
rand.h
rax.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
rax.h Squash merging 125 typo/grammar/comment/doc PRs (#7773) 2020-09-10 13:43:38 +03:00
rax_malloc.h Cluster: hash slots tracking using a radix tree. 2017-03-27 16:37:22 +02:00
rdb.c Add stream consumer group lag tracking and reporting (#9127) 2022-02-23 22:34:58 +02:00
rdb.h Add stream consumer group lag tracking and reporting (#9127) 2022-02-23 22:34:58 +02:00
redis-benchmark.c Enable redis-benchmark to use RESP3 protocol mode (#10335) 2022-02-27 10:30:39 +02:00
redis-check-aof.c Fix path copy error and add more logs. (#10324) 2022-02-22 10:09:34 +02:00
redis-check-rdb.c Added functions support to redis-check-rdb (#10154) 2022-01-20 11:10:33 +02:00
redis-cli.c redis-cli generates command help tables from the results of COMMAND (#10043) 2022-02-05 16:54:16 +02:00
redis-trib.rb Redis-trib deprecated: it no longer works and it 2018-07-13 10:51:58 +02:00
redisassert.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
redisassert.h Sanitize dump payload: performance optimizations and tuning 2020-12-06 14:54:34 +02:00
redismodule.h Implemented module getchannels api and renamed channel keyspec (#10299) 2022-02-22 11:00:03 +02:00
release.c Speedup INFO server section. 2019-10-02 11:30:20 +02:00
replication.c Fix a mistake in comments (#10312) 2022-02-20 08:09:19 +02:00
resp_parser.c Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202) 2021-08-04 16:28:07 +03:00
resp_parser.h Unified Lua and modules reply parsing and added RESP3 support to RM_Call (#9202) 2021-08-04 16:28:07 +03:00
rio.c Fix when the master connection is disconnected, replication retry read indefinitely (#10032) 2021-12-31 14:08:04 +02:00
rio.h diskless replication rdb transfer uses pipe, and writes to sockets form the parent process. 2019-10-07 21:06:30 +03:00
script.c Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
script.h Support function flags in script EVAL via shebang header (#10126) 2022-01-24 16:50:02 +02:00
script_lua.c Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
script_lua.h Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
sds.c Fix additional AOF filename issues. (#10110) 2022-01-18 12:52:27 +02:00
sds.h Fix additional AOF filename issues. (#10110) 2022-01-18 12:52:27 +02:00
sdsalloc.h Sanitize dump payload: fail RESTORE if memory allocation fails 2020-12-06 14:54:34 +02:00
sentinel.c Fixed typo in variable name (#10347) 2022-02-26 19:52:09 -08:00
server.c Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
server.h Sort out the mess around Lua error messages and error stats (#10329) 2022-02-27 13:40:57 +02:00
setcpuaffinity.c cpu affinity: DragonFlyBSD support (#7956) 2020-10-25 14:14:05 +02:00
setproctitle.c Fix failed tests on Linux Alpine and add a CI job. (#8532) 2021-02-23 12:57:45 +02:00
sha1.c Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
sha1.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
sha256.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
sha256.h fix explanation of sha256 (#9220) 2021-07-10 10:04:54 -05:00
siphash.c Add sanitizer support and clean up sanitizer findings (#9601) 2021-11-11 13:51:33 +02:00
slowlog.c slowlog get command supports passing in -1 to get all logs. (#9018) 2021-06-14 16:46:45 +03:00
slowlog.h Auto-generate the command table from JSON files (#9656) 2021-12-15 21:23:15 +02:00
solarisfixes.h
sort.c Redis Functions - Introduce script unit. 2021-12-01 23:54:23 +02:00
sparkline.c Squash merging 125 typo/grammar/comment/doc PRs (#7773) 2020-09-10 13:43:38 +03:00
sparkline.h
stream.h Add stream consumer group lag tracking and reporting (#9127) 2022-02-23 22:34:58 +02:00
syncio.c
t_hash.c sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127) 2022-01-23 10:05:06 +02:00
t_list.c sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127) 2022-01-23 10:05:06 +02:00
t_set.c Sort out the mess around writable replicas and lookupKeyRead/Write (#9572) 2021-11-28 11:26:28 +02:00
t_stream.c Fix XGROUP HELP message missing a newline (#10339) 2022-02-24 08:31:39 +02:00
t_string.c sub-command support for ACL CAT and COMMAND LIST. redisCommand always stores fullname (#10127) 2022-01-23 10:05:06 +02:00
t_zset.c Optimization: Avoid deferred array reply on ZRANGE commands BYRANK (#10337) 2022-02-24 14:20:00 +02:00
testhelp.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
timeout.c Add missing calls to raxStop (#7532) 2020-07-21 08:13:05 +03:00
tls.c Reduce system calls of write for client->reply by introducing writev (#9934) 2022-02-22 14:00:37 +02:00
tracking.c ACL V2 - Selectors and key based permissions (#9974) 2022-01-20 13:05:27 -08:00
util.c Fix additional AOF filename issues. (#10110) 2022-01-18 12:52:27 +02:00
util.h Fix additional AOF filename issues. (#10110) 2022-01-18 12:52:27 +02:00
valgrind.sup Sanitize dump payload: fuzz tester and fixes for segfaults and leaks it exposed 2020-12-06 14:54:34 +02:00
version.h Add Module API for version and compatibility checks (#7865) 2020-10-11 17:21:58 +03:00
ziplist.c fix invalid read on corrupt ziplist (#9831) 2021-11-23 14:56:52 +02:00
ziplist.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
zipmap.c Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
zipmap.h Add --large-memory flag for REDIS_TEST to enable tests that consume more than 100mb (#9784) 2021-11-16 08:55:10 +02:00
zmalloc.c zmalloc_get_rss openbsd implementation (#10149) 2022-01-19 20:56:12 +02:00
zmalloc.h Added INFO LATENCYSTATS section: latency by percentile distribution/latency by cumulative distribution of latencies (#9462) 2022-01-05 14:01:05 +02:00