…10353)

* Fix memory leak in RM_StreamIteratorStop
* Fix memory leak in moduleFreeKeyIterator

* Moved configuration storage from a list to a hash table
* Configs are returned in a non-deterministic order. It's possible that a client was relying on order (hopefully not).
* Fixed an esoteric bug where if you did a set with an alias with an error, it would throw an error indicating a bug with the preferred name for that config.

In order to resolve some flaky tests which hard rely on examine memory footprint.
we introduce the following fixes:

# Fix in client-eviction test - by @yoav-steinberg 
Sometime the libc allocator can use different size client struct allocations.
this may cause unexpected memory calculations to fail the test.

# Introduce new DEBUG command for disabling reply buffer resizing
In order to eliminate reply buffer resizing during specific tests.
we introduced the ability to disable (and enable) the resizing cron job

Co-authored-by: yoav-steinberg yoav@redislabs.com

1. since ZSKIPLIST_P is float, using it directly inside the condition used to causes floating point code to be used (gcc/x86)
2. In some operating system(eg.Windows), the largest value returned from random() is 0x7FFF(15bit), so after bitwise AND with 0xFFFF, the probability of the less operation returning true in the while loop's condition is no more equal to ZSKIPLIST_P.
3. In case some library has random() returning int in range [0~ZSKIPLIST_P*65535], the while loop will be an infinite loop.
4. on Linux where RAND_MAX is higher than 0xFFFF, this change actually improves precision (despite not matching the result against a float value)

Cluster node name is not null terminated, so need to be constrained.

Normally, `redis-cli` escapes non-printable data received from Redis, using a custom scheme (which is also used to handle quoted input). When using `--json` this is not desired as it is not compatible with RFC 7159, which specifies JSON strings are assumed to be Unicode and how they should be escaped.

This commit changes `--json` to follow RFC 7159, which means that properly encoded Unicode strings in Redis will result with a valid Unicode JSON.

However, this introduces a new problem with `--json` and data that is not valid Unicode (e.g., random binary data, text that follows other encoding, etc.). To address this, we add `--quoted-json` which produces JSON strings that follow the original redis-cli quoting scheme.

For example, a value that consists of only null (0x00) bytes will show up as:
* `"\u0000\u0000\u0000"` when using `--json`
* `"\\x00\\x00\\x00"` when using `--quoted-json`

Apparently using `\x` produces different results between tclsh 8.5 and
8.6, whereas `\u` is more consistent.

`Expected '*table size: 4096*' to match '*table size: 8192*'`

This test failed once on daily macOS, the reason is because
the bgsave has not stopped after the kill and `after 200`.
So there is a child process and no rehash triggered.

This commit use `waitForBgsave` to wait for it to finish.

Add a new REDISMODULE_EVENT_CONFIG event type for notifying modules when Redis configuration changes.

Add `DEPRECATED` doc_flag.

Currently, CLUSTER NODES is parsed and was not done correctly for IPv6
addresses.

introduced in #10147 since we blocked the first-arg mechanism on subcommands

since #9822, the static reply buffer is no longer part of the client structure, so we need to dismiss it.

#10364)

In some special commands like eval_ro / fcall_ro we allow no-writes commands.
But may-replicate commands are no-writes too, that leads crash when client pause write:

Deleting a stream while a client is blocked XREADGROUP should unblock the client.

The idea is that if a client is blocked via XREADGROUP is different from
any other blocking type in the sense that it depends on the existence of both
the key and the group. Even if the key is deleted and then revived with XADD
it won't help any clients blocked on XREADGROUP because the group no longer
exist, so they would fail with -NOGROUP anyway.
The conclusion is that it's better to unblock these clients (with error) upon
the deletion of the key, rather than waiting for the first XADD. 

Other changes:
1. Slightly optimize all `serveClientsBlockedOn*` functions by checking `server.blocked_clients_by_type`
2. All `serveClientsBlockedOn*` functions now use a list iterator rather than looking at `listFirst`, relying
  on `unblockClient` to delete the head of the list. Before this commit, only `serveClientsBlockedOnStreams`
  used to work like that.
3. bugfix: CLIENT UNBLOCK ERROR should work even if the command doesn't have a timeout_callback
  (only relevant to module commands)

Typo in conf file comment.

c->buf is not sds, so we should use dismissMemory instead of dismissSds to dismiss it.
This is a recent regression from #10371

* stats and latency commands have non-deterministic output.
* the ones about latency should be sent to ALL_NODES (considering
  reads from replicas)
* the ones about running scripts and memory usage only to masters.
* stats aggregation is SPECIAL (like in INFO)

* fix-replication-comments

The described capacity
 `and to schedule a new BGSAVE if there are slaves that attached while a BGSAVE was in progress`
was moved to `checkChildrenDone()`  named by `replicationStartPendingFork`

But the comment was not changed, may misleading others.

* remove-misleading-comments

The described capacity
 `to schedule a new BGSAVE if there are slaves that attached while a BGSAVE was in progress` 
and 
`or when the replication RDB transfer strategy is modified from disk to socket or the other way around` 
were not correct now.

It's confusing for this config to be modifiable since it only takes effect on startup

As a result we segfault when parsing and matching the command keys.

The following usage will output an empty newline:
```
> redis-cli help set
empty line
```

The reason is that in interactive mode, we have called
`cliInitHelp`, which initializes help.

When using `redis-cli help xxx` or `redis-cli help ? xxx`,
we can't match the command due to empty `helpEntries`,
so we output an empty newline.

In this commit, we will call `cliInitHelp` to init the help.
Note that in this case, we need to call `cliInitHelp` (COMMAND DOCS)
every time, which i think is acceptable.

So now the output will look like:
```
[redis]# src/redis-cli help get

  GET key
  summary: Get the value of a key
  since: 1.0.0
  group: string

[redis]#
```

Fixes #10378

This PR also fix a redis-cli crash when using `--ldb --eval`:
```
[root]# src/redis-cli --ldb --eval test.lua test 1
Lua debugging session started, please use:
quit    -- End the session.
restart -- Restart the script in debug mode again.
help    -- Show Lua script debugging commands.

* Stopped at 1, stop reason = step over
-> 1   local num = redis.call('GET', KEYS[1]);
redis-cli: redis-cli.c:718: cliCountCommands: Assertion
`commandTable->element[i]->type == 1' failed.
Aborted
```
Because in ldb mode, `COMMAND DOCS` or `COMMAND` will
return an array, only with one element, and the type
is `REDIS_REPLY_STATUS`, the result is `<error> Unknown
Redis Lua debugger command or wrong number of arguments`.

So if we are in the ldb mode, and init the Redis HELP, we
will get the wrong response and crash the redis-cli.
In ldb mode we don't initialize HELP, help is only initialized
after the lua debugging session ends.

It was broken in #10043

probably a copy paste error.

When updating SENTINEL with master’s new password (command:
`SENTINEL SET mymaster auth-pass some-new-password`), 
sentinel might still keep the old connection and avoid reconnecting 
with the new password. This is because of wrong logic that traces 
the last ping (pong) time to servers. In fact it worked fine until 8631e64 
changed the condition to send ping. To resolve it with minimal risk, 
let’s disconnect master and replicas once changing password/user. 

Based on earlier work of yz1509.

)

Better check the monitors list argument instead of server.monitors in the function,
although they are basically the same in the context, so this doesn't have any
impact on the current code.

#10408)

For an integer string like "123456789012345678901" which could cause
overflow-failure in string2ll() conversion, we could compare its length at
the beginning to avoid extra work.

* move LONG_STR_SIZE to be in declared in util.h, next to MAX_LONG_DOUBLE_CHARS

* missing parenthesis meant that the ubuntu and centos jobs were not
  skipped
* the recently divided freebsd, macos, and valgrind jobs, which are now
  split into distict jobs for redis, modules, sentinel, cluster. were
  all executed, producing a build, but not running anything.
  now they're filtered at the job level
* iothreads was missing from the skip list defaults, so was not skipped

This PR do some command json files cleanups:

1. Add COMMAND TIPS to some commands
- command-docs: add `NONDETERMINISTIC_OUTPUT_ORDER`
- command-info: add `NONDETERMINISTIC_OUTPUT_ORDER`
- command-list: add `NONDETERMINISTIC_OUTPUT_ORDER`
- command: change `NONDETERMINISTIC_OUTPUT` to `NONDETERMINISTIC_OUTPUT_ORDER`
- function-list: add `NONDETERMINISTIC_OUTPUT_ORDER`
- latency-doctor: add `NONDETERMINISTIC_OUTPUT`, `REQUEST_POLICY:ALL_NODES` and `RESPONSE_POLICY:SPECIAL`
- latency-graph: add `NONDETERMINISTIC_OUTPUT`, `REQUEST_POLICY:ALL_NODES` and `RESPONSE_POLICY:SPECIAL`
- memory-doctor: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:SPECIAL`
- memory-malloc-stats: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:SPECIAL`
- memory-purge: add `REQUEST_POLICY:ALL_SHARDS` and `RESPONSE_POLICY:ALL_SUCCEEDED`
- module-list: add `NONDETERMINISTIC_OUTPUT_ORDER`
- msetnx: add `REQUEST_POLICY:MULTI_SHARD` and `RESPONSE_POLICY:AGG_MIN`
- object-refcount: add `NONDETERMINISTIC_OUTPUT`
3. Only (mostly) indentation and formatting changes:
- cluster-shards
- latency-history
- pubsub-shardchannels
- pubsub-shardnumsub
- spublish
- ssubscribe
- sunsubscribe
4. add doc_flags (DEPRECATED) to cluster-slots,  replaced_by `CLUSTER SHARDS` in 7.0
5. command-getkeysandflags: a better summary (the old one is copy from command-getkeys)
6. adjustment of command parameter types
- `port` is integer, not string (`MIGRATE`, `REPLICAOF`, `SLAVEOF`)
- `replicationid` is string, not integer (`PSYNC`)
- `pattern` is pattern, not string (`PUBSUB CHANNELS`, `SENTINEL RESET`, `SORT`, `SORT_RO`)

This feature adds the ability to add four different types (Bool, Numeric,
String, Enum) of configurations to a module to be accessed via the redis
config file, and the CONFIG command.

**Configuration Names**:

We impose a restriction that a module configuration always starts with the
module name and contains a '.' followed by the config name. If a module passes
"config1" as the name to a register function, it will be registered as MODULENAME.config1.

**Configuration Persistence**:

Module Configurations exist only as long as a module is loaded. If a module is
unloaded, the configurations are removed.
There is now also a minimal core API for removal of standardConfig objects
from configs by name.

**Get and Set Callbacks**:

Storage of config values is owned by the module that registers them, and provides
callbacks for Redis to access and manipulate the values.
This is exposed through a GET and SET callback.

The get callback returns a typed value of the config to redis. The callback takes
the name of the configuration, and also a privdata pointer. Note that these only
take the CONFIGNAME portion of the config, not the entire MODULENAME.CONFIGNAME.

```
 typedef RedisModuleString * (*RedisModuleConfigGetStringFunc)(const char *name, void *privdata);
 typedef long long (*RedisModuleConfigGetNumericFunc)(const char *name, void *privdata);
 typedef int (*RedisModuleConfigGetBoolFunc)(const char *name, void *privdata);
 typedef int (*RedisModuleConfigGetEnumFunc)(const char *name, void *privdata);
```

Configs must also must specify a set callback, i.e. what to do on a CONFIG SET XYZ 123
or when loading configurations from cli/.conf file matching these typedefs. *name* is
again just the CONFIGNAME portion, *val* is the parsed value from the core,
*privdata* is the registration time privdata pointer, and *err* is for providing errors to a client.

```
typedef int (*RedisModuleConfigSetStringFunc)(const char *name, RedisModuleString *val, void *privdata, RedisModuleString **err);
typedef int (*RedisModuleConfigSetNumericFunc)(const char *name, long long val, void *privdata, RedisModuleString **err);
typedef int (*RedisModuleConfigSetBoolFunc)(const char *name, int val, void *privdata, RedisModuleString **err);
typedef int (*RedisModuleConfigSetEnumFunc)(const char *name, int val, void *privdata, RedisModuleString **err);
```

Modules can also specify an optional apply callback that will be called after
value(s) have been set via CONFIG SET:

```
typedef int (*RedisModuleConfigApplyFunc)(RedisModuleCtx *ctx, void *privdata, RedisModuleString **err);
```

**Flags:**
We expose 7 new flags to the module, which are used as part of the config registration.

```
#define REDISMODULE_CONFIG_MODIFIABLE 0 /* This is the default for a module config. */
#define REDISMODULE_CONFIG_IMMUTABLE (1ULL<<0) /* Can this value only be set at startup? */
#define REDISMODULE_CONFIG_SENSITIVE (1ULL<<1) /* Does this value contain sensitive information */
#define REDISMODULE_CONFIG_HIDDEN (1ULL<<4) /* This config is hidden in `config get <pattern>` (used for tests/debugging) */
#define REDISMODULE_CONFIG_PROTECTED (1ULL<<5) /* Becomes immutable if enable-protected-configs is enabled. */
#define REDISMODULE_CONFIG_DENY_LOADING (1ULL<<6) /* This config is forbidden during loading. */
/* Numeric Specific Configs */
#define REDISMODULE_CONFIG_MEMORY (1ULL<<7) /* Indicates if this value can be set as a memory value */
```

**Module Registration APIs**:

```
int (*RedisModule_RegisterBoolConfig)(RedisModuleCtx *ctx, char *name, int default_val, unsigned int flags, RedisModuleConfigGetBoolFunc getfn, RedisModuleConfigSetBoolFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
int (*RedisModule_RegisterNumericConfig)(RedisModuleCtx *ctx, const char *name, long long default_val, unsigned int flags, long long min, long long max, RedisModuleConfigGetNumericFunc getfn, RedisModuleConfigSetNumericFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
int (*RedisModule_RegisterStringConfig)(RedisModuleCtx *ctx, const char *name, const char *default_val, unsigned int flags, RedisModuleConfigGetStringFunc getfn, RedisModuleConfigSetStringFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
int (*RedisModule_RegisterEnumConfig)(RedisModuleCtx *ctx, const char *name, int default_val, unsigned int flags, const char **enum_values, const int *int_values, int num_enum_vals, RedisModuleConfigGetEnumFunc getfn, RedisModuleConfigSetEnumFunc setfn, RedisModuleConfigApplyFunc applyfn, void *privdata);
int (*RedisModule_LoadConfigs)(RedisModuleCtx *ctx);
```

The module name will be auto appended along with a "." to the front of the name of the config.

**What RM_Register[...]Config does**:

A RedisModule struct now keeps a list of ModuleConfig objects which look like:
```
typedef struct ModuleConfig {
    sds name; /* Name of config without the module name appended to the front */
    void *privdata; /* Optional data passed into the module config callbacks */
    union get_fn { /* The get callback specificed by the module */
        RedisModuleConfigGetStringFunc get_string;
        RedisModuleConfigGetNumericFunc get_numeric;
        RedisModuleConfigGetBoolFunc get_bool;
        RedisModuleConfigGetEnumFunc get_enum;
    } get_fn;
    union set_fn { /* The set callback specified by the module */
        RedisModuleConfigSetStringFunc set_string;
        RedisModuleConfigSetNumericFunc set_numeric;
        RedisModuleConfigSetBoolFunc set_bool;
        RedisModuleConfigSetEnumFunc set_enum;
    } set_fn;
    RedisModuleConfigApplyFunc apply_fn;
    RedisModule *module;
} ModuleConfig;
```
It also registers a standardConfig in the configs array, with a pointer to the
ModuleConfig object associated with it.

**What happens on a CONFIG GET/SET MODULENAME.MODULECONFIG:**

For CONFIG SET, we do the same parsing as is done in config.c and pass that
as the argument to the module set callback. For CONFIG GET, we call the
module get callback and return that value to config.c to return to a client.

**CONFIG REWRITE**:

Starting up a server with module configurations in a .conf file but no module load
directive will fail. The flip side is also true, specifying a module load and a bunch
of module configurations will load those configurations in using the module defined
set callbacks on a RM_LoadConfigs call. Configs being rewritten works the same
way as it does for standard configs, as the module has the ability to specify a
default value. If a module is unloaded with configurations specified in the .conf file
those configurations will be commented out from the .conf file on the next config rewrite.

**RM_LoadConfigs:**

`RedisModule_LoadConfigs(RedisModuleCtx *ctx);`

This last API is used to make configs available within the onLoad() after they have
been registered. The expected usage is that a module will register all of its configs,
then call LoadConfigs to trigger all of the set callbacks, and then can error out if any
of them were malformed. LoadConfigs will attempt to set all configs registered to
either a .conf file argument/loadex argument or their default value if an argument is
not specified. **LoadConfigs is a required function if configs are registered.
** Also note that LoadConfigs **does not** call the apply callbacks, but a module
can do that directly after the LoadConfigs call.

**New Command: MODULE LOADEX [CONFIG NAME VALUE] [ARGS ...]:**

This command provides the ability to provide startup context information to a module.
LOADEX stands for "load extended" similar to GETEX. Note that provided config
names need the full MODULENAME.MODULECONFIG name. Any additional
arguments a module might want are intended to be specified after ARGS.
Everything after ARGS is passed to onLoad as RedisModuleString **argv.

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>
Co-authored-by: Madelyn Olson <matolson@amazon.com>
Co-authored-by: sundb <sundbcn@gmail.com>
Co-authored-by: Madelyn Olson <34459052+madolson@users.noreply.github.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
Co-authored-by: Yossi Gottlieb <yossigo@gmail.com>

Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](actions/cache@v2...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…tils (#10497)

Create a utils.tcl in sentinel/tests/includes, and move two procs to it.
Allow sentinel test 08-hostname-conf run on its own.

On error, redis-cli was returning `REDIS_ERR` on some cases by mistake. `REDIS_ERR` is `-1` which becomes `255` as exit code. This commit changes it and returns `1` on errors to be consistent.

Fix three timing issues in the test

During 11-manual-takeover.tcl, if the killing of the instances happens
too slowly, one of the replicas might be able to promote itself.
I'm not sure why it was slow, but it was observed taking 6 seconds
which is enough time to do an election.
I was able to verify the error locally by adding a small delay (1 second)
during ASAN CI. A fix is just to disable automated failover until all the
nodes are confirmed dead.

Fix global `strval` not reset to NULL after being freed, causing a crash on alpine
(most likely because the dynamic library loader doesn't init globals on reload)
By the way, fix the memory leak of using `RedisModule_Free` to free `RedisModuleString`,
and add a corresponding test.

* Fix race condition where node loses its last slot and turns into replica

When a node has lost its last slot and finds out from the SETSLOT command
before the cluster bus PONG from the new owner arrives. In this case, the
node didn't turn itself into a replica of the new slot owner.

This commit adds the same logic to the SETSLOT command as already exists
for the cluster bus PONG processing.

* Revert "Fix new / failing cluster slot migration test (#10482)"

This reverts commit 0b21ef8.

In this test, the old slot owner finds out that it has lost its last
slot in a nondeterministic way. Either the cluster bus PONG from the
new slot owner and sometimes in a SETSLOT command from redis-cli. In
both cases, the result should be the same and the old owner should
turn itself into a replica of the new slot owner.

Fix by replacing in test blind sleep with wait_for_condition().

Co-authored-by: moticless <moticless@github.com>

function aofRewriteLimited in aof.c, deley->delay and NAX->MAX

Co-authored-by: judeng <judeng@didiglobal.com>

…10519)

If, for some reason, Redis decides not to execute the script, we need
to pop the function and error handler from Lua stack. Otherwise, eventually
the Lua stack will explode.

Relevant only for 7.0-rc1 and 7.0-rc2.

Unlike original reference code, Redis uses 64bit variable, so half the bits make it 32bit, not 16

The warning:
```
pqsort.c:106:7: warning: performing pointer subtraction with a null pointer has undefined behavior [-Wnull-pointer-subtraction]
loop:   SWAPINIT(a, es);
        ^~~~~~~~~~~~~~~
pqsort.c:65:47: note: expanded from macro 'SWAPINIT'
#define SWAPINIT(a, es) swaptype = ((char *)a - (char *)NULL) % sizeof(long) || \
```
Clang version:
```
Apple clang version 13.1.6 (clang-1316.0.21.2)
Target: x86_64-apple-darwin21.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
```

It is really outdated and i guessing no one will use it now. 
Remove the corresponding utils/hashtable directory.
closes #10235

…10527)

This macro was recently removed from redismodule.h, so no longer needed.

* Limit cluster node id length for CLUSTER commands loading
* Cluster node name sanity check for length and values

Co-authored-by: Madelyn Olson <madelyneolson@gmail.com>

…10500)

## Move library meta data to be part of the library payload.

Following the discussion on #10429 and the intention to add (in the future) library versioning support, we believe that the entire library metadata (like name and engine) should be part of the library payload and not provided by the `FUNCTION LOAD` command. The reasoning behind this is that the programmer who developed the library should be the one who set those values (name, engine, and in the future also version). **It is not the responsibility of the admin who load the library into the database.**

The PR moves all the library metadata (engine and function name) to be part of the library payload. The metadata needs to be provided on the first line of the payload using the shebang format (`#!<engine> name=<name>`), example:

```lua
#!lua name=test
redis.register_function('foo', function() return 1 end)
```

The above script will run on the Lua engine and will create a library called `test`.

## API Changes (compare to 7.0 rc2)

* `FUNCTION LOAD` command was change and now it simply gets the library payload and extract the engine and name from the payload. In addition, the command will now return the function name which can later be used on `FUNCTION DELETE` and `FUNCTION LIST`.
* The description field was completely removed from`FUNCTION LOAD`, and `FUNCTION LIST`


## Breaking Changes (compare to 7.0 rc2)

* Library description was removed (we can re-add it in the future either as part of the shebang line or an additional line).
* Loading an AOF file that was generated by either 7.0 rc1 or 7.0 rc2 will fail because the old command syntax is invalid.

## Notes

* Loading an RDB file that was generated by rc1 / rc2 **is** supported, Redis will automatically add the shebang to the libraries payloads (we can probably delete that code after 7.0.3 or so since there's no need to keep supporting upgrades from an RC build).

…tion (#10398)

The command json documents should just include information about the "arguments" and the "outputs".
I removed all of the 'functional wording' so it's clear.