Redis for Administrators and Developers

From Training Material
Jump to navigation Jump to search

THIS IS A DRAFT

This text may not be complete.

title
Redis for Administrators and Developers
author
Alexander Patrakov

Redis Overview ⌘

  • Fast in-memory NoSQL database
    • "REmote DIctionary Server"
    • More than 1 000 000 GETs/sec on a typical developer's laptop, even more on servers
    • Unique feature: performance guarantees for each command in Big-O notation
  • Different from memcached
    • Offers disk-based persistence
    • Has more than just strings to offer
  • Different from MongoDB
    • Redis data types are closely related to fundamental data structures, not JSON
    • No rich queries, just simple commands

Redis Releases ⌘

  • Major.Minor.Micro versioning
    • Excellent backwards compatibility even between major versions
    • Security fixes are provided (?) in the form of micro releases
  • Support policy: two latest minor versions
    • But actually there were no 3.0.x releases after 3.2.0
    • Security implications: always use the latest version?
    • Or rely on a Linux distribution to backport the security patches
      • Verify that they do backport security patches!
    • The reality may be not that sad: version 3.2.10 was released after 4.0.1, with a number of fixes

Redis Installation ⌘

  • Use packages? Sometimes they are good enough!
    • Debian/Ubuntu: apt install redis-server # from the main repository?
    • CentOS: yum install redis # from EPEL
    • Arch: pacman -S redis # from the community repository
    • SuSE: zypper install redis # from the server:database repository
  • Redis developers would like you to compile Redis from sources
    • So that you have the latest version

Installation on Windows ⌘

  • There was a fork by Microsoft Open Technologies
    • Redis-32 (very outdated) and Redis-64 NuGet packages available
    • MSI releases also available
    • Latest release in 2016, based on redis-3.0 codebase

What's new ⌘

  • Redis 4.0.x: 14-Jul-2017 - Present
    • Redis Cluster is now compatible with NAT
    • New replication engine
    • Memory optimizations, including online defragmentation
    • Background deletes
    • Redis modules system
  • Redis 3.2.x: 06-May-2016 - Present
    • Redis Sentinel is now compatible with NAT (3.2.2)
    • New command for efficient bit-field manipulation
    • New set of commands for geo indexing
    • Memory optimizations
      • May or may not fully apply to versions in Linux distributions - bundled vs system jemalloc issue
  • Redis 3.0.x: 01-Apr-2015 - 28-Jan-2016
  • Redis 2.8.x: 22-Nov-2013 - 18-Dec-2015

Typical Package contents ⌘

  • Binaries:
    • /usr/bin/redis-server
    • /usr/bin/redis-cli
    • /usr/bin/redis-sentinel
      • Symlink to redis-server
    • /usr/bin/redis-trib
    • /usr/bin/redis-benchmark
    • /usr/bin/redis-check-aof
    • /usr/bin/redis-check-rdb
  • Distributions add files such as:
    • /usr/lib/systemd/system/redis.service
    • /etc/logrotate.d/redis
    • /etc/redis.conf

What does what ⌘

  • redis-server: the server
    • Can start in standalone mode, in sentinel mode, or in cluster mode
  • redis-cli: a command-line client
    • Used by sysadmins for debugging, monitoring and reconfiguration
    • Also used in training courses to make them programming-language-neutral
  • redis-benchmark: the official way of benchmarking Redis
  • redis-check-aof, redis-check-rdb: check and possibly fix the structure of Redis database
  • redis-trib: a Ruby script that reconfigures a cluster
    • Redis 3.0+
    • Missing in Arch Linux because of unpackaged Ruby Redis client library
      • The missing library can be installed via gem, and the script will then work

Configuration file ⌘

  • Usually located at /etc/redis.conf or /etc/redis/redis.conf
  • Extensive documentation in the form of comments
  • The name of the configuration file is passed to redis-server as a parameter
  • Redis can be asked to rewrite its own configuration after dynamic changes
    • Here is how: redis-cli CONFIG REWRITE
      • Fails in Arch: (error) ERR Rewriting config file: Read-only file system
    • This feature is used by Redis Sentinel and Redis Cluster
    • For standalone Redis servers, a read-only configuration file is better

Security notes ⌘

  • Redis is insecure
    • Allows full access (including CONFIG SET and CONFIG REWRITE) to everyone who can connect and knows the password
  • Security features:
    • The "bind" directive (specifies on which IP and port to accept connections)
      • Defaults to 127.0.0.1 in Debian, missing elsewhere
    • The "requirepass" directive
      • Sets global password (no username)
  • Protected mode
    • If activated, and there is no "bind" and no "requirepass", don't accept commands on non-local connections

Security notes, continued ⌘

  • There is no SSL
    • Run Redis on a trusted network
    • Use stunnel/spiped/IPSec if absolutely needed
  • Don't run Redis as root
    • Or your root's authorized_keys will be overwritten from a hacked webapp
  • Don't let HTTP clients send queries to Redis
    • There was a browser-based demo that steals data
    • Does not work anymore - Redis developers have added protections against cross-protocol scripting
  • Set up a firewall
  • Don't ignore protections (apparmor, systemd ProtectSystem) set up by distributions!

How Redis works ⌘

  • At startup, loads the existing RDB or AOF file
    • Populates internal data structures in memory
  • Listens on sockets
  • Handles all clients in one thread using epoll
    • Attempts to read commands from clients
    • Attempts to write any buffered output to clients
      • Beware of clients and slaves behind slow networks - memory bloat issue
    • Executes one command at a time, puts any output into the buffer
      • Beware of slow commands - latency issue for other clients
  • If configured, preiodically forks and saves its state into RDB file

Versions in distributions ⌘

  • Data current as of 28-Aug-2017
  • Debian Stable: 2:3.2.6-1
    • No significant patches
    • Cross-protocol scripting protection is missing!
  • Debian Testing & Backports: 4:4.0.1-7 and 4:4.0.1-3~bpo9+1
    • Latest version, the Debian version churn is about dropping non-deterministic tests
  • Ubuntu 16.04 LTS: 2:3.0.6-1
    • Never updated since Ubuntu release
    • Is in Universe, so no bugfixes and no security support
  • Ubuntu 17.04: 2:3.2.1-1
    • So no NAT support in Sentinel

Versions in distributions ⌘

  • Arch always has the latest Redis version
  • Fedora 26: Redis 3.2.8
  • CentOS: install from EPEL
    • CentOS 6: Redis 2.4.10
    • CentOS 7: Redis 3.2.3
  • OpenSuSE 42.3: Redis 3.2.9-84.11
    • No significant patches over vanilla 3.2.9

Installation from source ⌘

  • No dependencies except gcc and related packages
  • make -j4
    • Takes 18 seconds on a Lenovo Yoga 2 Pro laptop
    • Possible tweaks: OPTIMIZATION=..., MALLOC=...
    • Default: OPTIMIZATION=-O2, MALLOC=jemalloc
  • Or: make 32bit
  • make test
    • Needs tcl installed
    • 2 minutes here, everything passes
    • Debian maintainers complain that some tests are sensitive to timing or otherwise unreliable
  • sudo make install
    • Installs in /usr/local/bin
    • Override: PREFIX=...

Situation with memory allocators ⌘

  • Default on Linux: MALLOC=jemalloc
    • Jemalloc-4.0.3 is bundled with redis-4.0.1
    • There was an upgrade attempt to 4.4.0, which failed due to deadlocks
    • Other options: libc, jemalloc, tcmalloc, tcmalloc_minimal
  • Distributions often undo the bundling of jemalloc
    • So, in Debian, redis uses the system jemalloc 3.6.0

Quick test without installation ⌘

  • ./src/redis-server ./redis.conf
  • ./redis-cli
    • See example session below
    • List of all commands (this training only covers a subset)
    • There is also a help command that provides the same help!
  • kill `pidof redis-server`
    • Look, there is a dump.rdb file!
127.0.0.1:6379> set foo bar
OK
127.0.0.1:6379> get foo
"bar"
127.0.0.1:6379> get boom
(nil)
127.0.0.1:6379> quit

Connecting to other servers/databases ⌘

  • redis-cli -h host -p port
    • TCP connection
  • redis-cli -s /run/redis/redis.sock
    • UNIX socket, will be faster, but can be used for local connections only
  • redis-cli -d database
    • By default, there are 16 databases, numbered from 0 (default) to 15
  • redis-cli -a password
    • The password is per-server
    • There is no username
  • There is no SSL - use stunnel or spiped

How to start and stop redis ⌘

  • Start: redis-server /etc/redis.conf
    • Will start listening on 127.0.0.1:6379 by default
  • Stop: two ways to do it
    • redis-cli shutdown # specify the port if you changed it
    • Or just kill the main process with SIGTERM
    • Both ways can fail in different cases
  • Linux distributions typically provide an init script or a systemd unit that does it for you

Integration with systemd ⌘

  • Should work out of the box if you installed from package
  • Otherwise, copy-paste the unit from Arch Linux or from Debian
    • Needs systemd >= 225 (released in August 2015)


Recommended sysctls ⌘

  • vm.overcommit_memory=1
    • The default is 0, can prevent a big (> RAMSIZE/2) Redis from forking
    • Redis needs to fork for saving its RDB file, for rewriting AOF, and for talking to slaves
  • net.core.somaxconn=512
    • TCP backlog size
  • Disable transparent hugepages
    • They cause latency spikes when Redis forks
    • Don't do this using /etc/rc.local - it would be too late!
    • The preferred method is the transparent_hugepage=never kernel argument
    • Another good way: two lines in /etc/tmpfiles.d/no-thp.conf:
      • w /sys/kernel/mm/transparent_hugepage/enabled - - - - never
      • w /sys/kernel/mm/transparent_hugepage/defrag - - - - never

What else is needed ⌘

  • Your application that wants to connect to Redis!
  • Libraries exist for many programming languages
    • C: hiredis
    • Python: redis
      • Can use hiredis for speedup
    • PHP: phpredis or Predis
    • Node.js: io_redis or node_redis

Accessing Redis from C# ⌘

  • The most popular libraries are StackExchange.Redis (open-source MIT license) and ServiceStack.Redis (commercial)
    • Free trial of ServiceStack.Redis is limited to 6000 queries per hour
      • Version 3.9.71 was the last truly free, but ServiceStack.Redis 3.x has been discontinued in 2014
    • There are other Redis client libraries
  • We can use StackExchange.Redis
    • Available on NuGet as StackExchange.Redis or StackExchange.Redis.StrongName
      • Linux/Mono users, please install StackExchange.Redis.Mono instead, or build from source
        • Either use monobuild.sh, or use "Mono" configuration
        • FEATURE_SOCKET_MODE_POLL is not supported, that's why
    • Reference it like this: using StackExchange.Redis; // even for Linux

Simple example: counter ⌘

Let's demonstrate the INCR command 

$ redis-cli
127.0.0.1:6379> incr counter:foo
(integer) 1
127.0.0.1:6379> incr counter:foo
(integer) 2
127.0.0.1:6379> incr counter:foo
(integer) 3
127.0.0.1:6379> incr counter:foo
(integer) 4

Naming Redis keys ⌘

  • Convention: use multi-part names with ":" (or sometimes "|") as a namespace separator
    • "object-type:id" is a good idea
    • Think about colons in untrusted user data that becomes part of the key name
  • All bytes are OK
    • But "*", "#" and "->" have a special meaning for SORT, so may be a problem
    • "{" and "}" have a special meaning for Redis Cluster
  • Redis doesn't care about UTF-8
    • Again, except in the SORT command, but that's about values
  • The empty string is also a valid key
  • Too-long names (>1000 bytes) are slow

Simple example: counter (C#) ⌘

using System;
using StackExchange.Redis;

namespace RedisCounter
{
  class MainClass
  {
    public static void Main (string[] args)
    {
      IConnectionMultiplexer conn = 
          ConnectionMultiplexer.Connect ("127.0.0.1");
      IDatabase db = conn.GetDatabase (0);
      long counter = db.StringIncrement ("counter:foo");
      Console.WriteLine (String.Format("The counter is {0}", counter));
    }
  }
}

Simple example: counter (Python) ⌘

#!/usr/bin/python2

import redis

conn = redis.client.StrictRedis.from_url("redis://127.0.0.1/0")
counter = conn.incr("counter:foo")
print "The counter is", counter

The only exercise for today ⌘

  • Counting votes
    • You can't do it all now. After I explain each data structure, think how it can help with the progress.
  • Users can vote (+1 or -1 or neutral) on articles
    • Each user and each article are identified by a numeric ID
    • Articles also have publication dates
  • Track the score that each article has
  • Prevent users from voting twice on the same article
    • Or if you want to complicate things: allow them to change their mind!
  • Find top 10 articles for a time period between 7 days ago and now
  • Make sure that votes are lost (and voting is impossible) when an article is deleted
  • It should be possible to delete all of the particular user's votes, too

Redis data types ⌘

  • Strings
  • Lists
  • Sets
  • Hashes
  • Sorted sets
  • Bitmaps (in fact, special operations on strings)
  • HyperLogLogs (2.8.9+)
  • Geoindexes (3.2.0+)

Redis data types, continued ⌘

  • The real question: which type is right for my use case?
    • Answer: most likely, a combination of several types
      • Yes, redundant data - but that's also how SQL databases keep indexes
    • Consider all possible requests that make sense now from a business standpoint
    • Store data separately for each use case
    • Common mistake: stored data, but don't know how to find it without the slow KEYS command

Strings ⌘

  • Are binary-safe
    • Commonly used to represent a number, a string, or just a blob of data
  • Use these commands:
    • GET key
      • Returns (nil) if the key does not exist
      • (nil) is different from an empty string
    • SET key value
    • APPEND key value
      • Non-existing keys are treated as holding an empty string

Redis strings in bindings ⌘

  • StackExchange.Redis names for these commands:
    • var res = db.StringGet(key);
      • key is of type RedisKey, but you can pass a string
      • Results are of the RedisValue type, see the next slides
    • bool success = db.StringSet(key, value);
      • Can fail only because of additional arguments not listed here
    • long len = db.StringAppend(key, value);
  • Pyredis just uses lowercased command names everywhere

Strings as numbers ⌘

  • Use these commands:
    • GET key, SET key value
    • INCR key, INCRBY key by, INCRBYFLOAT key by
    • DECR key, DECRBY key by, DECRBYFLOAT key by
      • When incrementing or decrementing a non-existing key, it is assumed to hold 0
  • StackExchange.Redis names for these commands:
    • StringGet(), StringSet()
    • overloaded variants of StringIncrement()
      • long StringIncrement (RedisKey key, long value=1);
      • double StringIncrement (RedisKey key, double value);
    • overloaded variants of StringDecrement()

RedisKey type ⌘

  • Accepted by almost all IDatabase methods
    • Used for things that can affect server selection in a cluster scenario
  • Can be implicitly converted from byte[] or from string
    • UTF-8 representation
  • Deals with key prefixes
    • Just a way to say that the key name is a concatenation of two strings
  • Not needed in Python: strings just work

RedisValue type ⌘

  • Returned by StringGet()
  • Needed in C# because of static typing
    • Provides conversions to string, byte[], int
    • Can be tested for (nil): bool isNil = value.IsNull;
    • Supports conversions to Nullable<T> types: var value = (int?)db.StringGet("abc");
  • Not needed in Python
    • The variable can contain a string or a None as needed

Some more string operations ⌘

  • Get string length: STRLEN key
    • Returns 0 if the key doesn't exist
    • C#: long StringLength(RedisKey key);
  • Substrings can be stored and retrieved: SETRANGE key offset value, GETRANGE key start end
    • In SETRANGE, the assumed length is the length of the value
      • The reply is the length of the resulting string
    • Good for arrays of fixed-size objects with O(1) access time
    • C#:
      • RedisValue StringSetRange(RedisKey key, long offset, RedisValue value);
      • RedisValue StringGetRange(RedisKey key, long start, long end);

Deleting, testing and renaming keys ⌘

  • DEL key1 key2 key2 ...: deleted the named keys
  • EXISTS key1 key2 ...: returns the number of listed keys that exist
  • RENAME old new: renames the key
    • Effectively deletes the new key if both keys exist
  • C#:
    • bool KeyDelete(RedisKey key);
    • long KeyDelete(RedisKey[] keys); // Pitfall: empty array causes exception
    • bool KeyExists(RedisKey key);
      • No way to test multiple keys for existence from C# or Python in one call
    • bool KeyRename(RedisKey key, RedisKey newKey);
  • DEL and RENAME commands work with all data types, not only with strings

Hashes ⌘

  • Contain a whole mapping of field-value pairs
    • Something like Redis inside Redis, under its own name
      • Design decision: objects as hashes vs json in strings
    • There is no Redis inside Redis inside Redis
  • Important commands: HSET, HGET, HSTRLEN, HEXISTS, HDEL, HINCR, HINCRBY, HINCRBYFLOAT, HDECR, HDECRBY, HDECRBYFLOAT
    • You already know them - they work just like their cousins without H, but take the key as the first argument
    • HSET key field value
    • bool HashSet(RedisKey key, RedisValue hashField, RedisValue value);
  • There is no HRENAME, HSETRANGE
  • HKEYS returns all fields, HVALS gets all values, HGETALL gets both, HLEN counts fields

Lists ⌘

  • Ordered collections of strings
    • Similar to linked lists - not C++ deque!
  • Fast Operations: O(1)
    • LPUSH/RPUSH key value: prepend/append a value
    • LPOP/RPOP key: remove and get the value from either end
      • Return (nil) when applied to an empty list
    • LLEN key: length of a list

Lists, continued ⌘

  • Slow operations: O(n) where n is the number of elements to traverse
    • LINSERT mylist AFTER Hello There: finds "Hello", inserts "There" after it
    • LINDEX key index: reads value by index
      • index works like in Python: 0 = leftmost, -1 = rightmost (and these cases are fast)
      • Use sorted sets instead if fast access by index is needed
    • LRANGE key start stop: reads multiple values (stop is inclusive, unlike in Python)
    • LSET key index value: sets a particular value
    • LREM key count value: remove elements equal to value
      • count = 0 means all, count > 0 means some first elements, count < 0 traverses from the right
      • In the worst case, traverses all elements

LTRIM ⌘

  • Tricky operation: O(n), where n is the number of values to remove
    • LTRIM key start stop: remove everything outside the specified range
    • So O(1) if it removes at most only one element (use case: capped collection)
      • RPUSH log:general "INFO: nothing has happened"
      • LTRIM log:general 0 999

List commands in C# ⌘

  • RedisValue ListGetByIndex(RedisKey key, long index);
  • long ListInsertBefore(RedisKey key, RedisValue pivot, RedisValue value); // and After
  • RedisValue ListLeftPop(RedisKey key); // and Right
  • long ListLeftPush(RedisKey key, RedisValue value);
  • long ListLeftPush(RedisKey key, RedisValue[] values);
  • long ListLength(RedisKey key);
  • RedisValue[] ListRange(RedisKey key, long start = 0, long stop = -1);
  • long ListRemove(RedisKey key, RedisValue value, long count = 0);
  • void ListSetByIndex(RedisKey key, long index, RedisValue value);
  • void ListTrim(RedisKey key, long start, long stop);

Sets ⌘

  • Like hashes, but without values
    • No intrinsic ordering
  • Important commands:
    • SADD key member1 mebber2 ... : adds members to the set
      • O(n) where n is the number of members to be added
    • SREM key member1 member2 ... : removes from the set
    • SCARD key : how many members are there?
    • SISMEMBER key candidate : does the candidate exist in the set?
    • SMEMBERS key : return all of them
    • SPOP key: remove and return some random member

Sets, continued ⌘

  • Sets support bulk operations
    • But they are slow: O(N + M + ...)
    • SINTER set1 set2 ...: returns all members of the intersection
    • SUNION set1 set2 ...: union
    • SDIFF a b c d ...: elements of a that don't occur in b, c, d or ...
  • Server-side store available
    • SINTERSTORE dest set1 set2 ...
    • SUNIONSTORE dest set1 set2 ...
    • SDIFFSTORE dest a b c d ...

Sorted sets ⌘

  • Like sets, but include a rank (a floating-point number) with each member
    • Members are intrinsically ordered by rank (ascending)
    • Members with the same rank are ordered lexicographically as arrays of bytes
  • Insertion of an element: ZADD key score1 value1 score2 value2 ...
    • O(log N) where N is the set size
  • Can look at ranges:
    • ZRANGE key start stop // by index
    • ZRANGEBYLEX key min max // by dictionary order
      • works correctly only if all members in the set have the same score
    • ZRANGEBYSCORE // by score
  • The following commands also work:
    • ZSCORE key member, ZRANK key member, ZCARD key, ZREM key member

Bit arrays ⌘

  • Not really a new data type - just a view on strings
  • Commands: GETBIT, SETBIT, BITOP [AND|OR|NOT|XOR], BITCOUNT, BITOPS
  • Since Redis 3.2.0, it is possible to use a string as a collection of fields representing many small integers
    • Use the BITFIELD command
    • Wrapping, saturating or overflow-checked, signed or unsigned arithmetics
    • Not currently available in Python or C# bindings

HyperLogLogs ⌘

  • Task: count distinct values in incoming stream of strings
  • Constraint: memory
    • Cannot store them all
  • Solution: probabilistic algorithm
  • Uses ~12K bytes per the estimator
    • Actually a string - no separate data type
  • Commands: PFADD key value, PFCOUNT key
  • Also: PFMERGE dst src src ...
  • C# methods: HyperLogLogAdd, HyperLogLogLength, HyperLogLogMerge

Geoindexes ⌘

  • Available in Redis 3.2.0+
    • No separate data structure - just a special case of sorted set
  • Not currently available in StackExchange.Redis
  • Available in git version of Python redis module
  • Store locations (with names and coordinates): GEOADD key lon lat name
    • Locations too close (<5 degrees latitude) to the poles cannot be added
    • No GEODEL, use ZREM instead
    • Get them back: GEOPOS key name
  • Calculate distances: GEODIST key name1 name2
  • Find locations close to a given point: GEORADIUS key lon lat radius m|km|mi|ft

Modules ⌘

  • Available in Redis 4.0.0+
    • "Shared objects" that can be loaded by Redis
      • Loading from config file: loadmodule /path/to/mymodule.so
      • MODULE LOAD command also exists
    • Typically written in C
    • Can provide arbitrary commands and data structures
    • Redis modules hub

Modules (C#) ⌘

  • var result = db.Execute("SOME COMMAND");
    • That's a recent API addition, 2017-04-29
  • The result is of RedisResult type
    • You can cast it to string, int, array of strings, RedisKey, RedisValue, and many other types, as needed to correctly interpret Redis output
    • Also you can test it for (nil): result.IsNull()
  • For some "important" modules, like RediSearch, there is explicit support of commands provided by them
    • So no need to run db.Execute()

Modules (Python) ⌘

  • There is a StrictRedis.execute_command() method that can run an arbitrary command
  • There is also StrictRedis.set_response_callback() that can be used so that the Redis reply is converted to the correct type

Pub/Sub ⌘

  • To subscribe to channels: SUBSCRIBE ch1 ch2 ...
    • No further commands allowed other than SUBSCRIBE, PSUBSCRIBE, UNSUBSCRIBE, PUNSUBSCRIBE, PING and QUIT
    • PSUBSCRIBE and PUNSUBSCRIBE work on channel name patterns
  • To publish to a channel: PUBLISH ch message
    • Returns the number of clients that receive it
  • PUBSUB is a debugging command to list channels or count subscribers
  • It is possible for Redis to auto-publish certain events in keyspace
    • Look for "notify-keyspace-events" in the config file
    • Off by default

Pub/Sub unreliability ⌘

  • Clients sometimes disconnect and reconnect
  • Messages sent during that time are lost for them
  • Solution (if lost messages are a problem):
    • Keep real messages in lists
    • Publish only notifications that the list has changed
    • On reconnection, assume that the list has changed

Pub/Sub from C# ⌘

  • Commands are issued on ISubscriver interface
  • ISubscriber sub = conn.GetSubscriber();
  • long Publish(RedisChannel channel, RedisValue message);
  • void Subscribe(RedisChannel channel, Action<RedisChannel, RedisValue> handler);
    • That action is just a function that accepts two arguments
      • Treat them as strings or byte arrays
    • Called from a separate thread
      • Message ordering is a separate topic
      • Want to process messages concurrently? conn.PreserveAsyncOrder = false;

Blocking list operations ⌘

  • Useful for stacks and queues
  • BLPOP, BRPOP key1 key2 ... timeout
  • Try to pop a key from one of the specified lists
    • Wait until something appears or until the timeout expires
    • Lists are checked in the order given
    • Timeout is in seconds, integer, 0 = block forever
  • Not available in StackExchange.Redis
    • And will never be, because of incompatibility with multiplexing a single connection among multiple clients
    • Instead, use normal list commands for data and Pub/Sub for notifications

Transactions ⌘

  • A way to group operations so that nothing else happens in between
    • It makes sense to combine transactions with pipelining
  • WATCH key1 key2 ...
  • GET ... or other operations
  • MULTI
  • SET ...
    • QUEUED
  • SET ...
    • QUEUED
  • EXEC
    • <actual replies for commands in the transaction>
    • Or a (nil) if at least one of the watched keys has changed

Transactions (C#) ⌘

  • StackExchange.Redis multiplexes connections
  • So can't use WATCH directly - it would affect other threads
  • Instead, some canned tests are available - and that's all
var tran = db.CreateTransaction();
tran.AddCondition(Condition.HashNotExists(...));
// ^^^ This is inserted between WATCH and MULTI
tran.HashSetAsync(...);
// ^^^ This goes between MULTI and EXEC
bool committed = tran.Execute();
// ^^^ if true: it was applied; if false: it was rolled back

Special cases of the SET command ⌘

  • In some cases, you don't need transactions
  • SETNX key value
    • Sets the key to a given value only if it doesn't exist
    • Alternative form: SET key value NX
    • In C#: include a "when: When.NotExists" argument
  • There is also a SET key value XX: sets only if the key already exists
  • GETSET key value: reads the value and replaces it with the new one

Lua scripts ⌘

  • An alternative to transactions
  • In transactions, you are unable to make decisions (or use results from GET) between MULTI and EXEC
    • Not a problem by itself, but with StackExchange.Redis you are also not able to make decisions between WATCH and MULTI!
  • SCRIPT LOAD script
    • persists until server restart, or SCRIPT FLUSH
  • EVAL script numkeys key1 key2 ... arg1 arg2 ...
  • EVALSHA sha1 numkeys key1 key2 ... arg1 arg2 ...
    • fails if the script does not exist
  • Lua tutorial
  • Lua string library tutorial

Lua scripts, continued ⌘

  • Sandboxed
  • Run atomically
    • Until natural termination or SCRIPT KILL or SHUTDOWN NOSAVE
    • SCRIPT KILL works only if there were no writes by the script
  • Can call commands: redis.call('GET', 'foo');
  • Can access keys and arguments as KEYS[] and ARGS[]
    • What's a key and what's an argument is important in the context of Redis Cluster
    • Please don't access other keys
      • It works for a standalone Redis, but will be a problem when you need to migrate to Cluster

Performance issues ⌘

  • "Redis is sometimes slow to respond"
    • Ask devs about pipelining
    • Check for slow commands
    • Call LATENCY DOCTOR
  • "Redis is fat"
    • Tune it, or change data structures
      • E.g. hashes are more memory-efficient than collections of "unrelated" strings
    • Or maybe there are some keys that you forget to delete?
  • "Network is too slow"
    • Often seen on WAN links
    • Use VPN with compression
  • "Cannot assign requested address"
    • Reuse connections

Benchmarking redis ⌘

  • There is an official benchmark tool: redis-benchmark
  • Try simulating the real workload:
    • type of commands
    • number of clients
    • client IP address or the usage of UNIX socket
    • pipeline depth
  • Can also repeat a custom command N times with variations

Monitoring Redis ⌘

  • INFO
    • Look at memory statistics
    • mem_fragmentation_ratio should be between 1 and 1.5 (except on empty server)
  • Latency problems?
    • 200 µs is just expected on a gigabit LAN
    • redis-cli --latency
    • redis-cli --latency-dist
    • Maybe it's not Redis? redis-cli --intrinsic-latency 100
      • Expect up to 75 µs spikes on commodity hardware
      • Does your KVM run with -realtime mlock=on?
        • 3-4 ms spikes happen on AliYun

What do other clients do? ⌘

  • redis-cli monitor
OK
1477053420.062978 [0 127.0.0.1:57626] "COMMAND"
1477053420.066968 [0 127.0.0.1:57626] "ping"
1477053442.487071 [0 127.0.0.1:57632] "COMMAND"
1477053442.490143 [0 127.0.0.1:57632] "sadd" "foo" "bar"

Monitoring slow commands ⌘

  • SLOWLOG GET 10
    • Prints last 10 slow commands
    • ID, timestamp, time in µs, the command itself
  • In redis.conf: slowlog-log-slower-than 100
    • The unit is µs
  • Change threshold at runtime: CONFIG SET slowlog-log-slower-than 50

Avoiding network latency ⌘

  • In the previous "Counter" example, we waited for the command to finish
    • The execution thread was blocked
  • Sometimes we don't need the result
    • CLIENT REPLY SKIP
  • Sometimes it is possible to do something else while waiting for the result
  • A useful trick is to send another command without waiting for the first one to finish
    • It's a good idea to write the two commands to the network in one syscall
    • This technique is called "Pipelining"
    • This is not related to transactions

Special cases ⌘

  • Modern versions of Redis support getting/setting multiple keys ar once
  • MSET, HMSET
  • Support is available in StachExchange.Redis:
    • public RedisValue[] StringGet(RedisKey[] keys);
    • public RedisValue[] HashGet(RedisKey key, RedisValue[] hashFields);
    • public bool StringSet(KeyValuePair<RedisKey, RedisValue>[] values);
    • public void HashSet(RedisKey key, HashEntry[] hashFields);

Avoiding network latency (C#) ⌘

  • If you don't need the result:
    • Specify "flags: CommandFlags.FireAndForget"
      • But it doesn't use CLIENT REPLY SKIP
  • If you want to do something else while Redis thinks about your command:
    • Use Async variants of methods
      • They return a Task, so: using System.Threading.Tasks
      • You can Wait() for the task later and then access its Result
        • Convenient wrapper: var result = db.Wait (theTask);
  • If you want to use pipelining, call db.CreateBatch ()
    • Batches only support async methods
    • After calling some async methods, call batch.Execute() and wait for all tasks

Avoiding network latency (Python) ⌘

  • Full async style programming is not possible with redis-python
  • Pipelining is still possible
pipe = conn.pipeline(transaction=False)
pipe.set('foo', 'bar')
pipe.get('bing')
pipe.execute()
# returns a list with queued command results: [True, 'baz']

Is pipelining effective? ⌘

  • How does a sysadmin know whether developers are using pipelining?
    • Read their source code
    • Run Wireshark on the client side
    • Look for extra commands being sent between a command and its reply
    • Look for multiple Redis commands in one TCP packet

Pipelining in reality (C#) ⌘

  • StackExchange.Redis uses a single Redis connection
    • Multiplexing, not pooling!
  • Uses a separate thread to communicate with Redis and parse replies
  • Uses a buffer to accept commands from other threads, interleaves them as needed
  • So, some form of pipelining can happen without you writing a single line of code for that!

Commands to avoid ⌘

  • KEYS pattern
    • Scans all keys, returns those that match the pattern
  • SMEMBERS bigset
  • Problem: scanning of a big dataset
    • Redis is single-threaded, so cannot do other work during this

What to do instead ⌘

  • SCAN, SSCAN, HSCAN, ZSCAN
  • All of them accept and return cursors
  • The initial value of the cursor should be 0
  • They all return a new value of the cursor and some (or maybe no) results
  • If the new cursor is 0, the scan is complete
  • Guarantees:
    • Will return all elements that lived through the whole scan, at least once
    • Will not mention any elements that were never there during the scan
  • May return some elements twice
  • May return empty slices
  • C# wraps that: public IEnumerable<HashEntry> HashScan(RedisKey key);

How Redis stores data ⌘

  • It chooses representations based on data sizes
  • Has efficient storage for:
    • Strings that represent integers
    • Small HyperLogLogs
    • Short lists and sets
    • Sets consisting of integers
    • Compressible data inside lists and sets
  • Debug: OBJECT ENCODING key

Memory optimization ⌘

  • Migrate to Redis 3.2 or later
  • Use efficient data types when there is such alternative
    • E.g. hashes vs "just collections of strings"
    • E.g. bitfields
  • Use short field names in hashes
    • Or even migrate to lists + convention regarding what is kept at what index
  • Tune thresholds that Redis uses to switch representations

Maybe it's a client problem? ⌘

  • Sometimes clients store keys and forget about them
  • OBJECT IDLETIME key

Memory buffer safeguards ⌘

  • Problem: client pipelined too much
    • Replies accumulate in a buffer
  • Problem: fast client, slow slave
    • Replicated commands accumulate in a buffer
  • Problem: slow Pub/Sub subscriber
    • Messages accumulate in a buffer
  • Solution: client-output-buffer-limit
    • Separate limits for normal clients, slaves and Pub/Sub

Other safeguards ⌘

  • Hard limit on memory usage: maxmemory
    • Comes with a policy what to do (evict keys, which ones, or give errors)
  • Limit on client idle time: timeout

Persistence ⌘

  • Two ways: RDB and AOF
    • You can enable both
  • RDB persistence is enabled by the "save" directive
    • Redis forks and saves its contents to a file
      • Uses atomic replacement so you can copy /var/lib/redis/dump.rdb, and you have a good backup!
    • save 300 10 means save RDB to disk if there are 10 changes in 300 seconds
  • AOF stands for Append-Only File
    • Essentially a log of all commands
    • Sometimes rewritten to save space
    • Enabled with "appendonly yes"
      • You also have to specify appendfsync always|everysec|no

Forcing Redis to persist ⌘

  • SAVE
  • BGSAVE
  • BGREWRITEAOF

Restoring a backup ⌘

  • If you enabled only RDB:
    • Stop redis
    • Copy dump.rdb to /var/lib/redis
    • Start redis
  • If you also enabled AOF:
    • Stop Redis
    • Delete AOF and temporarily disable it in the config
    • Copy dump.rdb to /var/lib/redis
    • Start Redis
    • CONFIG SET appendonly yes # this will block for a while

Inspecting RDB files ⌘

  • redis RDB tools (a third-party project)
    • Export to JSON
    • Compare dumps
    • Convert to Redis protocol
    • Estimate memory usage

More than one Redis: a big picture ⌘

  • Just a bunch of Redis nodes
    • E.g. one Redis for cache, one for sessions, etc.
    • Distributed locking (Redlock)
  • Partitioning/sharding
    • Always done based on the key
    • Old solution: twemproxy
    • Now supported in Redis Cluster
  • Replication
    • No special tools needed
    • Master-master replication is not available
  • Failover
    • Redis Sentinel
    • Now also Redis Cluster

Distributed locking (or rather, leases) ⌘

  • Objectives:
    • Make sure that two application server nodes don't concurrently access some shared resource
    • Do that using some Redis servers
      • With one Redis, that's easy: SETNX
    • Tolerate failure of a minor fraction of Redis servers
    • Tolerate network partitions
    • Tolerate crashing clients (i.e. auto-release the lock after a predefined timeout)
  • Answer: Redlock
    • Note: there were major criticisms
    • Libraries are available for C# with StackExchange.Redis

Redis Replication ⌘

  • Master/slave
  • Asynchronous
  • Disk-based and diskless modes
  • Effectively, the slave asks master to dump RDB, restores it, listens to new commands
    • While RDB is being dumped and downloaded, commands are buffered
    • They consume memory
    • Safeguard: CONFIG SET client-output-buffer-limit slave 256mb 64mb 60
      • Disconnect the slave if we buffered 256 MB, or coundn't go below 64 MB for 60 sec

Redis Replication setup ⌘

  • In config file: slaveof 1.2.3.4 6379
  • Dynamically: SLAVEOF 1.2.3.4 6379
  • Promote slave to master: SLAVEOF NO ONE
  • Other relevant configuration parameters:
    • slave-serve-stale-data yes
    • masterauth <password>
    • repl-backlog-size 1mb # on master
      • Quick replication after quick restart

Redis Sentinel ⌘

  • High availability solution
  • Monitors Redis servers
  • Reconfigures master-slave chain if master fails
  • Also functions as a way to locate masters and slaves
    • SENTINEL get-master-addr-by-name mymaster
    • SENTINEL master mymaster
    • SENTINEL slaves mymaster

Redis Sentinel in practice ⌘

  • You will need an odd number of sentinels
    • Place them on your application servers
  • Also some Redis servers
  • Configure the master-slave replication manually first
  • Place the master like this in redis-sentinel.conf:
    • sentinel monitor mymaster 1.2.3.4 6379 2
    • 2 = the number of sentinels that must agree that the master is down in order to fail it over
    • It will discover slaves by asking the master
    • It will announce itself and autodiscover other sentinels through PUB/SUB
  • Manual failover: redis-cli -p 26379 SENTINEL failover mymaster

Redis Sentinel bugs and limitations ⌘

  • Bug in postinst script in Debian backports: it hangs
    • Systemd expects Redis Sentinel to daemonize and write the PID file
    • Redis Sentinel is not configured to do so
      • daemonize yes
      • pidfile /var/run/redis/redis-sentinel.pid
      • logfile /var/log/redis/redis-sentinel.log
  • Unusable from C# (with StackExchange.Redis) anyway
    • Some code exists in tests, though
  • Authentication between sentinels is unsupported
    • So disable protected mode

Redis Cluster ⌘

  • Enabled in redis.conf by setting "cluster-enabled yes"
  • Needs at least 3 masters
  • Will shard keys among them
    • There are 16384 slots that can be assigned to masters in arbitrary proportions
  • Sharding is based on key hash
    • Except when there is a {...} in the key name
    • The first {...} is non-empty: hash it
    • The first {} is empty: hash the entire key
  • Will also manage slaves

Redis Cluster networking requirements ⌘

  • As of Redis 3.2.x, not compatible with NAT
    • The first stable version of Redis to support that will be 4.0, released as RC1 15th of October 2016
    • Fixing this required updating the protocol version
  • Needs an extra port (base_port + 10000, not configurable) for cluster bus

Redic Cluster: client side ⌘

  • Clients must handle the ASK and MOVED errors (they happen if the server is not responsible for the key slot)
    • It contains the correct IP and port to connect to
  • There is also a CLUSTER SLOTS command
    • Returns the full map of the cluster
    • The client
  • Clients must not issue cross-slot commands

Redis Cluster: operations ⌘

  • There is a CLUSTER command
    • Tedious to use - e.g. can move only one slot at a time
    • Redis comes with a redis-trib script
      • Needs a "redis" Ruby gem
      • "gem install redis", if your distribution hasn't packaged it
Retrieved from ‘https://training-course-material.com/index.php?title=Redis_for_Administrators_and_Developers&oldid=63492’