For this experiment we will use the docker-redis-cluster project. Our aim is, roughly, to translate the test.sh test they provide to RSpec. In the process we will also use the redis-rb-cluster project and example.rb.
redis-cluster setup was very easy (well done) and worked out of the box within a couple of seconds:
$ git clone https://github.com/AliyunContainerService/redis-cluster.git
Cloning into 'redis-cluster'...
remote: Counting objects: 67, done.
remote: Total 67 (delta 0), reused 0 (delta 0), pack-reused 67
Unpacking objects: 100% (67/67), done.
$ cd redis-cluster/
macbook:redis-cluster lookfwd$ ls
LICENSE docker-compose.yml test.sh
README.md sentinel
$ docker-compose up -d
Pulling master (redis:3)...
3: Pulling from library/redis
5233d9aed181: Pull complete
ca1b33d3f114: Pull complete
920cdc17d3c2: Pull complete
039bc0a8c4af: Pull complete
...
Creating rediscluster_master_1 ...
Creating rediscluster_master_1 ... done
Creating rediscluster_slave_1 ...
Creating rediscluster_slave_1 ... done
Creating rediscluster_sentinel_1 ...
Creating rediscluster_sentinel_1 ... done
$ docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------
rediscluster_master_1 docker-entrypoint.sh redis ... Up 6379/tcp
rediscluster_sentinel_1 sentinel-entrypoint.sh Up 26379/tcp, 6379/tcp
rediscluster_slave_1 docker-entrypoint.sh redis ... Up 6379/tcp
The test.sh test however isn't self-checking and seemed to be out-of-date. It didn't work as expected:
$ ./test.sh
Redis master: 172.17.0.3
Redis Slave: 172.17.0.4
------------------------------------------------
Initial status of sentinel
------------------------------------------------
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.17.0.3:6379,slaves=1,sentinels=1
Current master is
172.17.0.3
6379
------------------------------------------------
Stop redis master
rediscluster_master_1
Wait for 10 seconds
Current infomation of sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=sdown,address=172.17.0.3:6379,slaves=1,sentinels=1
Current master is
172.17.0.3
6379
------------------------------------------------
Restart Redis master
rediscluster_master_1
Current infomation of sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=172.17.0.3:6379,slaves=1,sentinels=1
Current master is
172.17.0.3
6379
Let's try try to use redis-rb-cluster
$ git clone https://github.com/antirez/redis-rb-cluster.git
Cloning into 'redis-rb-cluster'...
remote: Counting objects: 153, done.
remote: Total 153 (delta 0), reused 0 (delta 0), pack-reused 153
Receiving objects: 100% (153/153), 27.65 KiB | 2.30 MiB/s, done.
Resolving deltas: 100% (73/73), done.
$ cd redis-rb-cluster/
$ gem install redis
Fetching: redis-4.0.0.gem (100%)
Successfully installed redis-4.0.0
Parsing documentation for redis-4.0.0
Installing ri documentation for redis-4.0.0
Done installing documentation for redis after 0 seconds
1 gem installed
$ ruby example.rb
error Can't reach a single startup node. Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED)
error Can't reach a single startup node. Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED)
Now this is interesting. Our example system is nicely setup by docker-compose with local ports etc:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a9a86ae99256 rediscluster_sentinel "sentinel-entrypoi..." 7 minutes ago Up 7 minutes 6379/tcp, 26379/tcp rediscluster_sentinel_1
9d001cf1e9ab redis:3 "docker-entrypoint..." 7 minutes ago Up 7 minutes 6379/tcp rediscluster_slave_1
f7367841075d redis:3 "docker-entrypoint..." 7 minutes ago Up 7 minutes 6379/tcp rediscluster_master_1
We shut-down the docker setup with docker-compose down and then we add the networks segment and modernize a bit docker-compose.yml:
version: '3'
services:
master:
image: redis:3
slave:
image: redis:3
command: redis-server --slaveof redis-master 6379
links:
- master:redis-master
sentinel:
build: sentinel
environment:
- SENTINEL_DOWN_AFTER=5000
- SENTINEL_FAILOVER=5000
links:
- master:redis-master
- slave
networks:
default:
external:
name: test_network
We also create the network with: docker network create test_network. We bring the system back up with docker-compose up -d. As soon as the system is up, we can connect to it:
$ docker run --rm -i -t --network=test_network -v $(pwd):/src ubuntu /bin/bash
This image is a bit too empty so we have to install some software to it, in order to confirm with telnet that we can connect to Redis nodes and that replication works fine:
$ apt-get update
$ apt-get install -y telnet
$ telnet master 6379
Trying 172.18.0.3...
Connected to master.
Escape character is '^]'.
SET hello 123
+OK
GET hello
$3
123
$ telnet slave 6379
Trying 172.18.0.4...
Connected to slave.
Escape character is '^]'.
GET hello
$3
123
This is great news. We're in the container and we can contact the two redis nodes with telnet. Let's go to the /src directory where our example.rb should now be. Once again we're on an empty environment so we will have to install ruby and redis-rb to run this.
$ apt-get install -y ruby
$ gem install redis
we edit example.rb script to have the following:
startup_nodes = [
{:host => "master", :port => 6379},
{:host => "slave", :port => 6379}
]
Now we should be able to successfully run ruby example.rb.
$ ruby ./example.rb
error READONLY You can't write against a read only slave.
2
error READONLY You can't write against a read only slave.
error READONLY You can't write against a read only slave.
5
error READONLY You can't write against a read only slave.
6
error READONLY You can't write against a read only slave.
8
error READONLY You can't write against a read only slave.
9
error READONLY You can't write against a read only slave.
As you can see, the write fails quite a few times and this is because RedisCluster picks a random connection for a write. When we pause the master, we get just that error message without any successes. It's not the expected behavior but it's a behavior. Let's move on.
We install rspec and initialize a project:
$ gem install rspec -v 3.6.0
$ rspec --init
$ rspec
No examples found.
Finished in 0.00063 seconds (files took 0.09055 seconds to load)
0 examples, 0 failures
Let's try to add a basic test in spec/redis_spec.rb:
require './cluster'
startup_nodes = [
{:host => "dockerrediscluster_redis_cluster_1", :port => 7000},
{:host => "dockerrediscluster_redis_cluster_1", :port => 7001}
]
describe "Redis cluster session" do
subject(:rc) { RedisCluster.new(startup_nodes,32,:timeout => 0.1) }
# In case it exists, delete it
before { rc.del("my_key") }
it "sets and confirms" do
expect(rc.set("my_key", 12)).to eq "OK"
expect(rc.get("my_key")).to eq "12"
end
it "tries to get an unset key" do
expect(rc.get("my_key")).to be_nil
end
end
When we run it, we confirm it is all green:
$ rspec
..
Finished in 0.10678 seconds (files took 0.14111 seconds to load)
2 examples, 0 failures
Now that we have a basic test running, we are ready to put docker-compose into the loop. We exit our test container and restart it while adding the -v /var/run/docker.sock:/var/run/docker.sock argument. This will allow us to remote control docker from from RSpec!
Unfortunately we will have to re-install Ruby, ruby-redis and rspec. This time we will also install docker:
apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update
apt-get install -y docker-ce
curl -o /usr/local/bin/docker-compose -L "https://github.com/docker/compose/releases/download/1.15.0/docker-compose-$(uname -s)-$(uname -m)"
chmod +x /usr/local/bin/docker-compose
Try to run a docker ps from within the container to confirm it works. Now you can enrich the RSpec specification with docker commands. Here's the final test:
require './cluster'
host = "dockerrediscluster_redis_cluster_1"
startup_nodes = [
{:host => host, :port => 7000},
{:host => host, :port => 7001}
]
describe "Redis cluster session" do
subject(:rc) { RedisCluster.new(startup_nodes,32,:timeout => 0.1) }
it "sets and confirms" do
expect(rc.set("my_key", 12)).to eq "OK"
expect(rc.get("my_key")).to eq "12"
end
it "tries to get an unset key" do
expect(rc.get("my_key")).to be_nil
end
before do
system "docker-compose -f docker-redis-cluster/docker-compose.yml up > /dev/null&"
loop do
begin
rc = RedisCluster.new(startup_nodes,32,:timeout => 0.1)
rc.del("my_key")
break
rescue RuntimeError
puts "waiting for system to start"
sleep 2
end
end
end
after do
system "docker-compose -f docker-redis-cluster/docker-compose.yml stop"
end
end
The output is similar to this:
$ rspec
waiting for system to start
Creating dockerrediscluster_redis_cluster_1 ...
Creating dockerrediscluster_redis_cluster_1
Creating dockerrediscluster_redis_cluster_1 ... done
waiting for system to start
waiting for system to start
waiting for system to start
waiting for system to start
Stopping dockerrediscluster_redis_cluster_1 ... done
.Starting dockerrediscluster_redis_cluster_1 ...
Starting dockerrediscluster_redis_cluster_1
Starting dockerrediscluster_redis_cluster_1 ... done
waiting for system to start
waiting for system to start
waiting for system to start
waiting for system to start
Stopping dockerrediscluster_redis_cluster_1 ... done
.
Finished in 32.29 seconds (files took 0.14518 seconds to load)
2 examples, 0 failures
This comment has been removed by a blog administrator.
ReplyDelete