NabeWise uses Redis to store much of our neighborhood data. Capturing the identity of a neighborhood across 40+ (constantly changing) attributes does not fit well in the relational model and Redis is the mojo of features like our NabeFinder. Many of our pages, including our home page, never touch MySQL or Memcached or neighborhood results.
Our primary difficulty with working with Redis has been our need for essentially atomic operations that are composed of a few commands. When a number of changes are streaming in that all affect different facets of our encoded neighborhood records, we would prefer to not clobber the changes through interleaved gets and sets. These cases have been minimized (and we are constantly working to reduce them further). The obvious solution is to serialize access in these cases.
The standard Rails app runs with a number of worker processes, random rake tasks on cron or flights of manual folly, and probably an background task queue with one or more workers. This sucks when you’re trying to limit access to one process. After briefly looking at solutions to forcibly neuter concurrency, we decide to embrace and control it. Like Bond villains. With Erlang.
Thanks to GitHub and BERTRPC, exposing Erlang services to Rails is pretty trivial. Besides being good at dealing with and managing concurrency and distributed systems, Erlang makes expressing our data constraints and data access very clean and strict. The goal is still to keep certain write operations from interfering with others, but Ernie, our BERTRPC server, spawns a new process (in the Erlang sense) for every call. This is mostly what we want as the majority of any call can be completed as concurrently as we can go.
Two core features of Erlang are its OTP (Open Telecom Platform) and distributed Erlang. These allow the construction of networks of nodes and servers that can be used very flexibly. We constructed an OTP backend with gen_servers that would handle our precious order sensitive writes. Currently this is separated into single servers for each overall type of write (totals vs. ranks in our system), but should our load increase, we can easily separate at higher degrees of specificity to up our parallelism (and also allow for many possibilities in Redis sharding).
The last piece is hooking up Ernie to our distributed node. Distributed nodes are identified by names like fred@localhost or nabewise_attributes@127.0.0.1. Our first modification was to allow Ernie to be named by passing its executable the -name parameter. Distributed nodes are made aware of each other by pinging or otherwise accessing other nodes. Our second modification added the ability to specify an init function to be called on server start. This also allows us to share a redis connection between many short-lived processes. Our modified ernie is available at http://github.com/schleyfox/ernie .
This is our ernie.cfg
[{module, attributes},
{type, native},
{init, {attributes, init, []}},
{codepaths, ["ebin/"]}].The relevant bit of our attributes.erl is basically
init() ->
start_redis(),
net_adm:ping('nabewise_attributes@127.0.0.1').Our actual setup differs only in that we add in monitoring for the remote node and attempt to reconnect on disconnect.
Summary
With Erlang, adding services distributed across different access needs, computers, or code divisions to your Rails app is simple. In addition to a reliable environment with powerful control of concurrency, you also get features like painlessly easy instant, asynchronous processing (cast in BERTRPC). In practice at NabeWise, this has significantly increased the performance and flexibility of our features that touch our neighborhood attribute data. Future posts will go into more detail on how we use this technology for our site.
