YAWNDB – Time Series Databases

For many of our services, we have to provide clients with various statistics. Clients who rent dedicated servers need information on the traffic they’ve used, VPC users need stats on hardware and network resources, and Cloud Storage users need information on file transfers.

The simplest and most direct way to present statistics is in charts. There are many programs specifically designed for analyzing statistical data with subsequent visuals. We’ve looked for an appropriate tool with performance as our main criterion. As a result… well, let’s go through this step by step. We’ll start some theory.

A Bit of Theory

Any chart on network activity reflects the changes of several parameters over a defined period of time (a month, week, day, etc.). To compile a chart, we must process statistical materials which represent the correlation between time and value for a specified interval. These materials are called time series.

There are quite a few tools for analyzing a time series, most of which use round robin databases. A round robin database is one in which the amount of data saved doesn’t change because it is a constant: database cells activate in cycles.

In a round robin database, either one or several data sets are saved to round robin archives (RRA). In terms of their structure, round robin tables are the same as arrays, where the address of the last element matches the address of the first element. The location of the last updated element is saved as an indicator. Archives are linked to each other in such a way that the last archive saves information from its predecessor: one archive saves data with a short interval between writes, another saves consolidated data from the former after a given number of intervals, the third does the same even less frequently, etc.

This is done using consolidation functions that are integrated into the database and automatically applied when information is updated. A consolidation function implies that a minimum, maximum, mean, and general value is received for a specified period of time. Data is consolidated in a round robin database during the writing, not reading, process (this ensures quick execution).

RRDTool

The most well-known and widespread tool for analyzing time series and subsequent visuals is of course RRDTool.

We tried RRDTool for ourselves; however, for a myriad of reasons, we just weren’t happy with it. Firstly, it handles heavy loads very poorly.

When the number of files we wrote data to exceeded 1000, we started to see problems: for example, writing the same data started to take a lot longer. Sometimes data simply wouldn’t write, even though there weren’t any errors or interruptions.

This was enough to conclude that RRDTool just isn’t for us: with the number of virtual machines we have in the cloud, tens of thousands of operations need to be performed every second.

When working with a large amount of data in RRDTool, the number of write operations can be reduced using the RRDcacheD daemon, which caches data and then writes it to a database after a specific amount has been accumulated. Our experience with this, however, has shown that RRDcacheD is not the right tool for our work.

When collecting data, it doesn’t let the data be read from the cache, only be written to a disk. If data needs to be processed, then everything needs to be written to the disk and then read from the disk. The more data you have, the worse the cache works: the hard disk gets overloaded, which creates additional work for the processor….

Another point about RRDcacheD is that it writes data to the disk at the most unexpected and inconvenient times.

Another argument against RRDTool is that you cannot change the configuration of the round robin database. Of course this can be done by exporting data, creating a file with new parameters, and subsequently importing the old data to it, but that’s really just too tedious and inconvenient.

Having encountered all of these difficulties, we decided not to use RRDTool. We did try other data processing and visualization tools, like graphite, but they didn’t suit us because of their low performance.

The more we got acquainted with the tools available, the more obvious it became that none of them fit our requirements. Thus we got the idea to develop our own solution: one that we could be happy with. The fundamental requirements were, firstly, that it had to be flexible, configurable, and able to adapt to our services; secondly, it had to be high performance. And so, the in-memory round robin database YAWNDB was born.

YAWNDB: Overview

YAWNDB (don’t let the name fool you; YAWN is an acronym which means Yet Another iNvented Wheel) is an in-memory database; all data is saved to the RAM and is periodically written to the disk. It’s written in Erlang, which is built on the light-weight process model; this ensures that a large set of data can be processed quickly and without consuming a lot of system resources.

Data that enters YAWNDB is distributed to archives (in YAWNDB, these are called buckets) according to several rules.

Here, a rules is a set of properties for a particular statistic (data size, collection period, etc.).

All of this data is presented as a triplet, consisting of time, value, and keys (a key in YAWNDB is also called a path). The path is a sequence of lowercase letters and numbers which define where a triplet should be saved. In this sequence, the first element is the most important: the prefix. Rules include a prefix field and number, which defines how data is saved for a specific prefix. Several rules can be made for the same prefix, and in that case, a triplet will be written to a database according to each of these rules.

This means a specific “time — value” pair enters N buckets, where N is the number of rules that correspond to the data pair’s path prefix.

This approach is beneficial for the following reasons:

• irrelevant data is deleted without excessively wasting resources
• the amount of memory consumed is fixed for a defined number of keys
• access time is fixed for random writes

Architecture

YAWNDB is built on a round robin algorithm written in C and found in the Ecirca library.

Other modules, written in Erlang, interact with it using NIFs (Native Implemented Functions). Data is saved to a disk using the Erlang application Bitcask.

REST API is built on the Cowboy webserver.
Data is written via a socket and read using the REST API.

Installation

The LibYAML parser must be installed for YAWNDB to work. Then, the repository has to be cloned:

$ git clone git@github.com:selectel/yawndb.git

and the following commands must be executed:

$ cd yawndb
$ make all

YAWNDB is launched with the command:

$ ./start.sh

Prior to launch, an example of the configuration file must be copied to the same place as the original:

$ cp priv/yawndb.yml.example priv/yawndb.yml

or a corresponding symlink can be created.

Configuration

All YAWNDB settings are saved in the yawndb.yml configuration file.

As an example, we’ll look at the configuration file that processes statistics in Cloud Storage:

rules:
    # User statistics
    # detailed statistics per minute (for 24 hours)
    - name: per_min
      prefix: clientstats
      type: sum
      timeframe: 60
      limit: 1440
      split, backward
      value_size, large
      additional_values: []

    # statistics per hour (for the past month)
    - name: per_hour
      prefix: clientstats
      type: sum
      timeframe: 3600
      limit: 720
      split: backward
      value_size: large
      additional_values: []

    # statistics per day (saved for the past two years)
    - name: per_day
      prefix: clientstats
      type: sum
      timeframe: 86400
      limit: 730
      split: backward
      value_size: large
      additional_values: []

Practical Examples

Forming Packets for Writing Data (Python)

def encode_yawndb_packet(is_special, path, time, value):
    """
Form data packets to be sent to yawndb.    

    :param bool is_special: special value?  used for additional_values
    :param str path: metric identifier
    :param int value: metric value
    """
    is_special_int = 1 if is_special else 0
    pck_tail = struct.pack(
">BBQQ",
YAWNDB_PROTOCOL_VERSION, is_special_int, time, value
) + path
    pck_head = struct.pack(">H", len(pck_tail))
    return pck_head + pck_tail

Forming packets for writing data in C

// we set the protocol version
#define YAWNDB_PROTOCOL_VERSION 3
// we describe the packet structure 
struct yawndb_packet_struct {
    uint16_t    length;
    uint8_t     version;
    int8_t      isSpecial;
    uint64_t    timestamp;
    uint64_t    value;
    char        path[];
};
// we form the packet
yawndb_packet_struct *encode_yawndb_packet(int8_t isSpecial,
        uint64_t timestamp, uint64_t value, const char * path) {
    yawndb_packet_struct *packet;
    uint16_t length;

    lenght = sizeof(uint8_t) + sizeof(int8_t) + sizeof(uint64_t) + sizeof(uint64_t) + strlen(path);
    packet = malloc(length + sizeof(uint16_t));
    packet->length = htobe16(length);
    packet->version = YAWNDB_PROTOCOL_VERSION;
    packet->isSpecial = isSpecial;
    packet->timestamp = htobe64(timestamp);
    packet->value = htobe64(value);
    strncpy(packet->path, path, strlen(path));

    return packet;
}

Conclusions

We use YAWNDB for both public services and internal projects. The project’s source code can be found on GitHub. We’d be grateful if any of our readers gave it a try. We’d also be happy to hear your comments and suggestions for improving our product.