selectel.meetup: A High-Load Event

This past December 17th in St. Petersburg, Selectel held the first ever selectel.meetup—a conference dedicated to high-load systems. Developers came together in the new conference hall at Selectel’s Tsvetochnaya 2 data center to share their experiences. The event was sponsored by ITSumma, which provides technical support for websites and remote server administration.

During the event, a variety of presentations were given by our participants:

• Vyacheslav Axmetov (Selectel) — The evolution of memcache in high-load projects
• Mexti Musaev (Selectel) — Code metrics: Why we need them
• Evgenij Potapov (ITSumma) — Preliminary architecture optimization
• Sergej Sporyshev (ITSumma) — How Project1917 was created and how we support 1000 RPS on one server
• Aleks fon Rozen (220 Volts) — Highload on the fly
• Viktor Smirnov (220 Volts) — Twenty seven Fridays a week
• Yan Ashenkampf (KupiVip.ru) — How we rewrote KupviVIP
• Ilya Pyatin (CarPrice) — How to ensure two years of a project functionality’s unimpeded growth
• Sergej Averin (Acronis) — Common errors in database usage
• Artyom Gavrichenkov (QRator Labs) — Assessing the risk of DDoS attacks

While video of the conference is being prepared, below are summaries of the two presentations given by Selectel staff.

Mexti Musaev — Code Metrics: Why we need them

Code metrics are used to obtain numerical values for certain software properties and characteristics. The first code metrics appeared at the start of the 70s and helped determine the quality of software and expose errors or vulnerabilities in the code. Consequently, when theoretical and practical information technologies began to develop faster and in different fields, new metrics were created to meet other needs. An example of this would be the use of metrics to determine the obfuscation for a program.

The most popular code metrics now are used to determine a code’s quality. They’re considered when designing projects and determining the size of the code, software complexity, and estimate the amount of work needed to develop it. The following metrics are used today: code coverage, cohesion, cyclomatic complexity (or McCabe’s complexity), Halstead complexity, Maintainability index, and raw metrics (or lines of code).

Code coverage metrics are the most popular as they’re required for writing software tests. They show what percentage of a code is executed while running a particular test suite. The higher the coverage metric, the more code is run and, consequently, the less likely the software is to contain undetected errors.

Cohesion shows the extent to which elements of a module are interrelated—the higher the cohesion, the stronger the system interactions.

Cyclomatic complexity (or McCabe’s complexity) was developed by Thomas McCabe in 1976 and shows a program’s structural complexity, which can be determined from a control flow graph. An interesting feature of this metric is that it is calculated from the control flow graph of a program. More specifically, it’s a quantitative means of calculating linearly independent paths in the source code. This metric can also be applied separately for functions, modules, methods, and software classes.

The Halstead metric was published in 1977 and is used for identifying measurable software properties and the relations between them.

The maintainability index helps determine software support solutions. It is calculated from raw metrics, cyclomatic complexity, and Halstead complexity. It is popular in integrated software development environments (like MS Visual Studio).

Raw metrics show the amount of physical and logical code present.

Vyacheslav Axmetov — The evolution of memcache and high-load projects

How is memcache usually implemented and developed in an architecture? Usually, in the interest of saving time or resources, a database and application are installed on one server. In addition the economic benefits, sharing a server yields good performance as a result of the absence of network lag.

As the load grows, the application and database usually spread to different servers. Since this is a lecture on memcache, we’ll add a caching level at the start (which lets us lower loads on the database); however, the database will still grow over time, and we’ll still have to move it to a separate server (or move the application with the cache).

If application traffic continues to grow, and the resources on one server are no longer enough, we’ll add a new server and deploy a copy of the application along with the cache on it. The number of requests to the database will increase two-fold since caches don’t recognize one another and fill up independently. We have to cut off any data inconsistency issues in the cache.

To solve these problems, we move the cache to a separate server. Everything gets more or less resolved, but complications pop up when we can’t vertically scale the cache or just aren’t ready for an SPOF in the architecture.

Let’s look at some scaling options and their issues. To start with, we need to instruct all instances of the application to work with only one list of cache servers. All modern memcache clients have basic sharding key algorithms: Modulo and Ketama (Consistent Hashing). These algorithms have similar principles for distributing keys between servers, which is depicted below:

servers = [’server1:11211′, ’server2:11211′, ’server3:11211′]
server_for_key(key) = servers[hash(key) % servers.length]

Problems:

• adding or deleting servers results in errors/losses
• changes have to occur simultaneously on all application instances (otherwise data will be inconsistent)
• any node that temporarily goes offline will end up with irrelevant data

What can we do about this?
The vbucket algorithm was added to the beta version of memcached 1.6. The main difference between vbucket and Ketama is that keys are mapped to a fixed number of logical “virtual buckets” and not to servers. These buckets are, in turn, already mapped to servers. The number of vbuckets should remain constant (this can only be changed when data is fully cleared), but the number of servers can change.

What vbucket gives us:

• servers service only concrete vbuckets
• as a result, there’s no network overhead
• scaling up and down
• guaranteed consistency
• possibility to transfer data between servers
• replication

Sounds good, but how do we implement this?
To take advantage all of the features vbuckets have to offer, your application has be able to use them. Unfortunately, almost no popular library today supports vbucket. Here to the rescue is moxi proxy server, which supports all of vbuckets’ features.

Moxi
There are a lot of proxy servers for memcache, but moxi is best one that supports vbucket.

Multithreaded C
Libmemcached
Memcached
Libevent
Protocol conversion

The most popular products that support vbucket are zBase and couchbase.

Links:
http://dustin.sallings.org/2010/06/29/memcached-vbuckets.html
https://github.com/zbase/
https://github.com/couchbase
https://github.com/memcached/memcached/tree/engine-pu

This was an exciting new experience for Selectel: it was the first time we invited guests to our office! We’d like to thank everyone who came out. We hope you found selectel.meetup to be both interesting and comfortable.

Below are some photos from the first ever selectel.meetup.