Virtual Private Cloud: CoreOS and RancherOS

pr-3140-2

We recently updated the version of CoreOS available in our Virtual Private Cloud and also added RancherOS to our list of available images.

These operating systems are useful for managing a large number of applications in containers and for using different container clustering systems, like Kubernetes, Docker Swarm, Apache Mesos, etc.

CoreOS and RancherOS are different from most Linux distributions in that they contain minimal software for launching applications in containers; additionally, these operating systems don’t have a package manager or interpreter for various programming languages (Python, Ruby, etc.).

Below, we’ll look at at the specifics for CoreOS and RancherOS, as well as how to configure the etcd and fleet services. We will also configure a Docker Swarm cluster in RancherOS.

CoreOS

CoreOS was developed specifically for quickly deploying a large number of user applications in isolated containers.

CoreOS contains preinstalled tools for managing multiple nodes and containers in a cluster:

  • etcd — distributed data storage which nodes use to receive information about available services in clusters, configurations, and temporary files;
  • Docker — the main platform for delivering and launching applications in a container;
  • Rocket — an alternative to Docker, developed by the creators of CoreOS;
  • fleet — a distributed initialization system that lets you manage multiple instances of systemd on the independent nodes of a cluster and launch services on the nodes with the smallest loads.

Configuration

The basic way to change system parameters is by describing the necessary configuration in YAML in the cloud-config.yml configuration file. This file is used by Cloud-init to configure the system.

Please note that CoreOS uses its own version of Cloud-init. In this version, configuration files are only somewhat compatible with the more popular Launchpad-cloud-init, which is used by most distributions for configuring systems in various cloud services, including the Virtual Private Cloud.

Let’s look at an example of cloud-config.yml:

#cloud-config
 
# Set hostname
hostname: "node01"
 
# Set ntp servers
write_files:
    - path: /etc/systemd/timesyncd.conf
      content: |
        [Time]
        NTP=0.ru.pool.ntp.org 1.ru.pool.ntp.org
 
coreos:
    units:
    # Configure static network on eth0 interface
    - name: iface-eth0.network
      runtime: true
      content: |
        [Match]
        Name=eth0
 
        [Network]
        DNS=188.93.16.19
        DNS=188.93.17.19
        DNS=109.234.159.91
        Address=10.11.12.13/24
        Gateway=10.11.12.1
 
    # Change standard SSH port
    - name: sshd.socket
      command: restart
      runtime: true
      content: |
        [Socket]
        ListenStream=2345
        FreeBind=true
        Accept=yes

Using the given parameters, Clouid-init will change the hostname and standard port for the SSH service, configure a static network on interface eth0 and add ntp servers for time synchronization.

More detailed information on the cloud-config.yml structure can be found in the official documentation.

When you create a server in the Virtual Private Cloud from a prepared CoreOS image, you don’t have to perform the basic system setup. We’ve already made the necessary changes to the image, so your server, including the network interface, will be set up automatically.

SSH access to a server running CoreOS will initially be possible only with the key given when the server is created. This key will be added for the user core.

In addition to its own version of cloud-init, the creators of CoreOS developed the utility Ignition. This tool mimics cloud-init and also lets you perform low-level system configurations, like changing disk partition tables and formatting file systems. This is possible because Ignition initiates during the early stages of the system start up, while initramfs loads.

Ignition configuration files are written in JSON format.

Below is an example of a file that Ignition uses to format the root partition in Btrfs and then configure the system with the same parameters as the Cloud-init example above:

{
    "ignition": { "version": "2.0.0" },
    "storage": {
        "filesystems": [{
            "mount": {
                "device": "/dev/disk/by-label/ROOT",
                "format": "btrfs",
                "create": {
                    "force": true,
                    "options": [ "--label=ROOT" ]
                }
            }
        }],
        "files": [{
            "filesystem": "root",
            "path": "/etc/systemd/timesyncd.conf",
            "mode": 420,
            "contents": { "source": "data:,%5BTime%5D%0ANTP=0.ru.pool.ntp.org%201.ru.pool.ntp.org%0A" }
            },
            {
            "filesystem": "root",
            "path": "/etc/hostname",
            "mode": 420,
            "contents": { "source": "data:,node01" }
            }
        ]
    },
    "networkd": {
        "units": [{
            "name": "iface-eth0.network",
            "contents": "[Match]\nName=eth0\n\n[Network]\nDNS=188.93.16.19\nDNS=188.93.17.19\nDNS=109.234.159.91\nAddress=10.11.12.13/24\nGateway=10.11.12.1"
        }]
    },
    "systemd": {
        "units": [{
            "name": "sshd.socket",
            "command": "restart",
            "runtime": true,
            "contents": "[Socket]\nListenStream=2345\nFreeBind=true\nAccept=yes"
        }]
    }
}

More detailed information on all of Ingition’s abilities can be found in the official documentation.

CoreOS Cluster Configuration

In this example, we’ll first have to create three CoreOS servers with the following parameters in the VPC control panel:

pr-32843225_eng-01

Once our servers have been created, we need discovery-url for the etcd service. To do this, we can use the free public service discovery.etcd.io. We’ll run the following command on the host, node00:

core@node00 ~ $ curl -w "\n" 'https://discovery.etcd.io/new?size=3'
https://discovery.etcd.io/ec42cfef0450bd8a99090ee3d3294493

We add this URL and additional etcd parameters to the file /usr/share/oem/cloud-config.yml on the next server.

coreos:
    etcd2:
        discovery: https://discovery.etcd.io/ec42cfef0450bd8a99090ee3d3294493
        advertise-client-urls: http://192.168.0.10:2379,http://192.168.0.10:4001
        initial-advertise-peer-urls: http://192.168.0.10:2380
        listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
        listen-peer-urls: http://192.168.0.10:2380
 
    units:
        - name: etcd2.service
          command: start
 
        - name: fleet.service
          command: start

The parameters in the etcd2 section are the URL addresses in protocol://address:port format, which are needed for announcing nodes and exchanging data with other cluster members.

The units section lets you manage systemd units in the system; in our case, it launches the units etc2d.service and fleet.service.

For servers node01 and node02, you’ll need to make similar changes to cloud-config.yml: change the IP address from 192.168.0.10 to 192.168.0.11 on node01 and 192.168.0.12 on node02.

After the new settings have been added, we launch Cloud-init on all three servers (or we can just reboot these servers):

core@node00 ~ $ sudo coreos-cloudinit --from-file /usr/share/oem/cloud-config.yml

We also check the status of etcd:

core@node00 ~ $ etcdctl cluster-health
member 3aaf91fd1172594e is healthy: got healthy result from http://192.168.0.11:2379
member 8cf40790248e1dcd is healthy: got healthy result from http://192.168.0.12:2379
member 96b61f40c082cd0b is healthy: got healthy result from http://192.168.0.10:2379
cluster is healthy

We check distributed storage availability, add a new directory (or “key” in etcd terminology) with the values from node00:

core@node00 ~ $ etcdctl set /common/cluster-name selectel-test-cluster
selectel-test-cluster

Then we make sure that our new key is accessible from other nodes in the cluster.

core@node01 ~ $ etcdctl get common/cluster-name
selectel-test-cluster

Excellent! Etcd is working and we can start using fleet.

In our example, we’ll use fleet to launch Nginx in an isolated Docker container. To start with, we create systemd unit /etc/systemd/system/nginx.service:

[Unit]
Description=Nginx Test Service
 
[Service]
EnvironmentFile=/etc/environment
ExecStartPre=/usr/bin/docker pull nginx
ExecStart=/usr/bin/docker run --rm --name nginx -p 80:80 nginx
ExecStop=/usr/bin/docker kill nginx
 
[X-Fleet]
Conflicts=nginx.service

The Conflicts parameter in the X-Fleet section prevents two or more Nginx services from being launched on one node, which lets you “spread” the load over the cluster and increase service availability. Additional X-Fleet parameters, which can be used inside systemd units, are described in the fleet documentation.

The other sections in the file are standard for systemd; you can find some information on them in the instructions from CoreOS.

We use fleetctl to launch a new nginx.service unit:

core@node00 ~ $ fleetctl start /etc/systemd/system/nginx.service
Unit nginx.service inactive
Unit nginx.service launched on 1ad018e0.../192.168.0.11

Afterwards, we can find Nginx instances on any cluster node from the units list.

core@node02 ~ $ fleetctl list-units
UNIT            MACHINE                     ACTIVE  SUB
nginx.service   1ad018e0.../192.168.0.11    active  running
core@node02 ~ $ fleetctl list-unit-files
UNIT            HASH    DSTATE      STATE       TARGET
nginx.service   0c112c1 launched    launched    1ad018e0.../192.168.0.11

We can see that a container with Nignix was launched on the server with IP address 192.168.0.11, which relates to node01.

If we then disable or delete server node01, fleet will automatically move Nginx to another available node.

In this case, etcd announces the member is unavailable when it checks the cluster status.

core@node00 ~ $ etcdctl cluster-health
failed to check the health of member 3aaf91fd1172594e on http://192.168.0.11:2379: Get http://192.168.0.11:2379/health: dial tcp 192.168.0.11:2379: i/o timeout
failed to check the health of member 3aaf91fd1172594e on http://192.168.0.11:4001: Get http://192.168.0.11:4001/health: dial tcp 192.168.0.11:4001: i/o timeout
member 3aaf91fd1172594e is unreachable: [http://192.168.0.11:2379 http://192.168.0.11:4001] are all unreachable
member 8cf40790248e1dcd is healthy: got healthy result from http://192.168.0.12:2379
member 96b61f40c082cd0b is healthy: got healthy result from http://192.168.0.10:2379
cluster is healthy

We can see that Nginx’s status is fine; it was launched on another working node:

core@node00 ~ $ fleetctl list-units
UNIT            MACHINE                     ACTIVE  SUB
nginx.service   4a1ff11c.../192.168.0.10    active  running

This was a basic example of a etcd + Docker + fleet cluster configuration running on CoreOS. In the next section, we’ll look at another distribution: RancherOS. This is based on CoreOS and was also designed for actively using containers.

RancherOS

RancherOS is unique in that it launches not only user applications in containers, but entire system services. Moreover, the RacherOS Docker has a PID = 1, which is why it launches right after the system kernel.

The operating system contains two Docker instances, one of which is System Docker. This is where containers of udev, acpid, syslog, ntp, and other system-critical services are launched. The System Docker instance replaces traditional system instances (systemd, Upstart, SysV) that are present in standard Linux distributions.

The second Docker instance is used for launching user applications and is itself a special container launched in System Docker.

This segregation protects system containers from potentailly hazardous user actions.

pr-32843225_eng-02

Since RancherOS uses the standard Docker environment, we can use the standard Docker commands to manage user and system containers.

# check Docker version
[rancher@rancher-os ~]$ docker --version
Docker version 1.12.1, build 23cf638
# install a new container from Docker Hub
[rancher@rancher-os ~]$ docker run -d nginx
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
…
 
Status: Downloaded newer image for nginx:latest
# view a list of launched containers
[rancher@rancher-os ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS               NAMES
2ec5126e8dd5        nginx               "nginx -g 'daemon off"   About a minute ago   Up About a minute   80/tcp, 443/tcp     sad_fermat

To access system containers, we have to use the system-docker command with sudo.

We can view a list of the main OS services that have been launched with the command:

[rancher@rancher-os ~]$ sudo system-docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS               NAMES
a566d0c7cc4c        rancher/os-docker:1.12.1    "/usr/bin/user-docker"   8 minutes ago       Up 8 minutes                            docker
8697a04e90a4        rancher/os-console:v0.6.1   "/usr/bin/ros entrypo"   8 minutes ago       Up 8 minutes                            console
c8c337282aaa        rancher/os-base:v0.6.1      "/usr/bin/ros entrypo"   8 minutes ago       Up 8 minutes                            network
1e55244fc99c        rancher/os-base:v0.6.1      "/usr/bin/ros entrypo"   8 minutes ago       Up 8 minutes                            ntp
fd6a03fdb28f        rancher/os-udev:v0.6.1      "/usr/bin/ros entrypo"   8 minutes ago       Up 8 minutes                            udev
7e9e50c28651        rancher/os-acpid:v0.6.1     "/usr/bin/ros entrypo"   8 minutes ago       Up 8 minutes                            acpid
5fec48b060f2        rancher/os-syslog:v0.6.1    "/usr/bin/ros entrypo"   8 minutes ago       Up 8 minutes                            syslog

To restart the system container, we run:

[rancher@rancher-os ~]$ sudo system-docker restart ntp

Configuration

The primary tool for automatically configuring this system is its own implementation of cloud-init, which is split into two utilities: cloud-init-execute and cloud-init-save. Compared to other versions of cloud-init, the RancherOS version uses the declarative language YAML; however, cloud-config.yml for other distributions will not be compatible with RancherOS.

All of the configuration file’s possible directives are given in the official documentation.

System services use the parameters from files in the following order (each subsequent file from the list overwrites overlapping parameters from previous files):

  • /usr/share/ros/os-config.yml — default system settings;
  • /usr/share/ros/oem/oem-config.yml — in our case, this is the file that lets us automatically configure static networks to the parameters from the VPC control panel;
  • YAML files in the directory /var/lib/rancher/conf/cloud-config.d/;
  • /var/lib/rancher/conf/cloud-config.yml — the file that stores values set by the ros utility, which we’ll talk about later on;
  • kernel parameters starting with the keyword “rancher”;
  • /var/lib/rancher/conf/metadata — metadata from useable cloud services added by the cloud-init-save utility.

You can change system settings with the ros utility. New parameters will be shown in the file /var/lib/rancher/conf/cloud-config.yaml:

[rancher@rancher-os ~]$ sudo ros config set rancher.network.dns.nameservers "['188.93.16.19','188.93.17.19', '109.234.159.91']"
[rancher@rancher-os ~]$ sudo cat /var/lib/rancher/conf/cloud-config.yml
rancher:
    network:
        dns:
            nameservers:
              - 188.93.16.19
              - 188.93.17.19
              - 109.234.159.91

In addition to changing the settings of system services, the ros utility is also used for managing OS versions, Docker versions, and configuring TLS and SELinux.

When you create a server in the Virtual Private Cloud from a prepared RancherOS image, you do not have to perform the basic system setup (configure network interfaces, etc.).

When the installation is complete, the server will be accessible via the IP address you choose in the VPC control panel. You will be able to connect to the server over SSH, but authentication will initially only be possible with the key given when you create your server. This key will be added to the user rancher.

Managing Versions

You can find the current version of your operating system by running the command:

[rancher@rancher-os ~]$ sudo ros os version
v0.6.1

To view all available releases:

[rancher@rancher-os ~]$ sudo ros os list
rancher/os:v0.4.0 remote
rancher/os:v0.4.1 remote
rancher/os:v0.4.2 remote
rancher/os:v0.4.3 remote
rancher/os:v0.4.4 remote
rancher/os:v0.4.5 remote
rancher/os:v0.5.0 remote
rancher/os:v0.6.0 remote
rancher/os:v0.6.1 remote

You can install the latest stable version by running the command ros os upgrade or by selecting a version with the -i parameter:

[rancher@rancher-os ~]$ sudo ros os upgrade -i rancher/os:v0.5.0

Since we’ve chosen version 0.5.0, which is older than our current version 0.6.1, the system will downgrade instead of upgrade.

It’s just as easy to manage our version of Docker. Let’s look at several examples:

#to vew available versions
[rancher@rancher-os ~]$ sudo ros engine list
disabled docker-1.10.3
disabled docker-1.11.2
current  docker-1.12.1
#to install version 1.11.2
[rancher@rancher-os ~]$ sudo ros engine switch docker-1.11.2
INFO[0000] Project [os]: Starting project
INFO[0000] [0/19] [docker]: Starting
…
INFO[0010] Recreating docker
INFO[0010] [1/19] [docker]: Started

Docker Swarm Clusters on RancherOS

Integrated tools for managing Docker Swarm container clusters have been included in Docker 1.12. This makes configuring prepared clusters much easier.

In our example, we’ll use the RancherOS image available in the Virtual Private Cloud, which uses Docker ver. 1.12.1.

We’ll have to create three servers with the following parameters:

pr-32843225_eng-03

We’ll start by making our Docker Swarm cluster. We do this by running the following command on node manager0:

[rancher@manager0 ~]$ docker swarm init --advertise-addr 192.168.0.100
Swarm initialized: current node (8gf95qb6l61v5s6561qx5vfy6) is now a manager.
 
To add a worker to this swarm, run the following command:
 
    docker swarm join \
        --token SWMTKN-1-0hlhela57gsvxzoaqol70n2b9wos6qlu3ukriry3pcxyb9j2k6-0n1if4hkvdvmrpbb7f3clx1yg \
        192.168.0.100:2377
 
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

As you can see from the prompt, the following command will have to be run on nodes worker0 and worker1:

docker swarm join \
    --token SWMTKN-1-0hlhela57gsvxzoaqol70n2b9wos6qlu3ukriry3pcxyb9j2k6-0n1if4hkvdvmrpbb7f3clx1yg \
    192.168.0.100:2377

We’ll check the status of the cluster nodes by running the following command on manager0:

[rancher@manager0 ~]$ docker node ls
ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
366euocei01dw3ay66uwzb3pv    worker0   Ready   Active
67pt4fo5gi13cxkphtfzfbfbo    worker1   Ready   Active
8gf95qb6l61v5s6561qx5vfy6 *  manager0  Ready   Active        Leader

Docker Swarm is ready!

By default, new services will be launched on all nodes in the cluster. We’ll change this so that services are only launched on nodes worker0 and worker1, and manager0 will only act as manager.

To change these parameters, we’ll change the “availability” of manager0:

[rancher@manager0 ~]$ docker node update --availability drain manager0

We launch the service with the Nginx container:

[rancher@manager0 ~]$ docker service create --name webserver --replicas 1 --publish 80:80 nginx

We get the service identifier in response:

av1qvj32mz8vwkwihf0lauiz8

The initial number of replicas will be 0, and the only task related to this service will have the status “Preparing”:

[rancher@manager0 ~]$ docker service ls
ID            NAME       REPLICAS  IMAGE  COMMAND
av1qvj32mz8v  webserver  0/1       nginx
[rancher@manager0 ~]$ docker service ps webserver
ID                         NAME         IMAGE  NODE     DESIRED STATE  CURRENT STATE                 ERROR
cjxkza3qzx5m73bgbls0r26m6  webserver.1  nginx  worker1  Running        Preparing about a minute ago

After a few seconds, the required Nginx image will be automatically downloaded from the Docker Hub, after which the service status will change to “Running”:

[rancher@manager0 ~]$ docker service ls
ID            NAME       REPLICAS  IMAGE  COMMAND
av1qvj32mz8v  webserver  1/1       nginx
[rancher@manager0 ~]$ docker service ps webserver
ID                         NAME         IMAGE  NODE     DESIRED STATE  CURRENT STATE          ERROR
cjxkza3qzx5m73bgbls0r26m6  webserver.1  nginx  worker1  Running        Running 8 minutes ago

As we can see, the service was launched on worker1 (the value under the “Node” column). We’ll turn off this node from the VPC control panel and recheck the status of the webserver service:

[rancher@manager0 ~]$ docker service ps webserver
ID                         NAME             IMAGE  NODE     DESIRED STATE  CURRENT STATE             ERROR
5lwf4z54p74qtoopuilshw204  webserver.1      nginx  worker0  Running        Preparing 32 seconds ago
cjxkza3qzx5m73bgbls0r26m6   \_ webserver.1  nginx  worker1  Shutdown       Running 16 minutes ago

The status has changed to “Preparing”. Docker Swarm is trying to move the service to another working node, worker0. The delay is because the Nginx image from Docker Hub should be added to worker0 before it’s launched.

After the image has successfully been added, the webserver service successfully launches on node worker0:

[rancher@manager0 ~]$ docker service ps webserver
ID                         NAME             IMAGE  NODE     DESIRED STATE  CURRENT STATE               ERROR
5lwf4z54p74qtoopuilshw204  webserver.1      nginx  worker0  Running        Running about a minute ago
cjxkza3qzx5m73bgbls0r26m6   \_ webserver.1  nginx  worker1  Shutdown       Running 19 minutes ago

To avoid crashes when moving services from a dropped node, we can increase the number of replicas of the service.

We run the command (don’t forget to start worker1 back up):

[rancher@manager0 ~]$ docker service update --replicas 2 webserver

The service starts to replicate on the second node, worker1:

[rancher@manager0 ~]$ docker service ps webserver
ID                         NAME             IMAGE  NODE     DESIRED STATE  CURRENT STATE            ERROR
5lwf4z54p74qtoopuilshw204  webserver.1      nginx  worker0  Running        Running 11 minutes ago
cjxkza3qzx5m73bgbls0r26m6   \_ webserver.1  nginx  worker1  Shutdown       Complete 6 minutes ago
f3o7icke69zw83ag92ywol0dr  webserver.2      nginx  worker1  Running        Preparing 2 seconds ago

Conclusion

In this article, we took a brief look at Linux distributions that were designed for quickly deploying and easily managing clusters from isolated containers.

We’d like you to share your own experiences with CoreOS, RancherOS, and similar operating systems in the comments below.

Leave a Reply

Your email address will not be published.