5. Developer’s Guide

Note

We acknowledge that users and developers have different needs, so the user and developer documents should eventually be separated. However, since the project is still in its early stages, this section is kept in the Stork ARM for convenience.

5.1. Rakefile

Rakefile is a script for performing many development tasks like building source code, running linters, running unit tests, and running Stork services directly or in Docker containers.

There are several other Rake targets. For a complete list of available tasks, use rake -T. Also see the Stork wiki for detailed instructions.

5.2. Generating Documentation

To generate documentation, simply type rake doc. Sphinx and rtd-theme must be installed. The generated documentation will be available in the doc/singlehtml directory.

5.3. Setting Up the Development Environment

The following steps install Stork and its dependencies natively, i.e. on the host machine, rather than using Docker images.

First, PostgreSQL must be installed. This is OS-specific, so please follow the instructions from the Installation chapter.

Optional step: to initialize the database directly, the migrations tool must be built and used to initialize and upgrade the database to the latest schema. However, this is completely optional, as the database migration is triggered automatically upon server startup. This is only useful if for some reason it is desirable to set up the database but not yet run the server. In most cases this step can be skipped.

$ rake build_migrations
$ backend/cmd/stork-db-migrate/stork-db-migrate init
$ backend/cmd/stork-db-migrate/stork-db-migrate up

Once the database environment is set up, the next step is to build all the tools. Note the first command below downloads some missing dependencies and installs them in a local directory. This is done only once and is not needed for future rebuilds, although it is safe to rerun the command.

$ rake build_backend
$ rake build_ui

The environment should be ready to run! Open three consoles and run the following three commands, one in each console:

$ rake run_server
$ rake serve_ui
$ rake run_agent

Once all three processes are running, connect to http://localhost:8080 via a web browser. See Using Stork for initial password information or for adding new machines to the server.

The run_agent runs the agent directly on the current operating system, natively; the exposed port of the agent is 8888.

There are other Rake tasks for running preconfigured agents in Docker containers. They are exposed to the host on specific ports.

When these agents are added as machines in the Stork Server UI, both a localhost address and a port specific to a given container must be specified. This is a list of ports for particular Rake tasks and containers:

  • rake run_kea_container: Kea with DHCPv4, port 8888
  • rake run_kea6_container: Kea with DHCPv6, port 8886
  • rake run_kea_ha_containers (2 containers): Kea 1 and 2 with preconfigured HA, ports 8881 and 8882
  • rake run_bind9_container: port 9999

5.3.1. Installing Git Hooks

There is a simple git hook that inserts the issue number in the commit message automatically; to use it, go to the utils directory and run the git-hooks-install script. It will copy the necessary file to the .git/hooks directory.

5.4. Agent API

The connection between the server and the agents is established using gRPC over http/2. The agent API definition is kept in the backend/api/agent.proto file. For debugging purposes, it is possible to connect to the agent using the grpcurl tool. For example, a list of currently provided gRPC calls may be retrieved with this command:

$ grpcurl -plaintext -proto backend/api/agent.proto localhost:8888 describe
agentapi.Agent is a service:
service Agent {
  rpc detectServices ( .agentapi.DetectServicesReq ) returns ( .agentapi.DetectServicesRsp );
  rpc getState ( .agentapi.GetStateReq ) returns ( .agentapi.GetStateRsp );
  rpc restartKea ( .agentapi.RestartKeaReq ) returns ( .agentapi.RestartKeaRsp );
}

Specific gRPC calls can also be made. For example, to get the machine state, the following command can be used:

$ grpcurl -plaintext -proto backend/api/agent.proto localhost:8888 agentapi.Agent.getState
{
  "agentVersion": "0.1.0",
  "hostname": "copernicus",
  "cpus": "8",
  "cpusLoad": "1.68 1.46 1.28",
  "memory": "16",
  "usedMemory": "59",
  "uptime": "2",
  "os": "darwin",
  "platform": "darwin",
  "platformFamily": "Standalone Workstation",
  "platformVersion": "10.14.6",
  "kernelVersion": "18.7.0",
  "kernelArch": "x86_64",
  "hostID": "c41337a1-0ec3-3896-a954-a1f85e849d53"
}

5.5. ReST API

The primary user of the ReST API is the Stork UI in a web browser. The definition of the ReST API is located in the api folder and is described in Swagger 2.0 format.

The description in Swagger is split into multiple files. Two files comprise a tag group:

  • *-paths.yaml - defines URLs
  • *-defs.yaml - contains entity definitions

All these files are combined by the yamlinc tool into a single Swagger file swagger.yaml. Then, swagger.yaml generates code for:

  • the UI fronted by swagger-codegen
  • the backend in Go lang by go-swagger

All these steps are accomplished by Rakefile.

5.6. Backend Unit Tests

There are unit tests for backend part (agent and server) written in Go. They can be run using Rake:

$ rake unittest_backend

This requires preparing a database in PostgreSQL. One way to avoid doing this manually is by using a docker container with PostgreSQL which is automatically created when running the following Rake task:

$ rake unittest_backend_db

This one task spawns a container with PostgreSQL in the background and then it runs unit tests. When the tests are completed the database is shutdown and removed.

5.6.1. Unit Tests Database

When docker container with a database is not used for unit tests, the PostgreSQL server must be stared and the following role must be created:

postgres=# CREATE USER storktest WITH PASSWORD 'storktest';
CREATE ROLE
postgres=# ALTER ROLE storktest SUPERUSER;
ALTER ROLE

To point unit tests to our specific database set POSTGRES_ADDR environment variable, e.g.:

$ rake unittest_backend POSTGRES_ADDR=host:port

By default it points to localhost:5432.

5.6.2. Unit Tests Coverage

At the end of tests execution there is coverage report presented. If coverage of any module is below a threshold of 35% then an error is raised.

5.7. Docker Containers

To ease testing, there are several Docker containers available.

  • server - This container is essential. It runs the Stork server, which interacts with all the agents and the database and exposes the API. Without it, Stork will not be able to function.
  • postgres - This container is essential. It runs the PostgreSQL database that is used by the Stork server. Without it, the Stork server will produce error messages about an unavailable database.
  • webui - This container is essential in most circumstances. It provides the front-end web interface. It is potentially unnecessary with the custom development of a Stork API client.

There are also several containers provided that are used to samples and they are not strictly necessary. The following containers will not be needed in a production network, however they’re very useful to demonstrate existing Stork capabilities. They simulate certain services that Stork is able to handle:

  • agent-bind9 - This container runs a BIND 9 server. With this container, the agent can be added as a machine and Stork will begin monitoring its BIND 9 service.
  • agent-bind9-2 - This container also runs a BIND 9 server, for the purpose of experimenting with two different DNS servers.
  • agent-kea - This container runs a Kea DHCPv4 server. With this container, the agent can be added as a machine and Stork will begin monitoring its Kea DHCPv4 service.
  • agent-kea-ha1 and agent-kea-ha2 - These two containers should, in general, be run together. They each have a Kea DHCPv4 server instance configured in a HA pair. With both running and registered as machines in Stork, users can observe certain HA mechanisms, such as one taking over the traffic if the partner becomes unavailable.
  • traffic-dhcp - This container is optional. If started, it can be used to transmit DHCP packets to agent-kea. It may be useful to observe non-zero statistics coming from Kea. When running Stork in Docker, rake start_traffic_dhcp can be used to conveniently control traffic.
  • traffic-dns - This container is optional. If stated, it can be used to transmit DNS packets towards agent-bind9. It may be useful to observe non-zero statistics coming from BIND 9. If you’re running Stork in docker, you can conveniently control that using rake start_traffic_dns.
  • prometheus - This is a container with Prometheus for monitoring applications. It is preconfigured to monitor Kea and BIND 9 containers.
  • grafana - This is a container with Grafana, a dashboard for Prometheus. It is preconfigured to pull data from a Prometheus container and show Stork dashboards.

5.8. Packaging

There are scripts for packaging the binary form of Stork. There are two supported formats:

  • RPM
  • deb

The RPM package is built on the latest CentOS version. The deb package is built on the latest Ubuntu LTS.

There are two packages built for each system: a server and an agent.

There are Rake tasks that perform the entire build procedure in a Docker container: build_rpms_in_docker and build_debs_in_docker. It is also possible to build packages directly in the current operating system; this is provided by the deb_agent, rpm_agent, deb_server, and rpm_server Rake tasks.

Internally, these packages are built by FPM (https://fpm.readthedocs.io/). The containers that are used to build packages are prebuilt with all dependencies required, using the build_fpm_containers Rake task. The definitions of these containers are placed in docker/pkgs/centos-8.txt and docker/pkgs/ubuntu-18-04.txt.