6. Troubleshooting

6.1. `stork-agent`

This section describes the solutions for some common issues with the Stork agent.

Issue:

A machine is authorized in the Stork server successfully, but there are no daemons.

Description:

The user has installed and started stork-server and stork-agent and authorized the machine. The “Last Refreshed” column has a value on the Machines page and the “Error” column value shows no errors, but the “Daemons” column is still blank. The “Daemons” section on the specific Machine page is also blank.

Solution:

Make sure that the daemons are running:

Kea Control Agent, Kea DHCPv4 server, and/or Kea DHCPv6 server
BIND 9
PowerDNS

Stork looks for the processes named kea-ctrl-agent (for Kea CA), kea-dhcp4 or kea-dhcp6 (for Kea DHCP servers), kea-dhcp-ddns (for Kea DDNS), named (for BIND 9), or pdns_server (for PowerDNS). Make sure those processes are running and are named appropriately. Use the ps aux (or similar) command to determine whether the processes are running. Currently, Stork does not support detecting off-line services. If BIND 9 or PowerDNS is located in an uncommon location and the Stork agent is unable to detect it, there are several steps that may be helpful. First, enable the DEBUG logging level, so the agent will print more detailed information about locations being checked.

Stork attempts the next four actions to detect named configuration, in order:

Try to parse the -c parameter of the running process;
Use the STORK_AGENT_BIND9_CONFIG environment variable;
Try to parse the output of the named -V command;
Try to find the named.conf file in the default locations.

Stork attempts the next three actions to detect pdns_server configuration, in order: 1. Try to parse the –config-dir parameter of the running process; 2. Use the STORK_AGENT_POWERDNS_CONFIG environment variable; 3. Try to find the pdns.conf file (or a file having a name resulting from the use of the –config-name parameter) in the default locations.

The STORK_AGENT_BIND9_CONFIG or STORK_AGENT_POWERDNS_CONFIG environment variables may be defined to specify the exact locations of the BIND 9 or PowerDNS configuration files.

For BIND 9, make sure that the rndc channel is enabled. By default it is enabled, even if the controls clause is missing. Stork is able to detect default values, so typically there is no administrative action required, unless the rndc channel was explicitly disabled. Make sure the rndc key is readable by the Stork agent.

Also, make sure that BIND 9 has the statistics channel enabled, by adding a statistics-channels entry. Typically, this looks like the following:

statistics-channels {
    inet 127.0.0.1 port 8053 allow { 127.0.0.1; };
};

but it may vary greatly, depending on a given setup. Please consult the BIND 9 Administrator Reference Manual (ARM) for details.

For PowerDNS, make sure that the webserver API is enabled, and that the API key is set. For example:

api=yes
api-key=secret-api-key
webserver=yes
webserver-address=127.0.0.1
webserver-port=8085

Explanation:

If the “Last Refreshed” column has a value, and the “Error” column value has no errors, the communication between stork-server and stork-agent works correctly, which implies that the cause of the problem is between the Stork agent and the daemons. The most likely issue is that none of the Kea/BIND 9/PowerDNS daemons are running. stork-agent communicates with the PowerDNS, BIND 9 and Kea (since Kea version 3.0.0) daemons directly; however, for Kea versions prior to 3.0.0, it communicates with the Kea DHCPv4 and Kea DHCPv6 servers via the Kea Control Agent. If only the “CA” daemon is displayed in the Stork interface, the Kea Control Agent is running, but the DHCP daemons are not.

Issue:

An error occurs after clicking the “Show Zone” button for the selected zone in the zone viewer.

Description:

The “Show Zone” button is clicked for a zone, but the zone viewer shows an error message. The typical error message for PowerDNS server is: “failed to get AXFR credentials for zone 0.0.10.in-addr.arpa: allow-axfr-ips allows neither 127.0.0.1 nor ::1”.

Solution:

Make sure that the DNS server configuration allows zone transfers.

The PowerDNS server has straightforward configuration to enable zone transfers. In fact, the default allow-axfr-ips parameter allows zone transfers from the localhost. If additional IP addresses should be allowed to perform the zone transfer, make sure that the allow-axfr-ips parameter includes 127.0.0.1 or ::1. For example:

allow-axfr-ips=127.0.0.1,::1,172.24.0.10

For BIND 9, the zone transfer is enabled using the allow-transfer parameter. This parameter can be specified at different levels. If there is an issue with a transfer of one of the zones, but not others, it can be an indication that one of the zone-level allow-transfer parameters prohibits the transfer for that zone. Make sure that the allow-transfer permits the transfer. It is also important to note that Stork agent prefers running the zone transfer using a local loopback address (assuming that it finds the DNS server is listening on a local loopback address). In this case, the allow-transfer settings should allow running the zone transfer from the local loopback address. For example:

options {
    listen-on port 53 { 127.0.0.1; 172.24.0.10; };
    allow-transfer { 127.0.0.1; 172.24.0.15; };
};

Note that in the configuration example above, the listen-on parameter explicitly enables the DNS service on the local loopback address. The agent will therefore try to perform the zone transfer using this address. The BIND 9 configuration must also set the allow-transfer parameter to permit zone transfers from the local loopback address, otherwise the aforementioned error will result. Alternatively, the Stork agent supports using the keyword any in the listen-on and allow-transfer parameters; it will attempt zone transfers using the local loopback addresses.

Explanation:

Stork agent uses DNS zone transfer (AXFR) to retrieve the zone data from the monitored DNS servers. If the DNS server configuration does not allow zone transfer, or restricts the zone transfer, such that the Stork agent is not allowed to retrieve the zone data, the zone viewer will report an error.

Issue:: After starting the Stork agent, it gets stuck in an infinite “sleeping” loop.
Description:: stork-agent is running with server support (the --listen-prometheus-only flag is unused). The try to register agent in Stork server message is displayed initially, but the agent only prints the recurring sleeping for 10 seconds before next registration attempt message.
Solution 1:: stork-server is not running. Start the Stork server first and restart the stork-agent daemon.
Solution 2:: The configured server URL in stork-agent is invalid. Correct the URL and restart the agent.

Issue:: stork-agent starts but returns the loaded server cert: /var/lib/stork-agent/certs/cert.pem and key: /var/lib/stork-agent/certs/key.pem message.
Description:: stork-agent runs correctly and its registration is successful. After the started serving Stork Agent message, the agent prints the recurring message about loading server certs. The network traffic analysis to the server reveals that it rejects all packets from the agent (TLS HELLO handshake failed).
Solution:: Re-register the agent to regenerate the certificates, using the stork-agent register command.
Explanation:: The /var/lib/stork-agent/certs/ca.pem file is missing or corrupted. The re-registration removes old files and creates new ones.

Issue:: The cert PEM file is not loaded.
Description:: The agent fails to start and prints an open /var/lib/stork-agent/certs/cert.pem: no such file or directory could not load cert PEM file: /var/lib/stork-agent/certs/cert.pem error message.
Solution:: Re-register the agent to regenerate the certificates, using the stork-agent register command.

Issue:: A connection problem to the DHCP daemon(s) is occurring.
Description:: The agent prints the message problem with connecting to dhcp daemon: unable to forward command to the dhcp6 service: No such file or directory. The server is likely to be offline.
Solution 1:: Try to start the Kea service: systemctl start kea-dhcp4 kea-dhcp6.
Solution 2:: Ensure that the control-socket entry is specified in the Kea DHCP configuration file (kea-dhcp4.conf or kea-dhcp6.conf).
Explanation:: The kea-dhcp4.service or kea-dhcp6.service (depending on the service type in the message) may not be running. If the DHCP daemon is running and operational (it allocates the leases), but the problem is still occurring, inspect the DHCP daemon configuration file (kea-dhcp4.conf or kea-dhcp6.conf). The file must contain the top-level control-socket property with valid content. See the DHCPv4 or DHCPv6 section of the Kea ARM for details. This property is missing by default if Kea is installed from the Debian/Ubuntu repository. To avoid this and similar problems, we recommend using ISC’s official packages on Cloudsmith.

Issue:: stork-agent receives a remote error: tls: certificate required message from Kea.
Description:: The Stork agent and Kea are running, but they cannot establish a connection. The stork-agent log contains the error message mentioned above.
Solution:: Install valid TLS certificates in stork-agent or set the cert-required value in /etc/kea/kea-ctrl-agent.conf to false.
Explanation:: By default, stork-agent does not use TLS when it connects to Kea. If the Kea configuration includes the cert-required value set to true, it requires the Stork agent to use secure connections with valid, trusted TLS certificates. It can be turned off by setting the cert-required value to false when using self-signed certificates, or the Stork agent TLS credentials can be replaced with trusted ones.

Issue:: Kea returns a Kea error response - status: 401, message: Unauthorized message.
Description:: The Stork agent and Kea are running, but they cannot connect. The stork-agent logs contain similar messages: failed to parse responses from Kea: { "result": 401, "text": "Unauthorized" } or Kea error response - status: 401, message: Unauthorized.
Solution:: Check if there are any clients specified in the Kea configuration file in the authentication node.
Explanation:: Kea can be configured to use HTTP Basic Authentication. If it is enabled, the Stork agent will read credentials from the Kea daemon configuration files and use them to authenticate itself. The Stork agent chooses credentials with user name beginning with stork. If there is no such user, the agent will use the first user from the list.

Issue:: During the registration process, stork-agent returns a problem with registering machine: cannot parse address message.
Description:: Stork is configured to use an IPv6 link-local address. The agent prints the try to register agent in Stork server message and then the above error. The agent exists with a fatal status.
Solution:: Use a global IPv6 or an IPv4 address.
Explanation:: IPv6 link-local addresses are not supported by stork-server.

Issue:: A protocol problem occurs during the agent registration.
Description:: During the registration process, stork-agent prints a problem with registering machine: Post "/api/machines": unsupported protocol scheme "" message.
Solution:: The --server-url argument is provided in the wrong format; it must be a canonical URL. It should begin with the protocol (http:// or https://), contain the host (DNS name or IP address; for IPv6 escape them with square brackets), and end with the port (delimited from the host by a colon). For example: http://storkserver:8080.

Issue:: The values in /etc/stork/agent.env were changed, but stork-agent does not notice the changes.
Solution 1:: Restart the daemon.
Solution 2:: Send the SIGHUP signal to the stork-agent process.
Explanation:: stork-agent reads configurations at startup or after receiving the SIGHUP signal.

Issue:: The values in /etc/stork/agent.env were changed and the Stork agent was restarted, but it still uses the default values.
Description:: The agent is running using the stork-agent command. It uses the parameters passed from the command line but ignores the /etc/stork/agent.env file entries. If the agent is running as the systemd daemon, it uses the expected values.
Solution 1:: Load the environment variables from the /etc/stork/agent.env file before running Stork agent. For example, run . /etc/stork/agent.env.
Solution 2:: Run the Stork agent with the --use-env-file switch.
Explanation:: The /etc/stork/agent.env file contains the environment variables, but stork-agent does not automatically load them unless the --use-env-file flag is set; the file must be loaded manually. The default systemd service unit is configured to load this file before starting the agent.

Issue:: Stork shows only the Kea Control Agent on the Daemons page. The running Kea is older than 3.0.0. Stork detects no Kea DHCP servers, although the DHCP daemons are running and allocating leases.
Description:: There is only a single daemon titled “CA” on the Kea Daemons page, but no data about any DHCP daemon or DDNS. The Kea Control Agent and Kea DHCPv4 or Kea DHCPv6 daemon are running and serve leases.
Solution:: The kea-ctrl-agent.conf file is missing the control-sockets property.
Explanation:: Stork detects Kea components using the control socket list from the Kea Control Agent configuration file. The list must be configured properly to allow Stork to send commands to Kea daemons. See the Kea ARM <https://kea.readthedocs.io/en/latest/arm/agent.html#configuration> for details. This property is missing by default if Kea is installed from the Debian/Ubuntu repository. To avoid this and similar problems, we recommend using ISC’s official packages on Cloudsmith.

Issue:: The Stork agent fails to start and returns the following error: failed to load hooks from directory: '[HOOK DIRECTORY]': plugin.Open("[HOOK DIRECTORY]/[FILENAME]"): [HOOK DIRECTORY]/[FILENAME]: file too short or failed to load hooks from directory: '[HOOK DIRECTORY]': plugin.Open("[HOOK DIRECTORY]/[FILENAME]"): [HOOK DIRECTORY]/[FILENAME]: invalid ELF header.
Solution:: Remove the given file from the hook directory.
Explanation:: The file under a given path is not a valid Stork hook.

Issue:: The Stork agent fails to start and returns the following error: Cannot start the Stork Agent: incompatible hook version: 1.0.0.
Solution:: Update the given hook.
Explanation:: The hook is out-of-date and is incompatible with the Stork core application.

Issue:: The Stork agent fails to start and returns the following error: Cannot start the Stork Agent: plugin: symbol Version not found in plugin.
Solution:: Remove or fix the given file.
Explanation:: The hook directory contains the Go plugin, but not the hook; the Go hook does not contain a required symbol.

Issue:: The Stork agent fails to start and returns the following error: Cannot start the Stork Agent: hook library dedicated for another program: Stork Server.
Solution:: Move the incompatible hooks to a separate directory.
Explanation:: The Stork agent requires the hook directory to contain only agent hooks. The error message indicates that the hook directory contains hooks dedicated to the Stork server.

Issue:: The Stork agent starts but the hooks are not loaded. The logs include the following message: Cannot find plugin paths in: /usr/lib/stork-agent/hooks: cannot list hook directory: /usr/lib/stork-agent/hooks: open /usr/lib/stork-agent/hooks: no such file or directory.
Solution:: Create the hook directory or change the path in the configuration.
Explanation:: The hook directory does not exist.

Issue:: The Stork agent fails to start and returns the following error: Cannot start the Stork Agent: open [HOOK DIRECTORY]: permission denied cannot list hook directory.
Solution:: Grant read access to the hook directory to the stork-agent user.
Explanation:: The hook directory is not readable.

Issue:: The Stork agent fails to start and returns the following error: Cannot start the Stork Agent: readdirent [HOOK DIRECTORY]/[FILENAME]: not a directory cannot list hook directory.
Solution:: Change the hook directory path.
Explanation:: A file was found instead of a directory under the given hook directory path.

6.2. `stork-server`

This section describes the solutions for some common issues with the Stork server.

Issue:: The values in /etc/stork/server.env were changed, but stork-server does not notice the changes.
Solution 1:: Restart the daemon.
Solution 2:: Send the SIGHUP signal to the stork-server process.
Explanation:: stork-server reads configurations at startup or after receiving the SIGHUP signal.

Issue:: The values in /etc/stork/server.env were changed and the Stork server was restarted, but it still uses the default values.
Description:: The server is running using the stork-server command. It uses the parameters passed from the command line but ignores the /etc/stork/server.env file entries. If the server is running as the systemd daemon, it uses the expected values.
Solution 1:: Load the environment variables from the /etc/stork/server.env file before running the Stork server. For example, run . /etc/stork/server.env.
Solution 2:: Run the Stork server with the --use-env-file switch.
Explanation:: The /etc/stork/server.env file contains the environment variables, but stork-server does not automatically load them unless the --use-env-file flag is set; the file must be loaded manually. The default systemd service unit is configured to load this file before starting the agent.

Issue:: The server is running but rejects HTTP requests due to a TLS handshake error.
Description:: HTTP requests sent via an Internet browser or tools like curl are rejected. The clients show a message similar to: OpenSSL SSL_write: Broken pipe, errno 32. The Stork server logs contain a TLS handshake error entry with the tls: client didn't provide a certificate description.
Solution 1:: Leave the STORK_REST_TLS_CA_CERTIFICATE environment variable and the --rest-tls-ca flag empty.
Solution 2:: Configure the Internet browser or HTTP tool to use a valid and trusted TLS client certificate. The client certificate must be signed by the authority whose CA certificate was provided in the server configuration.
Explanation:: Providing the STORK_REST_TLS_CA_CERTIFICATE environment variable or the --rest-tls-ca flag turns on TLS client certificate verification. HTTP requests must be assigned with a valid and trusted HTTP certificate, signed by the authority whose CA certificate was provided in the server configuration; otherwise, the request is rejected. This option improves server security by limiting access to only trusted users; it should not be used if there is no CA configured, or if it is desirable to allow login to the Stork server from any computer without prior setup.

Issue:: The Stork server fails to start and returns the following error: permission denied for schema public.
Description:: A fresh installation of the Stork server is made and the database is empty. However, the Stork server does not start and the Stork tool returns an error on the database migration. The logs reveal denied access to the schema public.
Solution 1:: Execute GRANT ALL ON DATABASE stork_db TO stork_user; on the Stork database (replace stork_db and stork_user with the proper names).
Solution 2:: Perform the migration using the Stork tool with the maintenance (e.g., super-admin) database credentials.
Explanation:: In some Postgres installations (by default in Postgres 15 and above), the CREATE permission is only initially granted to the database owner. The Stork server needs this permission to perform the database migration on startup. This permission can be granted manually, or the Stork tool can be used to migrate the schema as the maintenance database user (e.g., super-admin).

Issue:: The Stork server fails to start and returns the following error: Cannot start the Stork Server: failed to load hooks from directory: '[HOOK DIRECTORY]': plugin.Open("[HOOK DIRECTORY]/[FILENAME]"): [HOOK DIRECTORY]/[FILENAME]: file too short or Cannot start the Stork Server: failed to load hooks from directory: '[HOOK DIRECTORY]': plugin.Open("[HOOK DIRECTORY]/[FILENAME]"): [HOOK DIRECTORY]/[FILENAME]: invalid ELF header.
Solution:: Remove the given file from the hook directory.
Explanation:: The file under the given path is not a valid Stork hook.

Issue:: The Stork server fails to start and returns the following error: Cannot start the Stork Server: incompatible hook version: 1.0.0.
Solution:: Update the given hook.
Explanation:: The hook is out-of-date and is incompatible with the Stork core application.

Issue:: The Stork server fails to start and returns the following error: Cannot start the Stork Server: plugin: symbol Version not found in plugin.
Solution:: Remove or fix the given file.
Explanation:: The hook directory contains the Go plugin but not the hook; the Go hook does not contain a required symbol.

Issue:: The Stork server fails to start and returns the following error: Cannot start the Stork Server: hook library dedicated for another program: Stork Agent.
Solution:: Move the incompatible hooks to a separate directory.
Explanation:: The Stork server requires the hook directory to contain only server hooks. The error message indicates that the hook directory contains hooks dedicated to the Stork agent.

Issue:: The Stork server starts but the hooks are not loaded. The logs include the following message: Cannot find plugin paths in: /usr/lib/stork-server/hooks: cannot list hook directory: /usr/lib/stork-server/hooks: open /usr/lib/stork-server/hooks: no such file or directory.
Solution:: Create the hook directory or change the path in the configuration.
Explanation:: The hook directory does not exist.

Issue:: The Stork server fails to start and returns the following error: Cannot start the Stork Server: open [HOOK DIRECTORY]: permission denied cannot list hook directory.
Solution:: Grant read access to the hook directory to the stork-server user.
Explanation:: The hook directory is not readable.

Issue:: The Stork server fails to start and returns the following error: Cannot start the Stork Server: readdirent [HOOK DIRECTORY]/[FILENAME]: not a directory cannot list hook directory.
Solution:: Change the hook directory path.
Explanation:: A file was found instead of a directory under the given hook directory path.

Issue:: The Stork server displays invalid utilization on the subnet or shared network page. The value is negative or significantly exceeds 100%.
Solution:: Check the Daemons list and the Machines list against the duplicated daemons. If the same daemons are detected twice, remove an instance that communicates through Kea CA. If there are no duplicates, verify whether some Kea servers that are not an HA pair don’t use the same address or PD pools.
Explanation:: In Stork 2.3.2 we introduced support for Direct Kea API. Since then, Stork can communicate directly with Kea DHCP servers, without the Kea Control Agent. Due to technical reasons, Stork cannot match the daemons detected via the Kea CA with those detected via the Direct Kea API. It caused the same Kea server to be represented twice in the Stork database. It significantly disrupts statistical calculations, especially when some servers are configured as an HA pair or a Hub-and-Spoke. As a result, the statistics may be unreliable, and utilization may be negative or above 100%. In such a case, the administrator should examine the Daemon list and remove the duplicates detected via Kea CA. It can be recognized by checking the daemon’s access point list—the instance to delete points to the address and port of the Kea CA control channel. Unreliable or duplicated statistics can also be observed if some separate daemons that are not configured as an HA pair share or overlap their address or PD pools. In such a situation, the administrator should reconfigure their Kea servers to separate address or PD pools. Stork detects such misconfiguration and presents recommendations as a Config Review Report on the daemon page.

Issue:

The Stork server does not display any leases in the Leases List page.

Solution:

Check the logs for both the Stork server and any agents which are expected to return leases. This could be caused by a number of conditions. Verify that (1) there is at least one Kea daemon configured to use the memfile lease storage backend, (2) all Stork agents monitoring memfile Kea daemons have lease tracking enabled via the STORK_AGENT_ENABLE_LEASE_TRACKING=1 environment variable or the --enable-lease-tracking command-line flag, and (3) the Stork agent user has permission to read Kea’s lease memfile (and permission to browse the directories containing it).

Explanation:

Presently, the Leases List feature only supports Kea daemons which use a memfile as the lease storage backend. This is a limitation which we anticipate lifting in the future, but it is the case for now. The system does not currently generate an error or warning message when there are no Kea daemons with the memfile lease storage backend configured, because this would create log spam for little benefit.

The Stork agent directly reads the Kea lease memfile to discover lease information, so the feature can only operate if the agent has read permissions for the lease memfile. If the agent lacks permissions, it will report errors in its logs like this:

WARN[2026-06-16 19:32:16]          monitor.go:507   Failed to refresh state of the daemon         daemon="dhcp4: control: unix:///var/run/kea/kea4-ctrl-socket/ (auth key: not found)" error="open /var/lib/kea/kea-leases4.csv: permission denied" stackTrace="open /var/lib/kea/kea-leases4.csv: permission denied"

One way to resolve this is to add the Stork agent user to the Kea group, using a command like # usermod -a -G _kea stork-agent (on Debian 13, using Kea from ISC’s repository).

In order to avoid incurring performance penalties when not desired, the Stork agent must be explicitly instructed to enable lease tracking features. If it has not been given this instruction, the Stork server will log messages like this:

time="2026-06-16 19:47:37" level="error" msg="Errors were encountered while executing a periodic action of Kea Leases puller" file=" periodicexecutor.go:178  " error="could not retrieve leases from daemon 16: failed to receive Kea leases from the agent: rpc error: code = Unknown desc = This feature is not enabled. Pass `--enable-lease-tracking` or set `STORK_AGENT_ENABLE_LEASE_TRACKING` when starting the agent."

Setting the environment variable or providing the command line flag both instruct the agent to enable this feature. Note that you will need to make the same change for every agent where you want this feature to be enabled.

Issue:

The Stork server does not display any zone transfers in the Zone Transfers List page when BIND writes logs to files.

Solution:

Check that zone transfer tracking is enabled on the respective Stork agents and that the BIND servers log zone transfers at least at info severity level. Make sure that log files containing zone transfers are accessible to the Stork agents.

Explanation:

There are numerous possible reasons why zone transfers are not displayed in the Zone Transfers List page. Since zone transfer tracking may potentially be very resource-intensive (depending on the number of zones and the frequency of zone transfers), it is disabled by default on the Stork agents. The admin must explicitly enable zone transfers using the --enable-zone-transfer-tracking command-line flag or the STORK_AGENT_ENABLE_ZONE_TRANSFER_TRACKING=1 environment variable.

Stork agents detect zone transfers by parsing the BIND servers’ log files. The useful logs are typically logged at info severity level. If logging at this level is too verbose, the admin can configure the servers to only log zone transfer events at this level and other events at less verbose levels. It is best to log zone transfer events into separate files, so the Stork agent does not have to parse unrelated logs among the zone transfer logs. Finally, the administrator must ensure proper permissions on the log files, so the Stork agent can read them. This is typically achieved by assigning the stork-agent user to a named or bind group.

Issue:

The Stork server does not display any zone transfers in the Zone Transfers List page when BIND writes logs to the systemd journal.

Solution:

Check that zone transfer tracking is enabled on the respective Stork agents and that the BIND servers log zone transfers at least at info severity level. Make sure that Stork agents were started with the --xfr-tracking-systemd-unit flag or the STORK_AGENT_XFR_TRACKING_SYSTEMD_UNIT environment variable set to the BIND service name (typically named.service or simply named). Finally, make sure that journalctl command is runnable by the Stork agent.

Explanation:

Stork agent has no means to know whether BIND was started using systemd or not. Therefore, the administrator must explicitly enable zone transfers tracking in the systemd journal. This is enabled with the --xfr-tracking-systemd-unit flag or the STORK_AGENT_XFR_TRACKING_SYSTEMD_UNIT environment variable. When enabled, Stork agent runs the journalctl command to monitor zone transfers logged into the systemd journal. The command is run with the following parameters:

$ journalctl -o short-iso-precise -f -u named.service --since "1 days ago"

To troubleshoot the problems with the journal monitoring, the administrator can run this command manually and check the output. Make sure that the user running the Stork agent (typically stork-agent) belongs to the group that can access the systemd journal. It is often the systemd-journal group.

Issue:

Timestamps in the Zone Transfers List page are invalid (i.e., 0001-01-01 00:00:00).

Solution:

Make sure that BIND is configured to print time in the logs using the print-time yes setting.

Explanation:

When Stork agent is started it looks into past logs to detect zone transfers before it waits for the new ones. It guarantees that no transfers are missed during the periods of Stork agent’s inactivity. To ensure correctness of the timestamps of these past transfers, the agent must use the timestamps from the BIND logs. The agent will still report the transfers when logs missing the timestamps. However, in that case, all transfers will have the zero timestamp printed in the dashboard. This can be corrected by adding print-time yes in the configuration of the logging channel used by xfer-in and xfer-out categories. For example:

channel xfer-in {
    file "/var/log/bind/xfer-in" versions 3 size 10M;
    print-time yes;
    severity info;
};

Issue:

After upgrading the Stork server from a version earlier than 2.5.0, users cannot log in via LDAP. The Stork UI displays the “Invalid login or password” message, even though the credentials are correct, and the Stork server logs an error like this:

time="2026-10-08 11:25:01" level="error" msg="Cannot authenticate a user" file="            users.go:168  " error="cannot authenticate a user: error occurred in the Authenticate callout: missing unique identifier attribute (entryUUID): LDAP user entry is missing required attribute" identifier="alice" method="ldap"

Solution:

Set the --ldap.object-class-user-unique-identifier command-line option or the STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_USER_UNIQUE_IDENTIFIER environment variable to a unique, immutable attribute supported by your LDAP server, e.g. objectGUID for Microsoft Active Directory, and restart the Stork server.

Explanation:

Since version 2.5.0, the Stork LDAP hook uses the entryUUID attribute by default to identify users. It is available in OpenLDAP and other servers that implement RFC 4530, but not in all LDAP servers. If the attribute is missing in the user entry, the authentication fails. After the attribute is changed, the users who logged in while Stork was running a version earlier than 2.5.0 must be migrated; see the next issue.

Issue:

After upgrading the Stork server from a version earlier than 2.5.0, users who logged in via LDAP before the upgrade cannot log in. Since Stork 2.6.0, the Stork UI displays the “Error during authentication process. Please contact Stork admin.” message, and the Stork server logs an error like this:

time="2026-10-08 11:25:01" level="error" msg="Internal server error after successful external authentication, cannot authenticate a user" file="            users.go:162  " error="database operation error while trying to add or update external user with ID '1234-5678': conflicting data in the database for external user with login 'alice' and email 'alice@example.org': ERROR #23505 duplicate key value violates unique constraint \"system_user_login_unique_idx\"" identifier="alice" method="ldap"

Solution:

Use the stork-tool command migrate-ldap-system-users to fetch the new unique identifiers of the users from the LDAP server. Use the same LDAP settings as configured for the Stork server, including the unique identifier attribute.

Explanation:

Before version 2.5.0, the Stork LDAP hook used the DN (distinguished name) as the unique identifier. We found that the DN is not suitable for this purpose because it can change, e.g. when users are moved between organizational units. Therefore, since version 2.5.0, the default is entryUUID. The Stork database still stores the DN as the unique identifier of users who logged in before the upgrade, so it does not match the new identifier returned by the LDAP server. Stork then tries to create a new account with the same login or email, which conflicts with the existing one. If both the login and the email of the user changed in LDAP, a new account is created instead, and the old one is left unused. The migration needs to be run once after upgrading from a version earlier than 2.5.0, and again after each subsequent change of the unique identifier attribute.

Alternatively, set the unique identifier attribute to dn to keep the previous behavior. This is not recommended unless distinguished names never change and are never reused in your organization.

6.3. High Virtual Memory Usage

Stork processes allocate a large amount of virtual memory, which is a common situation for applications written in Golang. The Go runtime uses virtual memory to manage memory efficiently. Virtual memory is not the same as physical memory. The size of the reserved virtual memory depends on the internal implementation details of the Go memory allocator. A high value of virtual memory usage is not alarming, as long as real memory usage is low.

Virtual and physical memory usage can be examined using the ps aux command. Virtual memory usage is displayed in the VSZ column; the RSS column shows physical memory usage.

The usual virtual memory usage of the Stork agent on a machine with 16GB RAM, Go 1.22.4, and Ubuntu 22.04 is about 2.5-3GB. The real memory usage is relatively low, about 10-40MB for Kea deployments with dozens of subnets and host reservations and 40-80MB for deployments with thousands of subnets and host reservations.

References:

6. Troubleshooting

6.1. stork-agent

6.2. stork-server

6.3. High Virtual Memory Usage

6.1. `stork-agent`

6.2. `stork-server`