Troubleshooting
This page covers problems that cut across the platform: start-up, sign-in, encryption keys, database, TLS of the interface, protocol gateways, deployments and integrations. Problems specific to one device or integration are on its own page; the tables below point to them.
First steps
| Step |
Command |
| State and health of every container |
sudo pkifactor-ctl status |
| System checks (Docker, configuration, images, TLS certificate, disk, firewall, SELinux) |
sudo pkifactor-ctl check |
| Readiness of the platform |
curl -sk https://clm.example.com/health/ready |
| Logs of one service, followed |
sudo pkifactor-ctl logs backend |
| Logs of a time window |
docker logs --since 30m pkifactor_backend |
| Errors only |
docker exec pkifactor_backend tail -n 100 /app/backend/logs/error.log |
| State of a high-availability cluster |
sudo pkifactor-ctl ha-status |
Service names for pkifactor-ctl logs: db (single node), backend, backend_worker (backend-worker on a high-availability node), frontend, nginx, acme, est, cmp, scep; in high availability also etcd, patroni, pgbouncer. The answers of /health/ready are explained in Monitoring and health.
Warnings and errors in the backend log carry a src field (file, line, function): include it, with the time of the problem, when you contact ZetaCert support.
Start-up
| Symptom |
Cause and fix |
A container stays starting or unhealthy |
Read its log (pkifactor-ctl logs <service>). Services wait for the database and the backend to be healthy before starting, so fix the first service that fails |
The backend stops at start with Missing required environment variables: ... |
A variable or a secret file is missing. On the appliance, secret files live in /opt/pkifactor/secrets/ and must not also be set in .env; pkifactor-ctl upgrade recreates missing ones. With Docker Compose on your own host, see Docker Compose |
The backend refuses to start with KMS provider failed (...), no DEK cache |
The key management service (Vault Transit or HSM) did not answer after three attempts and no key cache exists: PKIFactor refuses to start rather than fall back to a weaker key. Restore access to the KMS (network, Vault token or AppRole, Vault CA certificate, HSM library and PIN), then sudo pkifactor-ctl restart. See KMS and HSM |
DEK store not found at ... |
The encrypted data keys (/opt/pkifactor/data/keys/) are missing, typically after copying an installation without its data directory. Restore them from a backup (Backup and restore) |
/health/ready answers 503 with "reason": "schema_error" |
A database migration failed at start-up; the platform does not serve rather than run on a partial schema. The detail is on System > License (Administration workspace) and in the backend log. After an upgrade, pkifactor-ctl upgrade stops on this cause and keeps the backup taken before the upgrade (Upgrade) |
/health/ answers 404 |
There is no such route: use /health/live and /health/ready |
| Containers do not start after a reboot |
Check systemctl status pkifactor.service and that Docker is running; sudo pkifactor-ctl start |
| Disk full, database or logs failing |
pkifactor-ctl check reports free space on /opt/pkifactor. Move old backups off the host (/opt/pkifactor/data/backups/) and remove the Docker images of previous versions (docker image ls, docker image rm) |
Encryption keys and KMS at run time
| Symptom |
Cause and fix |
| Sign-in and browsing work, but issuance and revocation fail with KMS unavailable |
The KMS became unreachable: PKIFactor runs in read-only degraded mode and /health/ready reports degraded. Restore the KMS; PKIFactor leaves degraded mode by itself, or use Reload Keys on Infrastructure > Encryption (Administration workspace) |
Vault Transit: Could not find a suitable TLS CA certificate bundle |
The Vault CA certificate must be a PEM file (intermediate and root), not a directory and not pasted text |
Database
| Symptom |
Cause and fix |
password authentication failed in the backend or gateway logs |
The applications connect with the application role (DB_APP_USER, DB_APP_PASSWORD in .env), the database container with the administration role (DB_USER, DB_PASSWORD). The password in .env no longer matches the role in the database, typically after editing .env by hand. pkifactor-ctl restore realigns both passwords with the restored configuration; otherwise put back the previous values |
Could not align the database role passwords during a restore |
Same cause: check DB_PASSWORD and DB_APP_PASSWORD in .env, then run the restore again |
High availability: the backend cannot resolve db |
A node still has the single-node database settings. Nodes use PgBouncer (DB_HOST=pgbouncer, DB_PORT=6432); ha-init and ha-join write them. See High availability |
High availability: server closed the connection unexpectedly just after a failover |
Pooled connections to the former leader; clears by itself within seconds |
Sign-in
| Symptom |
Cause and fix |
| The initial administrator password is unknown |
On the appliance, the setup wizard showed it once in its summary when it was generated (or you typed it). Otherwise, when DEFAULT_ADMIN_PASSWORD was empty, it is printed once in the backend log at the first start, in a banner INITIAL ADMIN PASSWORD: docker logs pkifactor_backend 2>&1 \| grep -A6 "INITIAL ADMIN". It must be changed at the first sign-in (Users and roles) |
| Account locked after several attempts |
5 consecutive failures lock a local account for 15 minutes (adjustable in System > Settings, General section). Sign-in is also limited to 10 attempts per minute per address |
| Signed out sooner than expected |
Session timeout, 30 minutes of inactivity by default (System > Settings, General section, Session Timeout (minutes); applied without restart). After an upgrade to 1.8.50, a value saved earlier (60) applies, see the release notes. Changing a password ends the user's other sessions |
| The browser offers to choose a client certificate |
The CLM name requests an optional client certificate (mutual TLS) when a CA chain is installed with the platform certificate. Cancel the prompt to sign in with a password |
| SSO: the provider rejects the redirect URI |
Register exactly https://<CLM name>/sso/callback/<provider slug> at the identity provider. The CLM name comes from BASE_URL in .env; in high availability it is the load balancer's name |
SSO: sign-in refused for a provider declared in http:// |
HTTPS is mandatory towards the identity provider (token and JWKS endpoints). Change the provider's addresses to https:// (Single sign-on) |
| SSO: signed in, but no certificate or profile visible |
The user has no organization: organizations come from directory groups or token claims (Organizations and teams, Directory sources) |
| A menu entry or a whole workspace is missing |
The workspaces and entries depend on the role: only administrators have the Administration workspace, administrators and auditors the Oversight workspace. If the browser remembered a workspace the user may no longer open, PKIFactor opens RA. See Find your way in the interface |
TLS of the interface
| Symptom |
Cause and fix |
| Browser warning on the interface |
The platform still uses the self-signed certificate generated at installation, or a certificate whose names do not cover the CLM name. Install yours with sudo pkifactor-ctl tls update <cert> <key> [chain]; nginx is reloaded without interruption (Appliance) |
Some clients fail (unable to get local issuer certificate) while browsers work |
The chain is incomplete: provide the server certificate followed by the intermediate certificates |
| A gateway name (ACME, EST, CMP, SCEP) shows a certificate warning |
The certificate does not list that name: use one certificate whose subject alternative names cover the five names (Requirements) |
| Expiry date of the current certificate |
sudo pkifactor-ctl tls show. PKIFactor can renew its own certificates from a profile (Service certificates) |
Protocol gateways
| Symptom |
Cause and fix |
| The ACME, EST, CMP or SCEP name answers with the interface, or does not answer |
The gateway is not enabled: its variable (ACME_ENABLED, EST_ENABLED, CMP_ENABLED, SCEP_ENABLED) must be true and its service started. nginx only creates the virtual host of an enabled gateway |
| The ACME gateway restarts in a loop |
PUBLIC_ACME_URL is missing from .env (the setup wizard writes it as https://<ACME name>); in high availability, check it on every node |
| ACME client receives 404 on the directory |
The path after /acme/ names no ACME profile: check the directory URL given to the client (ACME server) |
| ACME HTTP-01 validation fails |
The validator connects to port 80 of the name being validated, without following redirections, and refuses private addresses unless internal networks are allowed in the ACME settings (Network flow matrix) |
| EST client authentication by certificate is ignored |
The client CA chain (ca-chain.pem) is missing next to the platform certificate: without it nginx removes the client certificate check at start-up (EST). Also check that EST_PROXY_SECRET is set in .env and passed to nginx and to the EST service: without it the EST gateway refuses every client certificate and logs EST_PROXY_SECRET absent (the setup wizard and pkifactor-ctl upgrade generate it) |
| CMP over CoAP does not answer |
5683/UDP is bound to the local host on a single node: publish it explicitly (CMP) |
A gateway is unhealthy |
pkifactor-ctl logs acme (or est, cmp, scep): the ACME and EST gateways check their database at each probe |
Authorities
| Symptom |
Cause and fix |
| Issuance fails with an authority error |
Read the error on the request, then pki.log and error.log in the backend logs. Network flows per authority are in the Network flow matrix |
| A valid CRL or OCSP answer is reported as unreachable or invalid |
The issuer chain is incomplete, or the CRL or OCSP URL is on a private network: allow it in the authority field Allowed internal networks (revocation) (PKI connectors) |
| Public CA order waiting for domain validation |
See Public CAs and domain validation |
Deployments
| Symptom |
Cause and fix |
| A binding is Failed |
Open the binding: it states the cause of the last failure, its date, the technical detail and the repair options. Then Test connection on the target |
| A binding is in Drift |
The certificate on the device was changed outside PKIFactor; PKIFactor does not overwrite it automatically (How deployment works) |
| Connection test fails |
Port or account of the target: management API on 443 for F5, FortiWeb, PAN-OS, Check Point and Kemp; WinRM over HTTPS (5986) for Windows; SSH (22) for Linux. See the page of the device: F5 BIG-IP, FortiWeb, Palo Alto Networks, Check Point, Kemp, Windows and IIS, SSH servers |
Integrations
| Symptom |
Cause and fix |
| ServiceNow: errors on calls from PKIFactor, requests from the catalogue not processed |
See ServiceNow |
| SIEM receives nothing |
Check the SIEM / Syslog section of Fleet monitoring > Monitoring (Oversight workspace), the protocol (UDP, TCP, TCP with TLS) and the port, and your firewall (SIEM) |
| No e-mail |
Check the SMTP / Email section of Fleet monitoring > Monitoring and the background job Email queue & expiry notices (E-mail and notifications) |
| Directory groups not synchronized |
Test the source on Governance > Group sources; synchronization runs at sign-in, on schedule and on demand (Directory sources) |
Upgrade and high availability
Problems during an upgrade are covered in Upgrade; cluster problems (etcd, Patroni, joining a node) in High availability.