Ansible CIS Hardening for Ubuntu 24.04

TL;DR: Seven Ansible roles, one per CIS section group, harden a default Ubuntu 24.04 server to CIS Level 1 – Server. OpenSCAP scans the machine before and after with the ComplianceAsCode profile, the playbook runs a second time to prove it changes nothing, and a small Python tool compares the scans and fails CI on a low score, a claimed control that still fails, a regression or a second run that changes something. Every control is traceable from its CIS id to the role that applies it and to the scanner rules that check it. Everything runs in GitHub Actions on a disposable runner. The CI runs are real; the host is a disposable CI runner.

1. Problem

A hardening baseline usually lives in two places that drift apart: a script or playbook that someone wrote once, and a benchmark document that auditors read. Nobody can say which CIS control a given task implements, whether the server still passes after the last change, or which controls were skipped on purpose and why. And a playbook that reports changes on every run hides real drift in noise.

2. Design

• Own roles by section. Seven roles follow the benchmark's sections (initial setup, services, network, firewall, access, logging, maintenance). Every task is named and tagged with its CIS id, so a control can be found, run or skipped on its own.
• An independent scanner. The roles do not grade themselves. OpenSCAP with the ComplianceAsCode profile cis_level1_server from a pinned SCAP Security Guide release (checked by SHA-512) scans before and after. If the release changes or disappears, the run stops; the scanner is never swapped silently.
• Claims from the real baseline. policy/controls.yml maps each CIS id to its role and to the scanner rules that check it. It was built from the first baseline scan of a default runner: every rule that failed and that a role fixes, plus the rules that only apply once a role installs their package — 91 controls, 126 rules. If any of them still fails after hardening, CI fails.
• Exceptions as data. A skipped control is an entry with the CIS id, its rules, a reason and an owner role. Tasks skip it, the report lists it and shows the score with and without it, and an exception for a rule the roles also claim, without a reason or for a rule the profile does not have, is a configuration error.
• Runner guardrails. The target is the runner itself, so the hardening must not cut the job off: outbound traffic stays open on the ports the job needs (DNS, HTTP, HTTPS, NTP) and the runner account keeps its sudo.

3. The cycle

1. Download SCAP Security Guide v0.1.82 and verify its SHA-512.
2. Scan before: oscap xccdf eval --profile …cis_level1_server (ARF and HTML).
3. ansible-playbook playbooks/harden.yml against localhost with become.
4. The same playbook again: the recap must show changed=0 and failed=0.
5. Scan after with the same content.
6. cisreport report compares the scans and writes HTML, Markdown and JSON; cisreport gate decides.

4. Measurement and gate

• Score = pass / (pass + fail) over the profile's selected rules, before and after, with all rules and excluding the exception rules. notapplicable and notchecked do not count either way.
• Fixed, regressions, claimed failing, open: rules that went from fail to pass; rules that went from pass to fail; claimed rules that still fail; failing rules that are neither claimed nor excepted.
• Gate (exit 1): the score excluding exceptions below 90 %, any claimed rule failing, any regression (even when the overall score rose), or a second run with changed > 0 or failed > 0.
• Errors, not scores (exit 2): a missing, empty or truncated scan, a profile that selected no rules, no rule that passed or failed, more than 5 % of the selected rules in error or notchecked, or a missing recap. A scan that did not happen must never read as 100 % or 0 %. A claimed rule that ends in error or notchecked counts as still failing.

5. Roles

Exceptions (in inventory/group_vars/all.yml):

6. Pipeline

It runs on every pull request, on pushes to main, weekly and on demand.

7. Results

From run 37353023826 (2026-10-05), the final code after the review fixes; docs/example-report.html is its report. OpenSCAP 1.3.9 (the runner's openscap-scanner package), SCAP Security Guide v0.1.82, profile cis_level1_server (408 selected rules).

• Fixed: 110 rules went from fail to pass. Regressions: 0. Claimed rules still failing: 0 (91 CIS controls, 126 rules claimed).
• Still failing (10): the 8 rules of the documented exceptions (separate /tmp partition, the nftables variant of the firewall section, AIDE) and 2 open items, reported, not claimed and not hidden:
• file_permission_user_init_files: the role sets every regular dot file in the home directories to 0740 or less, but the runner's home directories contain dot-name symbolic links (.ghcup), whose mode always reads as 0777 to the check.
• no_files_unowned_by_user: the only unowned files are inside Docker's image layers (/var/lib/docker/overlay2), whose UIDs exist only inside containers. The role deliberately leaves container storage alone — giving those files to root breaks containers on a real host.
• The first green run, 37330148201, scored 97.5 % (99.7 % excluding exceptions) with only the first open item: the role then gave every unowned file to root, Docker's image layers included. The final review flagged that as unsafe on a real server; the 0.3-point drop is the price of the fix.
• Applicability moved too (not applicable 64 → 53): 15 rules became applicable and pass — 12 password-quality rules once libpam-pwquality is installed and 3 systemd-timesyncd rules — and 4 chrony rules became not applicable once chrony was removed.
• Idempotency: first run ok=77 changed=55 failed=0; second run ok=68 changed=0 failed=0.
• Time: the harden job takes 12–16 minutes (two scans, two playbook runs); the whole pipeline about the same, since the other jobs run in parallel.

How it got there. The first complete cycle (run 37327562351) already scored 91.5 % (93.6 % excluding exceptions), over the 90 % threshold, and the gate still failed it: one regression, 8 claimed rules still failing and a second run that changed something. Each cause came from reading the scanner's own checks in the results file, and the fixes are in section 8.

Demo pull requests (closed unmerged; the ruleset blocks the merge):

8. Lessons

• The profile, not memory of the benchmark, decides what "compliant" means. The pinned profile expects systemd-timesyncd (not chrony) for time sync, SSH LogLevel INFO, rsyslog as the logging path, and does not select auditd at Level 1. The roles were written from the list of rules that actually failed on the baseline scan (408 selected rules: 222 pass, 122 fail, 64 not applicable), not from a reading of the benchmark.
• Small file details break privileged changes. The feasibility spike wrote its sudoers drop-in with tee, which left it 0644, and visudo -c rejected it; the role writes 0440 and validates its own file before installing it. On the runner, two AppArmor profiles claim the same browser binary, so aa-complain refuses to run at all; the role edits the unconfined flag in each profile directly and reloads only the profiles it changed.
• A service that has never run leaves no runtime directory. SSH on the runner is socket-activated, so /run/sshd does not exist and sshd -t refuses to validate a config; the role creates the directory first.
• Some controls do not fit a short-lived machine. Installing AIDE pulled in a mail server, and building its database did not finish in 15 minutes on the runner's large image; it became a documented exception instead of a quietly skipped task.
• Tools read configuration from where they run. Molecule runs from the role's directory, where the repository's ansible.cfg (and its collections path) is not read; CI passes the collections path explicitly.
• Another component can undo a setting. ufw re-applies its own /etc/ufw/sysctl.conf (with log_martians=0) when it is enabled, and Ubuntu's 10-network-security.conf sets rp_filter=2. Our sysctl file was right, yet the scanner failed four rules and the second playbook run "fixed" the value again — which is how the idempotency check found it. The role now makes every file that sets those keys agree.
• The check reads one file; the tool reads several. pam_pwquality reads pwquality.conf.d, but the profile's checks read only pwquality.conf, and the TMOUT check matches one exact three-line form. Settings that work are not always settings the scanner can see; the role writes them where and how the check looks.
• Fixing one rule can break another. Creating /etc/cron.allow (a rule that failed) with the wrong group made a rule that had passed — because the file did not exist — fail. The gate's regression check caught it even though the overall score had risen.
• Per-item loops do not scale to a real image. The runner has more than 15,000 world-writable files under its toolchains; one Ansible module call per file ran for 54 minutes and hit the job timeout. One find … -exec … {} + per rule does the same work in seconds and still reports whether anything changed.
• A configuration file in the wrong folder is silently ignored. The exceptions and tunables lived in a group_vars/ folder at the repository root, where cisreport read them but Ansible does not look; the play ran on the role defaults. It changed little — the excepted tasks were also off by default, and only the SSH AllowGroups list differed — and it surfaced only when the final review added an SSH lockout check. The file now sits next to the inventory, and CI fails if the play cannot see the exceptions.

9. Limits

• The target is a GitHub-hosted runner, not a production server; three controls are exceptions because the runner is booted and short-lived, and two rules stay open because of the runner image (dot-name symbolic links; Docker image layers).
• The score comes from the ComplianceAsCode profile, which follows CIS but is not CIS-CAT and not a certification.
• Settings that need a reboot (AppArmor on the kernel command line, the bootloader password) are written and checked by the scanner, not booted into.
• On a real server, some decisions stay with its owner (services to keep, outbound ports, the bootloader password hash, accounts with old passwords); the README lists them and the guards the roles apply.
• Not covered: Level 2, Windows Server, CIS-CAT Pro, fleet management.

10. Reproduce it

Fork the repository and enable Actions: every push runs the cycle on a fresh runner. On a throwaway Ubuntu 24.04 VM, bash scripts/run-hardening.sh hardens that VM and writes out/report.html and out/results.json. The roles can be tested in containers with molecule test from each role's directory.

11. Mapping