Two root causes wearing one error message
Clients connecting to the Splunk management port were getting SSL verification errors, even though the server was presenting a valid certificate chain. It was two unrelated problems producing one symptom — and my first fix made the error change rather than disappear, which is the part of this I actually want to talk about.
- Where
- Deltek, cloud operations
- When
- 2026
- Role
- Diagnosis, fix, and runbook
- Stack
- Splunk, OpenSSL, Amazon Linux 2
The symptom
An integration couldn’t authenticate to Splunk’s REST API on port 8089. The error was a certificate verification failure — self-signed certificate in certificate chain — which is the kind of message that points at a dozen different causes and commits to none of them.
The confusing part: the server was presenting a valid chain. Pull it apart in a browser or with openssl s_client and everything was there, signed by the corporate CA, in date. And connecting from the command line on the same box worked fine while the application failed.
That last detail is the one worth noticing. When the same TLS connection succeeds from one client and fails from another on the same host, the certificate probably isn’t the problem — whose list of trusted issuers is being consulted is the problem.
Root cause, part one: an incomplete CA bundle
Splunk keeps its own trust store. The bundle it was configured with contained the intermediate CA but not the root. An intermediate on its own can’t anchor anything — verification walks up the chain looking for something it already trusts, hits the intermediate’s issuer, finds nothing, and gives up. Splunk couldn’t build a complete chain because we’d only given it the middle of one.
Root cause, part two: the OS didn’t trust the root either
Separately, the operating system’s trust store had never been told about the corporate root CA. Amazon Linux ships with the public CAs — the ones that sign the public internet — and an internal root issued by the company is, by definition, not among them. Nobody had added it.
So anything doing verification at the system level failed too, for a completely different reason than the application-level failure, while producing a near-identical message.
The detour worth admitting
Having worked out that the OS trust store was missing the internal CA, I added a certificate to it — and picked the wrong one. I put in the intermediate instead of the root.
The error changed. It went from verify error 19 (self-signed certificate in chain) to error 2 (unable to get issuer certificate).
My first instinct was that this was progress. It wasn’t; it was the same problem wearing a different hat. Adding the intermediate gave the verifier one more link to walk but still no anchor, so instead of failing at the top it failed one step higher up. A changed error message feels like movement and frequently isn’t.
What made it legible was going back to the actual chain rather than reasoning about what should be in it — extracting all three certificates straight out of a live TLS handshake and looking at which one was self-signed. The self-signed one is the root. That’s the anchor. It isn’t subtle once you look, and I’d spent longer than I’d like inferring instead of looking.
When a fix changes the error instead of clearing it, treat that as neutral information, not as progress. And when you’re reasoning about a certificate chain, pull the real one off the wire — the chain you assume is being presented and the chain actually being presented are different objects surprisingly often.
The fix
| Step | Action | Purpose |
|---|---|---|
| 1 | Extract all three certificates from the live TLS handshake | See the full chain as presented, not as assumed |
| 2 | Build a proper CA bundle — intermediate plus root | Fix Splunk’s chain verification |
| 3 | Update Splunk’s CA certificate file with the full bundle | Fix trust at the application level |
| 4 | Identify the self-signed certificate as the root | Find the correct OS trust anchor |
| 5 | Place the root CA in the system anchors directory | Fix trust at the OS level |
| 6 | Rebuild the OS CA bundle and restart | Make it take effect |
Two fixes for two causes. Doing only one of them would have left the other failure in place, still producing a certificate error, and looking for all the world like the first fix hadn’t worked.
The part that mattered more than the fix
This condition existed on every server in the estate. It had only surfaced on one because that was the only one with a client strict enough to care yet.
So the deliverable wasn’t the fix, it was the runbook: verify current state, copy the verified root from the fixed host, rebuild the bundle, install the anchor, rebuild the OS trust store, confirm. Written so someone who hadn’t spent a day inside the problem could work through it in ten minutes, with the verification commands included so they’d know it had worked rather than assuming.
Then it went to the team with the summary above — what the problem was, the two root causes, what we did, and the sequence to repeat it. If you fix something that exists in fifty places and only fix it in one, you haven’t finished; you’ve just moved the deadline.
What I’d do differently
I’d have checked the OS trust store and the application trust store as two separate questions from the start. I spent the early part of this treating “SSL is broken” as one problem, when the useful first move is establishing which trust store the failing client consults — and confirming each one independently before touching either.
Server names, the internal CA’s name, domains, and the specific integration are left out. The commands here are the standard OpenSSL and system trust-store operations, deliberately described rather than reproduced with real paths.