Linux Sysadmin Notes

TLS Troubleshooting

5 min read

Tap Listen. On Android keep Chrome in the foreground and volume up.

TLS Troubleshooting

Concept

TLS failures are almost never “SSL is broken”. They are a mismatch in one of: certificate identity, validity window, chain of trust, protocol/cipher overlap, SNI, or the application not using the file you think it is using.

Why it matters

  • Browsers, load balancers, and language runtimes all fail differently on the same bad cert
  • A hostname mismatch or incomplete chain can look like a random timeout from some clients only
  • Expiry is still the most common production TLS incident and is 100% predictable
  • Protocol or cipher floors (TLS 1.2+, no old RSA-only stacks) break legacy clients after a “security hardening” change

openssl s_client is the source of truth for what the server actually presents, not what the ticket says was deployed.

Mental Model

Client hello (SNI + protocol + ciphers)
  → Server hello + certificate list
  → Client validates:
       name (SAN / CN) matches the name it connected to
       now is inside notBefore / notAfter
       chain walks to a local trust anchor
       usage / EKU is acceptable
  → Key exchange + finished
  → Application data

Break it into layers:
  DNS / VIP / SNI        → did I reach the intended cert?
  Leaf cert              → name, dates, key type
  Chain                  → intermediates sent? order?
  Trust store            → does *this* client trust the CA?
  Protocol / ciphers     → overlap between client and server
  App config             → which files, which passphrase, which server block?

Key Commands

# What the server presents (replace name and port)
echo | openssl s_client -connect example.com:443 -servername example.com -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer -dates -ext subjectAltName
 
# Full handshake noise (protocol, cipher, alerts)
openssl s_client -connect example.com:443 -servername example.com -tls1_2
 
# Verify against a CA bundle (system or custom)
echo | openssl s_client -connect example.com:443 -servername example.com -CAfile /etc/pki/tls/certs/ca-bundle.crt </dev/null
 
# Inspect a file on disk
openssl x509 -in /etc/pki/tls/certs/app.crt -noout -text | less
openssl x509 -in app.crt -noout -subject -issuer -dates -ext subjectAltName
openssl verify -CAfile ca-chain.pem app.crt
 
# Confirm key matches cert (hashes must be equal)
openssl x509 -noout -modulus -in app.crt | openssl md5
openssl rsa  -noout -modulus -in app.key | openssl md5
 
# Chain completeness: count certs the server sent
echo | openssl s_client -connect example.com:443 -servername example.com -showcerts 2>/dev/null | grep -c "BEGIN CERTIFICATE"
 
# Quick client-side view
curl -vI --max-time 10 https://example.com/
curl -vI --cacert /path/to/ca.pem https://internal.example/
 
# Expiry sweep
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -enddate

Use -servername whenever name-based virtual hosts or a TLS terminator is in front. Without SNI you may inspect the default cert, not the one users hit.

Common Failure Modes & Symptoms

What you seeLikely causeFirst checks
Browser NET::ERR_CERT_DATE_INVALIDExpired or not-yet-valid leaf-dates; load balancer vs origin cert
Name mismatchSAN does not include the URL host-ext subjectAltName vs the name in the URL
Some clients fail, browsers workIncomplete chain; browsers have cached intermediatesCount certs in s_client -showcerts
Works with curl, fails in Java/PythonDifferent trust storeApp truststore vs /etc/pki / /etc/ssl
Handshake alert / protocol versionTLS 1.0/1.1 disabled or cipher floors_client -tls1_2 / -tls1_3; nginx/apache ssl protocols
Connection resets at the LBLB health check uses HTTP or wrong SNITarget group cert, backend protocol
”certificate verify failed” after renewalNew cert, old key or stale file on diskmodulus match; which path the unit actually loads
Intermittent bad certMultiple backends, only one updatedHit each backend IP with s_client

Investigation Tips

  • Always pin the name you are testing (-servername and the URL host). Wildcards and SAN lists surprise people.
  • Test from a client outside the box and from localhost. Different paths (LB vs origin) serve different certs.
  • If chain depth is 1, the server is sending only the leaf. Bundle leaf + intermediates in the file the daemon loads.
  • After renewal, confirm the running process was reloaded and that it points at the new files. lsof the pid if needed.
  • Compare notAfter on disk, on the load balancer, and on what s_client shows. They diverge more often than they should.
  • Corporate TLS inspection MITM will fail clients that pin CAs or use their own bundle. Treat that as a trust-store problem, not an app bug.
  • Automate expiry alerts at 30/14/7 days; human calendars are not a control.

Personal Lessons Learned

  • A renewal “succeeded” in the ACME log and failed in the browser because nginx was still serving the old bundle. The new files were on disk; the worker had not been reloaded. Compare notAfter from s_client with notAfter on the file the unit actually opens (lsof).
  • One backend of four still had last year’s cert. The LB health check was HTTP, so it stayed in rotation. Hit each backend IP with s_client -servername before you call the rollout done.
  • Java failed with PKIX path building failed while curl was fine. Curl used the system bundle; the app used a private truststore that never got the new intermediate. “Works in curl” is not “works in the process”.

Graph View

Backlinks