Run distributed Splunk
Multi-site indexer clusters, search head clusters, deployers, deployment servers, heavy and universal forwarders, license managers. Installs, upgrades, capacity, retention, and the 2 a.m. version of all of it.
Alec Vogelsang
Principal Cloud Operations Engineer · Splunk & Observability
I've worn a lot of hats across 10+ years in enterprise IT, from operations analyst to engineering consultant. I got into Splunk doing security incident work, spent a year and a half consulting on it — a different client every few weeks — and now run logging and observability in-house for a software company's cloud operations org.
The work I'm proudest of is the unglamorous kind: taking five separate Splunk environments and making them one, then handing most of it to Ansible so it stays that way. I like the problems where the answer is a runbook someone else can follow.
5+
Splunk environments consolidated into one
100+
Splunk servers administered
6+
Regions across AWS, Azure and Oracle Cloud
99.7%
Availability target for the Splunk estate
What I do
Three things, mostly
Get data in, and make it mean something
Onboarding across AWS, Azure, Oracle Cloud, Akamai, CrowdStrike, Qualys, Active Directory and a long tail of others — then the parsing, CIM mapping, dashboards, and alerts that turn it into something a team will actually act on.
Automate it so it stays fixed
Terraform for the infrastructure, Ansible for everything Splunk — installs, app deploys, config management. Rebuilding an environment should be the same playbooks with a different inventory file.
Selected work
Four things worth writing up
These are written from my own notes. Specifics that belong to my employer — hostnames, index names, findings, anyone's name but mine — aren't here, and each page says what it left out.
Five Splunk environments into one
Separate deployments had grown up around each product line, with duplicated apps, inconsistent access, and a server bill for each. I owned the lift-and-shift into a single Ansible-managed environment — and finished it after my one teammate left. It's since become the place AWS, Azure, and Oracle Cloud all send their logs.
Read the write-up →
Turning compliance controls into alerts that page someone
A government-cloud environment had its vulnerability posture sitting in a scanner's dashboards that nobody checked on a schedule. I rebuilt them in Splunk, then went past replication: port baselining, weak-protocol detection, scan-freshness thresholds, all opening tickets automatically.
Read the write-up →
Two root causes wearing one error message
Clients hitting the management port got SSL verification failures. It turned out to be two independent problems that looked like one, and my first fix made the error change rather than go away — which is the part of this story I find most useful.
Read the write-up →
A dozen client deployments, and the question I opened every one with
Splunk Professional Services across ticketing SaaS, paper manufacturing, title insurance, rail telematics, county government and healthcare. The engagements varied; the first question never did, and it's the reason most of them landed.
Read the write-up →
Experience
Where I've done it
Apr 2021 – present
Principal since Dec 2023
Principal since Dec 2023
Principal Cloud Operations Engineer
Deltek · Herndon, VA (remote)
- SME for observability and site reliability through Splunk.
- Deploy and administer every component of a multi-site distributed Splunk estate — indexer and search head clusters, deployers, deployment servers, forwarders, license managers, network config.
- Delivered the consolidation of five-plus product-line environments into one centralised deployment, now taking logs from AWS, Azure, and Oracle Cloud across six-plus regions.
- Design and operate the estate against a 99.7% availability target, scaling it alongside business growth.
- Build dashboards, alerts, and reports for delivery, customer care, security, and operations teams.
- Run bi-weekly training and office hours to get people using Splunk themselves rather than filing tickets for it.
- Own upgrades end to end: research, test, change control, then rollout via CLI and Ansible, often in response to a vulnerability.
Jan 2020 – Apr 2021
Splunk Professional Services Engineer
SP6 / Aditum
- Implemented Splunk deployments for a dozen-plus clients against their requirements and Splunk PS best practice.
- Built custom apps and dashboards, including CIM-compliant data models for security use cases.
- Solved problems across hybrid cloud, full cloud, and on-premises deployments — a different environment every few weeks.
- Requirements gathering, cookbooks and runbooks, product testing, and the handover documentation that made engagements stick.
Jul 2017 – Jan 2020
Senior from Oct 2018
Senior from Oct 2018
Technology Architecture Senior Delivery Analyst
Accenture · Greater Sacramento
- Administered and monitored Splunk Enterprise on Linux via GUI and CLI; configured apps and add-ons across a clustered deployment.
- Integrated Splunk with ServiceNow, AWS, Palo Alto Networks, and Windows/Linux estates via REST and web APIs.
- Built data investigations using machine learning, statistical analysis, and narrative analysis.
- Created and deployed machine images using Chef, Terraform, and CloudFront.
- Earlier, as Delivery Analyst: first-level monitoring and incident triage, UNIX and SQL scripting for reporting and metric collection, security incident investigation against client SLAs, and dashboards in both Splunk and Kibana.
Aug 2016 – Jul 2017
Technical Analyst
BCforward · Greater Sacramento
- Technical support and monitoring for a state healthcare web portal — the first job where uptime was somebody's actual problem and that somebody was partly me.
- Oracle tooling end to end: Enterprise Manager Cloud Control, Real User Experience Insight, Service Bus, API Gateway, BI Publisher.
- Wrote and modified Python and Bash jobs; troubleshot using RUEI, AWR reports, and Splunk.
Oct 2015 – Aug 2016
Project Manager / All-Team Lead
Global TIES · UC San Diego
- Led a multidisciplinary team of 12 student engineers designing a solar-powered pathway lamp from bamboo and renewable materials, for a social enterprise with a Philippines-based NGO.
- Got manufacturing cost under $5 a unit with a 48-hour battery, then travelled to Bulacan to deploy prototypes for field testing.
- Ran the meetings, the fundraising, and the pitches — including at the USD Social Innovation Challenge.
2012 – 2016
B.S. NanoEngineering
UC San Diego
- Focus in mechanical engineering, with a good share of the major-specific coursework at graduate level. Multi-disciplinary by design — a bit of every branch of engineering.
- Worked IT through most of it: desktop support for the university, IT analyst at Scripps Institution of Oceanography, and IT assistant at the health system. Helpdesk, imaging, Active Directory, and the unglamorous end of keeping other people working.
- Also spent a summer as a research assistant in the health sciences department, on HIV prevention research — literature reviews, publication databases, and manuscript preparation.
- Got recruited into IT on graduating and followed that path. See Learning for how I've filled in the computer-science side since.
Skills
What I actually work with
Splunk
- Multi-site indexer clustering
- Search head clustering & deployers
- Deployment server / forwarder management
- HTTP Event Collector
- Enterprise Security (ES)
- Data models & CIM mapping
- SPL, base searches, acceleration
- Index & retention management
- Custom apps and add-ons
Cloud & infrastructure
- AWS (EC2, S3, VPC, CloudWatch, CloudTrail, GuardDuty, Config)
- Linux administration & performance tuning
- Terraform
- Capacity planning
- TLS / PKI & certificate management
- syslog (rsyslog, syslog-ng)
Automation
- Ansible (playbooks, inventories, roles)
- Jenkins
- BigFix
- Shell scripting
- Python (scripting, not software engineering — see below)
- Git
Observability & security
- AppDynamics
- SIEM & security operations
- Qualys (vulnerability + file integrity)
- CrowdStrike, Duo, Centrify, Palo Alto
- ServiceNow integration
- FedRAMP / ITAR continuous monitoring
Data I've onboarded
Getting a source into Splunk is rarely the hard part. Working out what its events actually mean, normalising them so they'll join to everything else, and keeping the volume affordable — that's the job. This is what I've done it for.
Cloud & platform
Security & identity
Applications & databases
Pipeline, ITSM & delivery
Where I'd want help
I write Python and shell to get a job done — onboarding scripts, metric collection, installers — but I wouldn't call myself a software engineer, and I'd rather say so than find out in week two. Same with Prometheus and Grafana: I know what they're for and I haven't run them in anger. Both are on my list.
Certifications & learning
Filling in the parts my degree didn't cover
Splunk Core Certified Consultant
recertified May 2026
Splunk Enterprise Certified Architect
2023
Splunk Enterprise Certified Admin
2023
Splunk Enterprise Security (ES) Accredited
—
I majored in NanoEngineering, so I didn't get a formal education in information technology. What I did instead was go back and take the coursework — computer networking and computer science fundamentals — because the gaps were real and I'd rather close them than talk around them.
Networking. The OSI model, addressing and subnetting, the TCP/IP suite and the protocols on top of it, plus laying out network architecture in Visio. Three midterms and a final. The OSI model in particular has earned its keep — it's still how I break down a "Splunk is broken" ticket, layer by layer, until the problem has nowhere left to hide.
Computer science fundamentals. Seventeen chapters of Computer Science Illuminated: number systems and data representation, gates and circuits, computing components, low-level languages and pseudocode, algorithms, operating systems, file systems, information systems, networks, the web, and computer security.
Security. Separate self-study on cybersecurity fundamentals alongside the day job in SecOps.
Outside work
Things I build when nobody's asking me to
Most of these started because I wanted the thing to exist. A few of them taught me more than work did that year.
VolleyGram
A volleyball rotation tool. Diagrams all six rotations for 4-2, 5-1 and 6-2 systems across base, serve-receive and defensive formations, drag-and-drop, with overlap-legality checking against the FIVB rule at the moment of serve. Share links encode the entire diagram in the URL, so there's no server and nothing to keep running.
A smart home that reports on itself
A HomeKit ecosystem tying the house together — lighting, the garage door, climate and weather alerting, and perimeter monitoring. It's the same instinct as the day job at a much smaller scale: instrument the thing, then let it tell you when something changed.
Photography & video
Portraits and pets mostly, with a portfolio site I built to go with it — static, on Cloudflare Pages, with an image pipeline that strips EXIF and GPS at build time. I also shoot and cut volleyball footage, which is where the scriptwriting habit comes from.
Raspberry Pi LED matrix
Took a Python project off GitHub and modified it until it did what I wanted. Small, but it's how I got comfortable reading someone else's code instead of starting from scratch.
The other thing I do
I coach people through interviews — friends, colleagues, anyone who asks. I keep notes for it the same way I keep notes for work. It's the same muscle as the office hours I run for Splunk: the job isn't finished when the thing works, it's finished when somebody else can do it too.
Contact
Get in touch
If you've got a Splunk estate that's grown in four directions, or you need someone who can own logging and observability end to end, I'd like to hear about it. Happy to talk through anything on this site in more detail.
Or skip the form — smartalecv@gmail.com · LinkedIn