Five Splunk environments into one

The situation

Splunk had arrived at the company more than once. Each product line stood up its own deployment, and by the time I picked this up we were running separate environments side by side, plus a government-cloud environment with its own rules.

The duplication was the expensive part. The same reporting app existed in two environments under the same name, maintained separately, drifting apart. Users needed to know which environment held their data before they could go looking for it, and access requests routinely landed on the wrong one. Every environment carried its own indexers, its own search heads, its own bill.

The brief was to merge them into one — internally we called the target Global OSS — across all five regions we operated in.

How I scoped it

The first real decision was the one that made the rest possible: migrate configuration, not data.

Moving years of indexed data between environments would have been the slowest, riskiest part of the project, and it would have bought almost nothing. Instead the old environments stayed live and searchable until their retention windows expired on their own, then got decommissioned. New data went to the new environment from cutover onward. Nobody lost access to anything; the old estate just aged out.

The per-environment work broke into two halves:

• Lift and shift — verify and migrate the product configurations, audit what was actually deployed on the deployment server, move log sources and dashboards across, revalidate every forwarder’s outputs config to point at the new indexers, then validate data and dashboards on the far side.
• Operationalising the service — user migration, review and demo sessions with each product team, and working with product engineering to define what they actually needed out of Splunk rather than assuming the old setup was right.

Deciding what not to bring

A migration is a rare licence to throw things away, and I wanted to use it. Lifting everything verbatim would have moved the mess rather than fixing it.

• Disabled vendor apps didn’t come. If it had been switched off long enough that nobody noticed, it wasn’t load-bearing.
• Custom apps were archived, not deployed. They went onto the deployment server so nothing was lost, but weren’t pushed anywhere until someone asked for them. Most nobody did.
• Users started from a blank slate. Access was rebuilt persona-based rather than migrated account by account, and we moved people over case by case as they came to us. This was the unpopular call and the right one — the old permission sprawl was half the reason for the project.
• Duplicated apps got merged rather than carried across as two variants of the same thing with different suffixes.

Cutting over without losing data

The pattern I used everywhere, and the reason nothing got lost: write to both destinations, validate, then remove the old one. Never repoint and hope.

For the AWS log sources that meant an explicit sequence — stand up the new bucket in the target account, configure the log services to deliver to both old and new, confirm both were receiving, repoint Splunk at the new one, and only then start removing the old log groups and the redundant streaming config. The last step was always a sanity check across every source, account, and region rather than a spot check on one.

That work also let me take out a cost problem on the way past. Logs were being routed through a managed streaming service that we didn’t have the in-house expertise to run efficiently, and the quotes for doing it properly were eye-watering. Going direct to object storage and having Splunk collect from there did the same job for a fraction of it.

Making it stay fixed

Consolidating once is worth very little if the estate drifts apart again. The build was split deliberately:

• Terraform for the infrastructure.
• Ansible for everything Splunk — installs, app deploys and updates from an apps repository, and config management down to authentication and service limits. Adding an app to a deployment meant naming its folder in the install playbook, not logging into a server.

The test of whether that worked came when we rebuilt the engineering environment. It was the same playbooks with a different inventory file. That is the whole point.

Where it landed

• One environment in place of five, across six-plus regions with multi-site indexer clustering and 100+ Splunk servers between me and one other engineer.
• About two-thirds of those servers are managed through Ansible. The remaining third sits in a FedRAMP-scoped environment and is still handled manually — the automation story is real but it isn’t total, and pretending otherwise would misrepresent the job.
• Access issues dropped because there was no longer a wrong environment to be in.
• Infrastructure and server costs came down as the old estate was decommissioned.
• I was promoted to Principal off the back of it.

What’s happened since

The part I didn’t anticipate is how much the consolidation would end up being infrastructure for things that hadn’t been proposed yet. Having one place for logs to go turned out to be worth more than the cost saving that justified the project.

• It went multi-cloud. We stood up accounts in Azure and in Oracle Cloud Infrastructure, and both log into the same platform. That was never in the original scope. It was possible because the target of a new log source is one environment rather than a question about which of four it belongs to.
• More regions. The footprint has grown past the five we built out originally, and adding one is now a known quantity rather than a project.
• Another environment folded in. A smaller product environment was migrated across well after the main work finished — same config-first, dual-delivery approach, at a fraction of the effort the first ones took.

That last one is the real measure of whether this worked. The first migration was a multi-quarter project with a team. The most recent one was a normal piece of work.

What I’d do differently

I’d have pushed harder, earlier, on defining what “done” meant per environment. We had a good task breakdown and a clear lift-and-shift checklist, but acceptance criteria arrived later than they should have, and the fuzziness cost us in the middle of the project. I ask that question first on consulting engagements and I should have asked it as loudly internally.

I’d also have written the runbooks concurrently rather than at the end. Doing the first region teaches you nearly everything about the second, and I re-derived more than I needed to.

This write-up describes the shape of the work rather than the specifics. Environment and host names, index names, account identifiers, vendor and cost figures, and everyone I worked with are left out deliberately — that detail belongs to my employer, not on my portfolio.