Configuration testing for behaviour that holds when the settings change
A release can be correct and still fail because production is set up differently from where it was tested. We inventory the settings that change behaviour, test the combinations that carry risk, and make environment differences visible and deliberate.
Overview
What it is. Configuration testing verifies that software behaves correctly under the different configurations it will meet: environment settings, feature flags, tenant options, regional settings, and supported software and hardware combinations.
Who it is for. Teams deploying to several environments, products with feature flags or tenant-level options, platforms installed at customer sites, and organisations whose releases often behave differently in production.
Why it matters
The problem it addresses. Configuration is code that nobody reviews like code. Small differences between environments or between customers cause failures that are hard to reproduce and are often blamed on the wrong thing.
When it matters most. Before a release moves environments, when new flags or options are added, when onboarding a customer with unusual settings, and after infrastructure changes.
What happens without it. Issues appear only in one environment or for one customer, take days to trace, and sometimes expose data or functions that were meant to be off.
What we test
Configuration inventory
settings, flags, secrets (by name, not value), tenant options and their allowed values
Environment comparison
differences between development, test, staging and production
Feature-flag combination testing
Tenant and customer-specific configuration checks for multi-tenant products
Installation and upgrade behaviour across supported software and hardware combinations
Missing, invalid and expired configuration handling
does the system fail safely and clearly
Configuration-related security checks, such as exposed defaults and debug settings
How UpDoer approaches Configuration Testing
Understand
Requirements, tickets, code diffs and product context.
We collect environment descriptions, configuration files and flag lists, and ask where behaviour has differed between environments before.
Risk Mapping
Where a failure would cost the most.
Settings are classified by risk: those that affect money, access, data and availability come first, and combinations that are known to interact are listed.
Test Strategy
Coverage chosen, test cases drafted.
We choose which configurations to test fully, which to sample and which to document, using combination-reduction techniques where the space is large.
Execute
Manual and automated testing runs.
Tests run against each chosen configuration, including deliberately wrong values to see whether the system fails safely and clearly.
AI-Assisted Quality Engineering
Failures classified, release risk scored.
AI compares configurations across environments, flags drift, and suggests a reduced set of combinations that still covers the interactions that matter.
Human Verification
A QA engineer checks every flagged result.
A QA engineer judges which differences are intentional, confirms which combinations carry business risk, and checks failure messages make sense to a user or operator.
Report & Improve
The verified quality signal, then better coverage.
We report the configuration inventory, the combinations tested, the differences found and the ones accepted. The inventory becomes a living reference.
Configuration: same code, different settings
01
Where it breaks
Works in staging onlyFlag combinations untestedMissing setting, vague error02
Test workflow
EnvironmentConfigDeployVerify03
AI assists
Detect drift between setupsReduce combinationsDraft negative cases04
Humans decide
Which differences are intended?Which combinations matter?05
Release signal
Config inventoryDifferences accepted or fixedGo / no-go by a person
Illustrative workflow. Not a client result or a measured outcome.
Configuration coverage matrix
A dash means no AI step is described for that area.
| Configuration area | What we validate | AI assistance | Human validation |
|---|---|---|---|
| Environment settings | Behaviour identical where intended; differences listed | — | Decides which differences are deliberate |
| Feature flags | Risky flag combinations behave correctly | Proposes a reduced combination set | Chooses which combinations carry business risk |
| Tenant and customer options | Settings isolate correctly between tenants | Suggests option combinations from data | Judges isolation and billing impact |
| Missing or invalid values | System fails safely with a useful message | Drafts negative cases from the inventory | Reviews messages for clarity |
Configuration area: Environment settings
- WHAT WE VALIDATE
- Behaviour identical where intended; differences listed
- AI ASSISTANCE
- —
- HUMAN VALIDATION
- Decides which differences are deliberate
Configuration area: Feature flags
- WHAT WE VALIDATE
- Risky flag combinations behave correctly
- AI ASSISTANCE
- Proposes a reduced combination set
- HUMAN VALIDATION
- Chooses which combinations carry business risk
Configuration area: Tenant and customer options
- WHAT WE VALIDATE
- Settings isolate correctly between tenants
- AI ASSISTANCE
- Suggests option combinations from data
- HUMAN VALIDATION
- Judges isolation and billing impact
Configuration area: Missing or invalid values
- WHAT WE VALIDATE
- System fails safely with a useful message
- AI ASSISTANCE
- Drafts negative cases from the inventory
- HUMAN VALIDATION
- Reviews messages for clarity
Where AI helps
- Suggesting a reduced set of configuration combinations that still exercises the interactions most likely to cause failures.
- Drafting test cases for missing and invalid values from the settings inventory.
- Grouping failures that share a configuration cause so they are investigated once.
- Summarising which settings differ between a failing and a passing environment.
How we use AI: AI can accelerate test design, analysis and repetitive QA work, but important outputs remain subject to human review and validation. It informs the release decision; it does not make it.
Where humans matter
- Deciding which differences between environments are intentional and which are mistakes.
- Choosing which combinations matter to the business and accepting the risk of the rest.
- Judging whether a failure message helps a user or operator recover.
- Handling secrets sensibly: testing that they are required and protected without ever handling production values.
- Interpreting results when two settings interact in a way nobody predicted.
Tools we build with
- Automation
- SeleniumPlaywrightAppiumKatalonTestComplete
- API & Performance
- PostmanSoapUISwaggerApache JMeter
- CI/CD & Reporting
- JenkinsTeamCityAllureMaven
Related services
Relevant industries
Frequently asked questions
How do you handle huge numbers of combinations?
We test the ones that carry risk and use combination-reduction techniques for the rest, then state the combinations that were not tested.
Is this the same as compatibility testing?
They overlap. Compatibility asks whether the product works on the browsers, devices and operating systems users have. Configuration asks whether it works across settings, environments and options. Many projects need both.
Can this run in our pipeline?
Environment comparison and key combination tests can run automatically, with failures reviewed by a QA engineer. The depth depends on your tooling.