Chapter 14: Why Use a Survey
Common Objections to Survey-Based Research
Two most common questions about the research:
- Why use surveys (vs. system-generated data)?
- Can you trust survey data?
Types of Data Collection
System-Generated Data
Data generated by tools (logs, metrics, monitoring systems). Advantages:
- Precise
- Objective
- Not subject to response bias
Limitations:
- Only captures what the systems are designed to measure
- May not reflect human experience or organizational dynamics
- Different organizations measure different things in different ways
- Can’t capture culture, satisfaction, identity, or burnout
Survey Data
Data collected via questionnaires from humans. With proper methodology, it can be as rigorous and trustworthy as system-generated data.
Why Surveys Are Appropriate Here
Scale: The research collected data from 2,000+ organizations in four years. Getting system-generated data from thousands of organizations across different technology stacks, CI tools, deployment pipelines, and monitoring systems is impractical.
Breadth: The research needs to measure things that can only be measured through human perception:
- Organizational culture
- Job satisfaction and burnout
- Identity and engagement
- Leadership behaviors
Cross-industry validity: System data from Netflix looks nothing like system data from a government agency. Survey responses about lead time (e.g., “between one hour and one day”) can be compared across contexts.
Snowball Sampling
The data collection method was snowball sampling: send survey invitations to mailing lists and social media, encourage respondents to invite friends and peers.
This is appropriate because:
- The target population (technology professionals familiar with DevOps) is hard to enumerate exhaustively
- Network effects help reach practitioners who wouldn’t see random advertisements
- Discussed in detail in Chapter 15 — the method is appropriate for the research goals
Making Survey Data Trustworthy
Several steps were taken to ensure data quality:
- Likert-type questions instead of yes/no — richer data, less susceptible to binary interpretation
- Psychometric validation — discriminant validity, convergent validity, reliability testing (see Ch. 13)
- Theory-driven hypotheses — stated before analysis, preventing data fishing
- Replication across years — same findings across 2014, 2015, 2016, 2017 increase confidence
- Peer review — results published in academic peer-reviewed journals
- Large sample — 23,000+ responses provides statistical power
Survey Data in Your Own Organization
The research demonstrates a broader principle: well-designed survey instruments can measure things that system data cannot. Organizations can use these same validated survey instruments internally to:
- Measure their Westrum culture score
- Measure employee identity, satisfaction, burnout risk
- Assess leadership quality
- Track changes over time as practices improve
The chapter encourages practitioners to view surveys not as inferior to system data, but as the right tool for measuring human and organizational factors.