Production your tests

WIFM: I’ll learn how to write more realistic tests with conditions and input that resemble the production environment. By knowing this I’ll deploy code with fewer bugs

TL&DR: Why traditional testing falls short in catching many production bugs and how modern techniques that are based on production behaviour can save your day

• Intro (10min)
  • Problem: When traditional testing falls short
  • New approach: The 3 different faces of ‘testing in production’ means
  • Principle: Shift left first (@Yael)
• Solutions(40 min)
  • Testing in production as the last safety net (5min)
    • Production E2E tests
  • Testing before production (10min)
    • Traffic shadowing
  • Production your tests(25)
    • Fuzzing
    • Property-based testing #java-example (@dana – Tell me more about your use case)
    • Production replay

TOC

Chapter 1: When traditional testing falls short or the epic story on the poor developer who brought the internet down

This talk starts with the epic story on CloudFlare’s downtime which took 10M (!) website down. No traditional testing technique could prevent this bug, this will make the case for testing in/with production

⏰ 8 min

Concepts: Testing in production, shift right

Demos: None

Chapter 2: Testing over the holy production itself and the story about the E2E tests that saved my day  

Why testing over the real production itself is valuable and can be safe enough. This includes a demo on running E2E tests over the production itself

⏰ 7 min

Concepts: MTTR, UX monitoring

Demos: Production E2E

Chapter 3: Deployment testing or what can we learn from military units that fire missiles

How to tell if the next version is safe over real-world production conditions with minimal risk. This includes understanding the fundamental term ‘blast radius’, a brief on canary deployment and a rich demo of traffic shadowing

⏰ 10 min

Demo: Traffic shadowing

Concepts: Blast radius, canary

Demos: Traffic shadowing

Chapter 4: Production your tests or the monkey who types William Shakespeare’s script using a typewriter

Writing realistic tests upfront to discover issues even earlier. This section shows multiple techniques to spice-up our tests during development and make them resemble production. The following concepts and techniques will be presented: shift left, fuzz testing, property-based testing and a rich demo of production-replay

⏰ 25 min

Demo: Testing with production data

Concepts: Shift left, fuzzing, test canons

Demos: Property-based testing, production replay