Bounded Black Box Testing

Bounded black-box crash testing (B3) is a new approach to testing file-system crash consistency. B3 is a black-box testing approach which requires no modification to file-system code. B3 exhaustively generates and tests workloads in a bounded space. We implement B3 by building two tools - CrashMonkey and Ace.

• Fully automated
• File-system agnostic
• No modification to file-system code

CrashMonkey

CrashMonkey is a file-system agnostic record-replay-and-test framework. CrashMonkey simulates crashes at specific points along the workload, and automatically tests for data and metadata consistency of persisted files. CrashMonkey works in three phases.

• Records block IO operations during workload execution
• Replays recorded IO until persistence point
• Tests consistency of persisted files and directories

Automatic Crash Explorer (ACE)

Ace is an automatic workload generator, that exhaustively generates workloads within a given bound. Bounds include the number of file-system operations in the workload, parameters to these operations and the initial size and state of the file system. ACE operates in 4 phases.

• Pick the skleton for the workload
• Fill in arguments to each file-system operation
• Add persistence operations to the workload
• Satisfy dependencies to build a POSIX complaint workload

1

CrashMonkey and ACE: Systematically Testing File-System Crash Consistency [ TOS '19 ]

[ pdf ]

2

Finding Crash-Consistency Bugs with Bounded Black-Box Crash Testing [ OSDI '18 ]

[ pdf ] [ slides ] [ bibtex ] [ blog ] [ demo video]

3

CrashMonkey: A Framework to Automatically Test File-System Crash Consistency [ HotStorage '17 ]

[ pdf ] [ slides ] [ bibtex ]