RFuzz In Theory (a scientific way to destroy stuff)

RFuzz uses the concept of fuzzing apparently developed at UoMW by B.P. Miller as a way to measure stability in systems (and just break stuff I imagine). With fuzzing BPM and friends are able to find quite a huge number of flaws in operating systems with just a small set of tests.

How Fuzzing Works

The idea behind fuzzing (according to me, not BPM) is that by passing large amounts of random inputs to a system, you expand the range of test input values outside those expected by the developer when they wrote the system. It’s kind of a perversion of “garbage in, garbage out”. Instead with fuzzing you are saying, “garbage in, lots of crashing”.

Fuzzing can typically find errors in logic, buffer overflows, algorithm exploits, and unexpected crashes on input. The main advantage of fuzzing is that you don’t plan the input. You just turn the flood gates on, fix whatever comes up as an error, and then repeat until your system handles the garbage well.

Measurement And Sampling

Fuzzing is related to measurement and sampling theory. Put simply, if you attempt to measure something for the purpose of statistical analysis then you need to use random samples. Using random sampling techniques removes human bias and decision from the tests making them more reliable.

This relationship between fuzzing and sampling means that it’s fairly simple to generate quality metrics from your fuzzing sessions. Since a fuzzing session assumes little knowledge of the system, and is only throwing randomness at it, you can take the number of defects found as a quality metric. Combine this with time and you have a good correlation similar to Mean Time Between Failure (MTBF) that measures how long it takes to cause a defect with fuzzing.

Intellectual Honesty

Another view of fuzzing is that fuzzing is the inverse of Test Driven Development (but not an alternative, you need all the help you can get so use both). With TDD you are writing tests for what you know could be defects. Your tests ensure that future changes and the current code all work as you expected.

Fuzzing however uses injection of random inputs to find unexpected defects in the current system. These unexpected defects are important since they are the ones that an attacker will typically exploit due to their origin being from an input value.

A deadly combination is having TDD tests to validate your system, and fuzzing tests to keep you on your toes. When fuzzing finds a defect you didn’t know about, fix it and then augment your TDD suites to cover it for the future.

This practice of assuming that your test suites don’t cover all possible defects instills intellectual honesty in your quality practices. You go from saying, “Wow I have lots of tests, this thing is solid.” And move to, “Tests found X defects, let’s see what fuzzing finds…” You go from rhetoric to measurement.

Gaining Depth With Randomness

A major criticism of fuzzing is that it only finds “shallow bugs”. This is actually true if the fuzzing is purely random input without any knowledge of the system. Fuzzing will only find defects that are on the surface of the system nearest the input points.

RFuzz tries to improve this by being written in Ruby and having a decent DSL (Domain Specific Language) to let you write smarter fuzzing tests. You can start with simplistic random fuzzing, then move to smarter mixtures of fuzzing and traditional TDD test suites.