First impressions of property testing with StreamData

Yesterday Andrea Leopardi released StreamData, a data generation and property testing library for Elixir.

In 2015 at Erlang Factory Light in Berlin I attended a talk by Thomas Arts about property testing and it’s benefits, so I was already interested and gave StreamData a little test run today.

I took the PryIn client library and replaced some tests for which I thought a property testing based approach would make sense.

After writing my first test and trying to run it with mix test, I got an error message:

** (ArgumentError) application :stream_data is not loaded, or the configuration parameter :initial_size is not set

This was easily fixed by adding the following line to my test_helper.exs:

{:ok, _} = Application.ensure_all_started(:stream_data)

The reason this was necessary is that the PryIn client still defines an applications list in it’s mix.exs to support Elixir versions before 1.4. And because :stream_data is only a test dependency I did not want to add it to that list.

The next little problem I ran into was that some of my tests rely on ExUnit’s setup callback to reset some global state. With StreamData, setup still gets called, but only once before the whole property definition, not once for each check. To fix that, I just needed to call the reset code as part of my checks. Not a big problem, but something to be aware of.

Writing property tests was quite easy then. I mainly changed tests that were using constant values before to now use generated data. Where before for example I had something like this:

@env %Macro.Env{ # <== constant test values
  module: PryIn.TestController,
  function: {:"custom_instrumentation_action", 2},
  file: "test/support/test_controller.ex",
  line: 123
}

test "custom instrumentation within a running trace" do
  PryIn.CustomTrace.start(group: "test", key: "test")
  data = CustomInstrumentation.start("expensive_api_call", @env)
  CustomInstrumentation.finish(5000, data)
  PryIn.CustomTrace.finish()

  [interaction] = InteractionStore.get_state.finished_interactions
  [custom_instrumentation] = interaction.custom_metrics

  assert custom_instrumentation.offset > 0
  assert custom_instrumentation.duration == 5
  assert custom_instrumentation.key == "expensive_api_call"
  assert custom_instrumentation.file =~ "test/support/test_controller.ex"
  assert custom_instrumentation.function == "custom_instrumentation_action/2"
  assert custom_instrumentation.module == "PryIn.TestController"
  assert custom_instrumentation.line == 123
  assert custom_instrumentation.pid == inspect(self())
end

Now I have this:

property "custom instrumentation within a running trace adds a custom metric" do
  check all module_name <- PropertyHelpers.string(),
    function_name <- PropertyHelpers.string(),
    function_arity <- int(0..255),
    file <- PropertyHelpers.string(),
    line <- PropertyHelpers.positive_int(),
    key <- PropertyHelpers.string(),
    duration <- int(0..10_000) do

    env = %Macro.Env{ # <== dynamic test values
      module: String.to_atom(module_name),
      function: {String.to_atom(function_name), function_arity},
      file: file,
      line: line
    }

    InteractionStore.reset_state() # <== reset state manually
    PryIn.CustomTrace.start(group: "test", key: "test")
    data = CustomInstrumentation.start(key, env)
    CustomInstrumentation.finish(duration, data)
    PryIn.CustomTrace.finish()

    [interaction] = InteractionStore.get_state.finished_interactions
    [custom_instrumentation] = interaction.custom_metrics

    assert custom_instrumentation.offset > 0
    assert custom_instrumentation.duration == trunc(duration / 1000)
    assert custom_instrumentation.key == key
    assert custom_instrumentation.file =~ file
    assert custom_instrumentation.function == "#{function_name}/#{function_arity}"
    assert custom_instrumentation.module == inspect(env.module)
    assert custom_instrumentation.line == line
    assert custom_instrumentation.pid == inspect(self())
  end
end

You can see how I replaced the constant @env module attribute with dynamically generated values.

I also created a little helper module to hold some custom generators:

defmodule PryIn.PropertyHelpers do

  def string() do
    StreamData.alphanumeric_string()
  end

  def to_stringable() do
    StreamData.one_of([
      string(),
      StreamData.int(),
      StreamData.unquoted_atom(),
      StreamData.boolean(),
      StreamData.uniform_float()
    ])
  end

  def positive_int() do
    StreamData.map(StreamData.int(), &abs/1)
  end

  def non_empty_string() do
    StreamData.filter(string(), & &1 != "", 100)
  end
end

I was actually trying to come up with a generator for all possible valid strings but that turned out to be quite difficult. For now I’m sticking to only testing with alphanumeric strings, which StreamData provides a generator for.

So far no new bugs were revealed, but I quite like this way of writing tests. Besides from possibly surfacing problems, it also just frees me of having to come up with artificial values.

If you want to check out my new tests, you can find them in the stream_data_property_testing branch on Github.