Journal: Property-Based Testing with PropEr, Erlang, and Elixir

Community Journals & Book Clubs
,
22

Erlang
(page 112)
test/prop_generators.erl

To view the results of generating paths, it is very convenient to use the function io:format/2.

[left,up,up,up]
.[right,up,left,up,left,up,left,down,down,right,down,left,left]
.[down,down,left]
...
.[down,left,down,down,left,down,down]
.[down]
.[up,left,down,down,down,right,up,right,up]
.[right,right,down,down,down,down,down,left,down,down,right,up,right,down,down,
 down]

OK: Passed 100 test(s).
===>
1/1 properties passed

Property with generators into rebar3 project.

2 Likes
23

Erlang
(page 118)
test/prop_generators.erl
Symbolic Calls

For a more informative view of the behavior of properties and symbolic calls, I would very much like to be able to view intermediate results. For this, a functions calls io:format were added to the property, as well as the if-expression:

prop_dict_autosymb() ->
    ?FORALL(D,
            dict_autosymb(),
            begin
                %io:format("~p~n", [D]),
                Result = dict:size(D) < 5,
                if Result ->
                       io:format("~p~n", [D]),
                       true;
                   true ->
                       false
                end
            end).

Yes, of course, when using symbolic calls, informational messages are a little more clear.

(10 time(s))                                                                                  
{'$call',dict,store,[0,0,{'$call',dict,store,[1,0,{'$call',dict,store,[-1,0,{'$call',dict,stor
e,[2,0,{'$call',dict,store,[-2,0,{'$call',dict,new,[]}]}]}]}]}]}                              
===>                                                                                          
0/1 properties passed, 1 failed                                                               
===> Failed test cases:                                                                       
prop_generators:prop_dict_autosymb() -> false

rebar3 project.

2 Likes
24

Erlang
(page 118)
test/prop_generators.erl
Question 4
Be careful when checking the tree generator. If you just run it, you will not see anything, since trees will be created endlessly and only a forced interruption of the process will stop the computation by Ctrl-C keys for instance.

% Do not run it - it is infinite tree maker.
% For illustration only.
prop_tree() ->
    ?FORALL(Type, tree(),
        begin
            io:format("~p~n",[Type]),
			true
        end).

Using a macro ?LAZY provides some probability of stopping the creation of trees, but this solution does not work regularly and at the 2nd or 3rd run of the property, the system may freeze, since trees will still be created and created. You can also terminate this by forcibly interrupting the test.

Using a macro ?SIZED solves all the previous problems.

It is very convenient to add limiting generator without parameters limited_tree().

I also noticed that it would be convenient and logical to combine the sequential call of limited_tree() generators into one using fixed_list([Gen]) generator.

To reduce informational noise and better see the resulting structures, you can use the boolean() generator as a content for limited_tree() generator.

limited_tree() ->
    limited_tree(boolean()).

rebar3 project

2 Likes
25

Erlang
(page 125
test/prop_generators.erl
Question 5
Note that in stamps1 generator no sorting by content occurs when comparing tuples. I tried to correct the lack of this check and wrote a check property on the basis of this.

%% Return two ordered timestamps
stamps1() ->
    ?SUCHTHAT({S1, S2}, {stamp(), stamp()}, 
		begin
		    {H1,M1,SEK1} = S1, 
      		{H2,M2,SEK2} = S2,
		    S1 =< S2 andalso H1 =< H2 andalso 
            M1 =< M2 andalso SEK1 =< SEK2
		end
		).

rebar3 project

2 Likes
26

Erlang
(page 127)
test/prop_generators.erl
Question 6 is the most difficult task that I have encountered while reading the book (So far I have read only the first 4 chapters several times.).

The first difficulty - for the input parameter of the generator,

file(Name) ->
    ?SIZED(
       Size,
       lines(Size, {'$call', ?MODULE, file_open, [Name, [read,write,raw]]})
    ).

it requires the name (path) of the file (path to the file). It must be implemented in some way (the author is silent on how to do this, since this is not a significant part of the question). The complexity is compounded by the fact that creating a path to a file (temporary file) in itself is not such an easy task.

I solved the generation of a temporary file using the following functions:

mktemp() ->
    mktemp("tmp").

-spec mktemp(Prefix) -> Result
   when Prefix   :: string(),
        Result   :: TempFile  :: file:filename().
		
mktemp(Prefix) ->
    Rand = integer_to_list(binary:decode_unsigned(crypto:strong_rand_bytes(8)), 36),
	TempDir = filename:basedir(user_cache, Prefix),
	os:cmd("mkdir " ++ "\"" ++ TempDir ++ "\""),
			
	TempFilePath = filename:join(TempDir, Rand),
	TempFilePath.

and implement property based test of their functions Project source code.

Then I tried using the suggested generator file/1. When starting the test,

rebar3 proper -m prop_solutions -p prop_make_tmp_file -n 1

I noticed that Automated Symbolic Call do not work:

===>
0/1 properties passed, 1 failed
===> Failed test cases:
prop_solutions:prop_make_tmp_file() -> {'EXIT',
                                        {undef,
                                         [{prop_solutions,file_open,
                                           [".../AppData/Local/tmp/Cache/15U3A5WBAWWOS",

This means that when trying to call a function file_open, it is not in the expected place. Very strange. :face_with_raised_eyebrow:

Replaced Automated Symbolic Call with Symbolic Call this error has been corrected.

===> Testing prop_solutions:prop_make_tmp_file()
.
OK: Passed 1 test(s).
===>
1/1 properties passed

So, I will continue my research work. I hoped that I had find solutions and everything will work correctly.

Unfortunately, this did not lead to the desired result - the created temporary file does not contain anything. I tested this by introducing the logging function io:format and also directly checked what the files were inside.

That functions that should be called using Symbolic Call are not called all the way.

Also I noticed that the compiler does not perceive symbolic function calls as using them. To prevent the compiler from printing warning messages - I added an extra line to the beginning of the test code file.

-compile([{nowarn_unused_function, [{ file_open, 2}, {file_write, 2}]}]).

So, I will continue my research work with this functionality (look at PropEr documentation). I hope that I will find solutions and everything will work correctly and predictable.

2 Likes
27

After rereading description of author’s Question 6 solution (page 417) I understand my fault - I do not export help functions file_open/2 and file_write/2 into property module (from it to it). So after adding

-export[file_open/2,file_write/2].

undef error has gone. Project source code. Unfortunately, the temporary files remain empty. I will continue to look for a solution.

2 Likes
28

Erlang
(page 417)
test/prop_solutions.erl
Question 6
I think I’ve gotten the most of my current capabilities in this Question (I’ve created properties that work - a temp file is created and populated). @ferd, thanks a lot for this question! He made me suffer, to work with the documentation in more detail. I still have some questions (I would like to improve these tests after all). Maybe (I hope someone can help me answer them when has the time for that).

rebar3 projects:

2 Likes
29

(page 123)

In Wrapping Up section, in the fourth chapter, the author, preparing an inquisitive reader to read the subsequent chapters, says that here the theory of testing based on properties is completed.

Since after the passed chapters, I still do not understand the theory very confidently, I decided to look at the manuals of the authors of the PropEr framework. There is a series of tutorials on PropEr website.

Before proceeding to reading the next chapters, I will read these tutorials (work through, create projects based on these guides).

2 Likes
30

Erlang
(page 136)
Section CSV Parsing

While studying this section, I was a little upset that the author chose such a simple data format encoding / decoding implementation to demonstrate the concept of Responsible Testing. In fact, it is useless for even small data.
I think that having simplified the implementation so much, it is impossible to fully reveal its features, if the tests were more complex - it would be more interesting and exciting.

I tried to make an alternative implementation. So far, decoding has turned out without map module (I use proplists instead). I use EUnit-tests to verify that private functions work. It is fun.

Tests will help me implement a more interesting property-based testing solution. It also has no limitations that maps module has - duplicate columns are possible.

2 Likes
31

Erlang
(page 148)
/bday/test/prop_csv.erl

I think that if it is possible do not lose data when decoding CVS like in this test shows.

rfc_double_quote_test() ->
    ?assertEqual([#{"aaa" => "zzz",
                    "bbb" => "yyy",
                    "ccc" => "xxx"}],
                 bday_csv:decode("\"aaa\",\"bbb\",\"ccc\"\r\nzzz,yyy,xxx")).

I make a solution without double quote removing.
In alternative implementation (without using module maps), I changed the behavior of the decoder and it does not remove the quotes.


rfc_double_quote_test() ->
    Expected = [[{"\"aaa\"", "zzz"}, {"\"bbb\"", "yyy"}, {"\"ccc\"", "xxx"}]],
    Result = bday_csv_tuple:decode("\"aaa\",\"bbb\",\"ccc\"\r\nzzz,yyy,xxx"),
    ?assertEqual(Expected, Result).

Also note that I made the test more readable and easier to understand by breaking it down into Expected, Result, assertEqual blocks.
Source code

2 Likes
32

Erlang
(page 136)
Section CSV Parsing

It is very interesting how the combination of EUnit and Property based tests helps to consistently improve the existing solution.

The algorithm of working with it is as follows:

  1. Writing EUnit tests first then your solution.
  2. When it seems that everything seems to be done - the tests are working perfectly, the solution looks workable - you need to include Property based tests in the work (PropEr). The framework brings you, like a dog to a hunter, a specific combination of inputs that you haven’t tested yet via EUnit.
  3. You create new tests and the process continues until the tests work correctly.

So far, neither my solution without the maps module nor the solution with the maps module have completed tests using the PropEr framework. First of all, I will achieve work (CSV parser) without the maps module, and then I will improve the solution with the maps module - I will write EUnit tests for private functions of that parser and optimize it (will remove duplicate functionality at first).

Source code in progress is here.

2 Likes
33

Erlang
(page 141)
/bday/test/prop_csv.erl
Note that if you do not export csv_source/0 function generator you can not use it in a console and get an error:

Eshell V11.2  (abort with ^G)
1> proper_gen:pick(prop_csv:csv_source()).
** exception error: undefined function prop_csv:csv_source/0
2 Likes
34

CSV parser

When testing using the Symmetric properties of my solution (without using the maps module) - CSV parser, I encountered an unexpected problem - the generator csv_source() produces asymmetric data.

[[
{[],[125]},
{[113],[56,113]},
{[],[120]},
{[98],[]},
{[],[]}
]]

As a result, my encoder cannot distribute data correctly, which causes an error. I think the problem is with the generators. I should test them.

I noticed that the author only tests the final generator - csv_source() (pages 141-142), and does not test intermediate ones. Well, I’ll fill this gap and share the results. Maybe I will found something interesting. To implement this idea, the generators under test need to be exported.

-export([unquoted_text/0,quotable_text/0,field/0,name/0,record/1,entry/2,header/1]).

Next, in the console, we observe generated data of our generators one by one.

>rebar3 as test shell

Eshell V11.2  (abort with ^G)

unquoted_text

1> proper_gen:pick(prop_csv_tuple:unquoted_text()).
{ok,"!!'+"}
2> proper_gen:pick(prop_csv_tuple:unquoted_text()).
{ok,":-F_ d"}
3> proper_gen:pick(prop_csv_tuple:unquoted_text()).
{ok,"1>[Os8"}
4> proper_gen:pick(prop_csv_tuple:unquoted_text()).
{ok,"A;Ar8l"}

quotable_text

5> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"+JqhA="}
6> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"Flgvs"}
7> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"IA2:h;FB"}
8> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"blw"}
9> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"E"}
10> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"Bt,M\nno"}
11> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"LBf7VdMto"}

field

12> proper_gen:pick(prop_csv_tuple:field()).
{ok,"N]~/P\\TQ`s"}
13> proper_gen:pick(prop_csv_tuple:field()).
{ok,"8u0u2g"}
14> proper_gen:pick(prop_csv_tuple:field()).
{ok,"wcbQ.2R"}
15> proper_gen:pick(prop_csv_tuple:field()).
{ok,"VB`UFwD'7"}
16> proper_gen:pick(prop_csv_tuple:field()).
{ok,"%"}
17> proper_gen:pick(prop_csv_tuple:field()).
{ok,"+"}
18> proper_gen:pick(prop_csv_tuple:field()).
{ok,"-?`1a2T_"}
19> proper_gen:pick(prop_csv_tuple:field()).
{ok,":H"}
20> proper_gen:pick(prop_csv_tuple:field()).
{ok,".py`o`%HsE"}

name

21> proper_gen:pick(prop_csv_tuple:name()).
{ok,"[Z* Nf-"}
22> proper_gen:pick(prop_csv_tuple:name()).
{ok,"Xz4S{vy6@E"}
23> proper_gen:pick(prop_csv_tuple:name()).
{ok,"G*Ng`G"}
24> proper_gen:pick(prop_csv_tuple:name()).
{ok,"r\\;!bgnT"}
25> proper_gen:pick(prop_csv_tuple:name()).
{ok,"@L;Vz6z?;7"}

Unfortunately, we cannot run generators that require input numeric values in the console, since the functions that we need for this are hidden inside the PropEr framework. So let’s get down to writing and testing properties.

We will implement the properties of the generators and check them.

$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_unquoted_text
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_quotable_text
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_field
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_header
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_record

Source code.

to be continued…

2 Likes
35

CSV parser

entry generator

I noticed that testing the entry generator has its own peculiarity. If you just try to speed up just copy it from the example book and try to test it, for example:

%%%%%%%%%%%%%%%%%%
%%% Properties %%%
%%%%%%%%%%%%%%%%%%
prop_entry() ->
    ?FORALL(Entry,
           ?SIZED(Size, entry(Size+1, header(Size + 1))),
            begin
                io:format("~p~n", [Entry]),
                true
            end).

%%%%%%%%%%%%%%%%%%
%%% Generators %%%
%%%%%%%%%%%%%%%%%%

entry(Size, Keys) ->
    ?LET(Vals, record(Size), lists:zip(Keys, Vals)).

$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_entry

you will get an endless description of errors to the console.

To understand what this generator should be like, I have implemented a series of generators using parts of this generator, as well as their tests.

entry2/2

%%%%%%%%%%%%%%%%%%
%%% Properties %%%
%%%%%%%%%%%%%%%%%%
prop_entry2() ->
    ?FORALL(Entry,
           ?SIZED(Size, entry2(Size+1)),
            begin
                %io:format("~p~n", [Entry]),
                is_list(Entry) and lists:all(fun(Elem)->
				                     is_tuple(Elem) andalso 2 == tuple_size(Elem)
									 end, Entry)
            end).

%%%%%%%%%%%%%%%%%%
%%% Generators %%%
%%%%%%%%%%%%%%%%%%
entry2(Size) ->
    ?LET(Vals, record(Size), lists:zip(Vals, Vals)).

entry3/2

%%%%%%%%%%%%%%%%%%
%%% Properties %%%
%%%%%%%%%%%%%%%%%%
prop_entry3() ->
    ?FORALL(Entry,
           ?SIZED(Size, entry3(Size+1, header(Size + 1))),
            begin
                io:format("~p~n", [Entry]),
                true
            end).

%%%%%%%%%%%%%%%%%%
%%% Generators %%%
%%%%%%%%%%%%%%%%%%
prop_entry3() ->
    ?FORALL(Entry,
            ?SIZED(Size, entry3(Size + 1, header(Size + 1))),
            begin
                %io:format("~p~n", [Entry]),
                is_list(Entry)
                and lists:all(fun(Elem) -> is_tuple(Elem) andalso 2 == tuple_size(Elem) end,
                              Entry)
            end).

Let’s check the work:
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_entry2 -n 1000
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_entry3 -n 1000

Now writing the generator entry has become obvious, as well as checking it using a property:

entry/2

%%%%%%%%%%%%%%%%%%
%%% Properties %%%
%%%%%%%%%%%%%%%%%%
entry(Size, KeysGen) ->
    ?LET({Vals, Keys},
         {record(Size), KeysGen},
         begin
             %io:format("Keys = ~p, Vals = ~p~n",[Keys,Vals]),
             lists:zip(Keys, Vals)
         end).

%%%%%%%%%%%%%%%%%%
%%% Generators %%%
%%%%%%%%%%%%%%%%%%
prop_entry() ->
    ?FORALL(Entry,
            ?SIZED(Size, entry(Size + 1, header(Size + 1))),
            begin
                io:format("~p~n", [Entry]),
                is_list(Entry)
                and lists:all(fun(Elem) -> is_tuple(Elem) andalso 2 == tuple_size(Elem) end,
                              Entry)
            end).

Check it:
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_entry -n 1000

...
 {"M&cPk%~ZV&`-'P^2~Pi-Z=6P.4y","RM"},
 {"=Lu$%#R<O$,jyKj*F","\rwVk8dsn/ Z\nN\"\"GadpW%< "},
 {"Bj*5#H`KeAs92//iWA6&;vq-GM*jJ:QF\\","'$a3\"6&s:0q{-G#!UgyUoN#\"4V]m?8\\p"}]
.
OK: Passed 1000 test(s).
===>
1/1 properties passed

Cool.

2 Likes
36

Once again, I found I had an error. I got confused about the Erlang data format which I’ve built recentry and therefore my property based tests kept failing over and over again. I figured out what was the matter - I was encoding the format incorrectly into a string.

As is often the case, the simplest solutions are the most difficult. Everything is going well so far. I rewrote the tests and they work.

I noticed that it is already easier for me to work with all this testing infrastructure. The author wrote about this at the beginning of the book. You need to work harder to get used to it, you need to write, code, think and correct your mistakes.

Source code

2 Likes
37

Erlang #erlang
ch05\ex02_bday\apps\bday\src\bday_csv_tuple.erl
As an experiment, I decided to move the tests of the private functions into separate files so that they would not clutter up the source code file. I did it using the -include directive. Checked it out. Everything works the same.

bday_csv_tuple.erl
                ├───encode.tests
                ├───decode.tests

bday_csv_tuple.erl

%...

%%
%% Tests
%%
-ifdef(TEST).

-include_lib("eunit/include/eunit.hrl").

%%%%%%%%%%%%%%%%
%%% Encoding %%%
%%%%%%%%%%%%%%%%

-include("encode.tests").

%%%%%%%%%%%%%%%%
%%% Decoding %%%
%%%%%%%%%%%%%%%%

-include("decode.tests").

-endif.

Source code

1 Like
38

During the implementation of decoding testing functionality, I realized that I was being misled by ok atom. Its meaning does not carry any semantic meaning. It’s incomprehensible (unless you read the comment in the source code what it means). I decided to dispel this uncertainty and add a new atom go_on into operation. This atom already carries a semantic load and it is already clear that it is needed to indicate the continuation of work. Atom done is used that the processing is complete.

I also continued to use the directive -include (it seems to me very convenient and its use greatly simplifies testing management - now I can quite easily disable groups of tests that I don’t want to pay attention to at the moment).

decode.tests

%-include("decode_unquoted.tests").
%-include("decode_quoted.tests").
-include("decode_field.tests").
%-include("decode_row.tests").

Source code

2 Likes
39

I have a certain disadvantage in the source code that is added via -include directive. Tests is not run automatically when source code updated and tests running. This is a little annoying.

I solved this problem by using rebar3 format command. After that, the code is successfully updated and everything works correctly. :blush:

2 Likes
40

Huh? Can you elaborate on this? I’m curious…

1 Like
41

#erlang
I see. The recompilation just doesn’t happen.

I am now actively looking for an error in my code - the test generator is not working correctly or the code is not working correctly. This is very interesting to me myself.

For the purpose of testing the operation of private functions, I began to actively use -include directive. While making changes to included files and restarting tests, I noticed that added new tests were not running.

After starting the code helper - formatting, everything worked.

Source code

Thank you, @OvermindDL1!

2 Likes