Journal: Property-Based Testing with PropEr, Erlang, and Elixir

Erlang
(page 118)
test/prop_generators.erl
Question 4
Be careful when checking the tree generator. If you just run it, you will not see anything, since trees will be created endlessly and only a forced interruption of the process will stop the computation by Ctrl-C keys for instance.

% Do not run it - it is infinite tree maker.
% For illustration only.
prop_tree() ->
    ?FORALL(Type, tree(),
        begin
            io:format("~p~n",[Type]),
			true
        end).

Using a macro ?LAZY provides some probability of stopping the creation of trees, but this solution does not work regularly and at the 2nd or 3rd run of the property, the system may freeze, since trees will still be created and created. You can also terminate this by forcibly interrupting the test.

Using a macro ?SIZED solves all the previous problems.

It is very convenient to add limiting generator without parameters limited_tree().

I also noticed that it would be convenient and logical to combine the sequential call of limited_tree() generators into one using fixed_list([Gen]) generator.

To reduce informational noise and better see the resulting structures, you can use the boolean() generator as a content for limited_tree() generator.

limited_tree() ->
    limited_tree(boolean()).

rebar3 project

2 Likes

Erlang
(page 125
test/prop_generators.erl
Question 5
Note that in stamps1 generator no sorting by content occurs when comparing tuples. I tried to correct the lack of this check and wrote a check property on the basis of this.

%% Return two ordered timestamps
stamps1() ->
    ?SUCHTHAT({S1, S2}, {stamp(), stamp()}, 
		begin
		    {H1,M1,SEK1} = S1, 
      		{H2,M2,SEK2} = S2,
		    S1 =< S2 andalso H1 =< H2 andalso 
            M1 =< M2 andalso SEK1 =< SEK2
		end
		).

rebar3 project

2 Likes

Erlang
(page 127)
test/prop_generators.erl
Question 6 is the most difficult task that I have encountered while reading the book (So far I have read only the first 4 chapters several times.).

The first difficulty - for the input parameter of the generator,

file(Name) ->
    ?SIZED(
       Size,
       lines(Size, {'$call', ?MODULE, file_open, [Name, [read,write,raw]]})
    ).

it requires the name (path) of the file (path to the file). It must be implemented in some way (the author is silent on how to do this, since this is not a significant part of the question). The complexity is compounded by the fact that creating a path to a file (temporary file) in itself is not such an easy task.

I solved the generation of a temporary file using the following functions:

mktemp() ->
    mktemp("tmp").

-spec mktemp(Prefix) -> Result
   when Prefix   :: string(),
        Result   :: TempFile  :: file:filename().
		
mktemp(Prefix) ->
    Rand = integer_to_list(binary:decode_unsigned(crypto:strong_rand_bytes(8)), 36),
	TempDir = filename:basedir(user_cache, Prefix),
	os:cmd("mkdir " ++ "\"" ++ TempDir ++ "\""),
			
	TempFilePath = filename:join(TempDir, Rand),
	TempFilePath.

and implement property based test of their functions Project source code.

Then I tried using the suggested generator file/1. When starting the test,

rebar3 proper -m prop_solutions -p prop_make_tmp_file -n 1

I noticed that Automated Symbolic Call do not work:

===>
0/1 properties passed, 1 failed
===> Failed test cases:
prop_solutions:prop_make_tmp_file() -> {'EXIT',
                                        {undef,
                                         [{prop_solutions,file_open,
                                           [".../AppData/Local/tmp/Cache/15U3A5WBAWWOS",

This means that when trying to call a function file_open, it is not in the expected place. Very strange. :face_with_raised_eyebrow:

Replaced Automated Symbolic Call with Symbolic Call this error has been corrected.

===> Testing prop_solutions:prop_make_tmp_file()
.
OK: Passed 1 test(s).
===>
1/1 properties passed

So, I will continue my research work. I hoped that I had find solutions and everything will work correctly.

Unfortunately, this did not lead to the desired result - the created temporary file does not contain anything. I tested this by introducing the logging function io:format and also directly checked what the files were inside.

That functions that should be called using Symbolic Call are not called all the way.

Also I noticed that the compiler does not perceive symbolic function calls as using them. To prevent the compiler from printing warning messages - I added an extra line to the beginning of the test code file.

-compile([{nowarn_unused_function, [{ file_open, 2}, {file_write, 2}]}]).

So, I will continue my research work with this functionality (look at PropEr documentation). I hope that I will find solutions and everything will work correctly and predictable.

2 Likes

After rereading description of author’s Question 6 solution (page 417) I understand my fault - I do not export help functions file_open/2 and file_write/2 into property module (from it to it). So after adding

-export[file_open/2,file_write/2].

undef error has gone. Project source code. Unfortunately, the temporary files remain empty. I will continue to look for a solution.

2 Likes

Erlang
(page 417)
test/prop_solutions.erl
Question 6
I think I’ve gotten the most of my current capabilities in this Question (I’ve created properties that work - a temp file is created and populated). @ferd, thanks a lot for this question! He made me suffer, to work with the documentation in more detail. I still have some questions (I would like to improve these tests after all). Maybe (I hope someone can help me answer them when has the time for that).


rebar3 projects:

2 Likes

(page 123)

In Wrapping Up section, in the fourth chapter, the author, preparing an inquisitive reader to read the subsequent chapters, says that here the theory of testing based on properties is completed.

Since after the passed chapters, I still do not understand the theory very confidently, I decided to look at the manuals of the authors of the PropEr framework. There is a series of tutorials on PropEr website.

Before proceeding to reading the next chapters, I will read these tutorials (work through, create projects based on these guides).

2 Likes

Erlang
(page 136)
Section CSV Parsing

While studying this section, I was a little upset that the author chose such a simple data format encoding / decoding implementation to demonstrate the concept of Responsible Testing. In fact, it is useless for even small data.
I think that having simplified the implementation so much, it is impossible to fully reveal its features, if the tests were more complex - it would be more interesting and exciting.

I tried to make an alternative implementation. So far, decoding has turned out without map module (I use proplists instead). I use EUnit-tests to verify that private functions work. It is fun.

Tests will help me implement a more interesting property-based testing solution. It also has no limitations that maps module has - duplicate columns are possible.

2 Likes

Erlang
(page 148)
/bday/test/prop_csv.erl

I think that if it is possible do not lose data when decoding CVS like in this test shows.

rfc_double_quote_test() ->
    ?assertEqual([#{"aaa" => "zzz",
                    "bbb" => "yyy",
                    "ccc" => "xxx"}],
                 bday_csv:decode("\"aaa\",\"bbb\",\"ccc\"\r\nzzz,yyy,xxx")).

I make a solution without double quote removing.
In alternative implementation (without using module maps), I changed the behavior of the decoder and it does not remove the quotes.


rfc_double_quote_test() ->
    Expected = [[{"\"aaa\"", "zzz"}, {"\"bbb\"", "yyy"}, {"\"ccc\"", "xxx"}]],
    Result = bday_csv_tuple:decode("\"aaa\",\"bbb\",\"ccc\"\r\nzzz,yyy,xxx"),
    ?assertEqual(Expected, Result).

Also note that I made the test more readable and easier to understand by breaking it down into Expected, Result, assertEqual blocks.
Source code

2 Likes

Erlang
(page 136)
Section CSV Parsing

It is very interesting how the combination of EUnit and Property based tests helps to consistently improve the existing solution.

The algorithm of working with it is as follows:

  1. Writing EUnit tests first then your solution.
  2. When it seems that everything seems to be done - the tests are working perfectly, the solution looks workable - you need to include Property based tests in the work (PropEr). The framework brings you, like a dog to a hunter, a specific combination of inputs that you haven’t tested yet via EUnit.
  3. You create new tests and the process continues until the tests work correctly.

So far, neither my solution without the maps module nor the solution with the maps module have completed tests using the PropEr framework. First of all, I will achieve work (CSV parser) without the maps module, and then I will improve the solution with the maps module - I will write EUnit tests for private functions of that parser and optimize it (will remove duplicate functionality at first).

Source code in progress is here.

2 Likes

Erlang
(page 141)
/bday/test/prop_csv.erl
Note that if you do not export csv_source/0 function generator you can not use it in a console and get an error:

Eshell V11.2  (abort with ^G)
1> proper_gen:pick(prop_csv:csv_source()).
** exception error: undefined function prop_csv:csv_source/0
2 Likes

CSV parser

When testing using the Symmetric properties of my solution (without using the maps module) - CSV parser, I encountered an unexpected problem - the generator csv_source() produces asymmetric data.

[[
{[],[125]},
{[113],[56,113]},
{[],[120]},
{[98],[]},
{[],[]}
]]

As a result, my encoder cannot distribute data correctly, which causes an error. I think the problem is with the generators. I should test them.

I noticed that the author only tests the final generator - csv_source() (pages 141-142), and does not test intermediate ones. Well, I’ll fill this gap and share the results. Maybe I will found something interesting. To implement this idea, the generators under test need to be exported.

-export([unquoted_text/0,quotable_text/0,field/0,name/0,record/1,entry/2,header/1]).

Next, in the console, we observe generated data of our generators one by one.

>rebar3 as test shell

Eshell V11.2  (abort with ^G)

unquoted_text

1> proper_gen:pick(prop_csv_tuple:unquoted_text()).
{ok,"!!'+"}
2> proper_gen:pick(prop_csv_tuple:unquoted_text()).
{ok,":-F_ d"}
3> proper_gen:pick(prop_csv_tuple:unquoted_text()).
{ok,"1>[Os8"}
4> proper_gen:pick(prop_csv_tuple:unquoted_text()).
{ok,"A;Ar8l"}

quotable_text

5> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"+JqhA="}
6> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"Flgvs"}
7> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"IA2:h;FB"}
8> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"blw"}
9> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"E"}
10> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"Bt,M\nno"}
11> proper_gen:pick(prop_csv_tuple:quotable_text()).
{ok,"LBf7VdMto"}

field

12> proper_gen:pick(prop_csv_tuple:field()).
{ok,"N]~/P\\TQ`s"}
13> proper_gen:pick(prop_csv_tuple:field()).
{ok,"8u0u2g"}
14> proper_gen:pick(prop_csv_tuple:field()).
{ok,"wcbQ.2R"}
15> proper_gen:pick(prop_csv_tuple:field()).
{ok,"VB`UFwD'7"}
16> proper_gen:pick(prop_csv_tuple:field()).
{ok,"%"}
17> proper_gen:pick(prop_csv_tuple:field()).
{ok,"+"}
18> proper_gen:pick(prop_csv_tuple:field()).
{ok,"-?`1a2T_"}
19> proper_gen:pick(prop_csv_tuple:field()).
{ok,":H"}
20> proper_gen:pick(prop_csv_tuple:field()).
{ok,".py`o`%HsE"}

name

21> proper_gen:pick(prop_csv_tuple:name()).
{ok,"[Z* Nf-"}
22> proper_gen:pick(prop_csv_tuple:name()).
{ok,"Xz4S{vy6@E"}
23> proper_gen:pick(prop_csv_tuple:name()).
{ok,"G*Ng`G"}
24> proper_gen:pick(prop_csv_tuple:name()).
{ok,"r\\;!bgnT"}
25> proper_gen:pick(prop_csv_tuple:name()).
{ok,"@L;Vz6z?;7"}

Unfortunately, we cannot run generators that require input numeric values in the console, since the functions that we need for this are hidden inside the PropEr framework. So let’s get down to writing and testing properties.

We will implement the properties of the generators and check them.

$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_unquoted_text
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_quotable_text
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_field
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_header
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_record

Source code.


to be continued…

2 Likes

CSV parser

entry generator

I noticed that testing the entry generator has its own peculiarity. If you just try to speed up just copy it from the example book and try to test it, for example:

%%%%%%%%%%%%%%%%%%
%%% Properties %%%
%%%%%%%%%%%%%%%%%%
prop_entry() ->
    ?FORALL(Entry,
           ?SIZED(Size, entry(Size+1, header(Size + 1))),
            begin
                io:format("~p~n", [Entry]),
                true
            end).

%%%%%%%%%%%%%%%%%%
%%% Generators %%%
%%%%%%%%%%%%%%%%%%

entry(Size, Keys) ->
    ?LET(Vals, record(Size), lists:zip(Keys, Vals)).
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_entry

you will get an endless description of errors to the console.

To understand what this generator should be like, I have implemented a series of generators using parts of this generator, as well as their tests.

entry2/2

%%%%%%%%%%%%%%%%%%
%%% Properties %%%
%%%%%%%%%%%%%%%%%%
prop_entry2() ->
    ?FORALL(Entry,
           ?SIZED(Size, entry2(Size+1)),
            begin
                %io:format("~p~n", [Entry]),
                is_list(Entry) and lists:all(fun(Elem)->
				                     is_tuple(Elem) andalso 2 == tuple_size(Elem)
									 end, Entry)
            end).

%%%%%%%%%%%%%%%%%%
%%% Generators %%%
%%%%%%%%%%%%%%%%%%
entry2(Size) ->
    ?LET(Vals, record(Size), lists:zip(Vals, Vals)).

entry3/2

%%%%%%%%%%%%%%%%%%
%%% Properties %%%
%%%%%%%%%%%%%%%%%%
prop_entry3() ->
    ?FORALL(Entry,
           ?SIZED(Size, entry3(Size+1, header(Size + 1))),
            begin
                io:format("~p~n", [Entry]),
                true
            end).

%%%%%%%%%%%%%%%%%%
%%% Generators %%%
%%%%%%%%%%%%%%%%%%
prop_entry3() ->
    ?FORALL(Entry,
            ?SIZED(Size, entry3(Size + 1, header(Size + 1))),
            begin
                %io:format("~p~n", [Entry]),
                is_list(Entry)
                and lists:all(fun(Elem) -> is_tuple(Elem) andalso 2 == tuple_size(Elem) end,
                              Entry)
            end).

Let’s check the work:
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_entry2 -n 1000
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_entry3 -n 1000

Now writing the generator entry has become obvious, as well as checking it using a property:

entry/2

%%%%%%%%%%%%%%%%%%
%%% Properties %%%
%%%%%%%%%%%%%%%%%%
entry(Size, KeysGen) ->
    ?LET({Vals, Keys},
         {record(Size), KeysGen},
         begin
             %io:format("Keys = ~p, Vals = ~p~n",[Keys,Vals]),
             lists:zip(Keys, Vals)
         end).

%%%%%%%%%%%%%%%%%%
%%% Generators %%%
%%%%%%%%%%%%%%%%%%
prop_entry() ->
    ?FORALL(Entry,
            ?SIZED(Size, entry(Size + 1, header(Size + 1))),
            begin
                io:format("~p~n", [Entry]),
                is_list(Entry)
                and lists:all(fun(Elem) -> is_tuple(Elem) andalso 2 == tuple_size(Elem) end,
                              Entry)
            end).

Check it:
$ rebar3 proper -d apps/bday/test -m prop_csv_tuple -p prop_entry -n 1000

...
 {"M&cPk%~ZV&`-'P^2~Pi-Z=6P.4y","RM"},
 {"=Lu$%#R<O$,jyKj*F","\rwVk8dsn/ Z\nN\"\"GadpW%< "},
 {"Bj*5#H`KeAs92//iWA6&;vq-GM*jJ:QF\\","'$a3\"6&s:0q{-G#!UgyUoN#\"4V]m?8\\p"}]
.
OK: Passed 1000 test(s).
===>
1/1 properties passed

Cool.

2 Likes

Once again, I found I had an error. I got confused about the Erlang data format which I’ve built recentry and therefore my property based tests kept failing over and over again. I figured out what was the matter - I was encoding the format incorrectly into a string.


As is often the case, the simplest solutions are the most difficult. Everything is going well so far. I rewrote the tests and they work.

I noticed that it is already easier for me to work with all this testing infrastructure. The author wrote about this at the beginning of the book. You need to work harder to get used to it, you need to write, code, think and correct your mistakes.

Source code

2 Likes

Erlang #erlang
ch05\ex02_bday\apps\bday\src\bday_csv_tuple.erl
As an experiment, I decided to move the tests of the private functions into separate files so that they would not clutter up the source code file. I did it using the -include directive. Checked it out. Everything works the same.

bday_csv_tuple.erl
                ├───encode.tests
                ├───decode.tests

bday_csv_tuple.erl

%...

%%
%% Tests
%%
-ifdef(TEST).

-include_lib("eunit/include/eunit.hrl").

%%%%%%%%%%%%%%%%
%%% Encoding %%%
%%%%%%%%%%%%%%%%

-include("encode.tests").

%%%%%%%%%%%%%%%%
%%% Decoding %%%
%%%%%%%%%%%%%%%%

-include("decode.tests").

-endif.

Source code

1 Like

During the implementation of decoding testing functionality, I realized that I was being misled by ok atom. Its meaning does not carry any semantic meaning. It’s incomprehensible (unless you read the comment in the source code what it means). I decided to dispel this uncertainty and add a new atom go_on into operation. This atom already carries a semantic load and it is already clear that it is needed to indicate the continuation of work. Atom done is used that the processing is complete.

I also continued to use the directive -include (it seems to me very convenient and its use greatly simplifies testing management - now I can quite easily disable groups of tests that I don’t want to pay attention to at the moment).

decode.tests

%-include("decode_unquoted.tests").
%-include("decode_quoted.tests").
-include("decode_field.tests").
%-include("decode_row.tests").

Source code

2 Likes

I have a certain disadvantage in the source code that is added via -include directive. Tests is not run automatically when source code updated and tests running. This is a little annoying.

I solved this problem by using rebar3 format command. After that, the code is successfully updated and everything works correctly. :blush:

2 Likes

Huh? Can you elaborate on this? I’m curious…

1 Like

#erlang
I see. The recompilation just doesn’t happen.

I am now actively looking for an error in my code - the test generator is not working correctly or the code is not working correctly. This is very interesting to me myself.

For the purpose of testing the operation of private functions, I began to actively use -include directive. While making changes to included files and restarting tests, I noticed that added new tests were not running.

After starting the code helper - formatting, everything worked.

Source code


Thank you, @OvermindDL1!

2 Likes

#erlang

I still can’t debug my implementation of the CSV converter, but I continue to work on it systematically (I find and fix errors in my code, I try to simplify the solution (I refactor, redistribute functionality between generators and the converter itself)).

Thanks to my work, I get experience with testing. Solving constantly emerging problems stimulates my development.


I love using the formatting command (rebar3 format) - I love making the code easy to read. Unfortunately, you have to be careful when using this command. I recommend that you first make sure that the code is correct (using rebar3 compile command), and only then do the formatting.

If this is not done, the result can be disastrous. The formatting feature can destroy the source code text.

2 Likes

#erlang

My thoughts on improving the CSV decoder/encoder led me to the idea of using the functionality of regular expressions. This would simplify some of the checks and data analysis would be greatly simplified.

Unintentionally, I don’t have enough experience with regular expressions in Erlang yet. So I need to practice with the re module.

For this purpose, I created a new repository in which I will post examples of working with the re module’s functionality.

2 Likes