Journal : Designing Elixir systems with OTP

Community Journals & Book Clubs
1

This is my Journal for readings on Designing Elixir Systems with OTP.

Will post chapter 01 tomorrow! Stay tuned!

3 Likes
2

Corresponding tweet for this thread:

Share link for this tweet.

3 Likes
3

Nice one TT! Another book I am interested in reading at some point! :003:

3 Likes
4

Chapter 01

Concepts / RE-Thinks:

DFTwBLWB : Do Fun things with Big Loud Worker Bees - acronym to think about layering application.

DFT = for building blocks of project

BLWB = for letting components work together.

Why layers?

  1. Testing each component with SIMILAR functions together (because they are grouped together! )
  2. Healing - supervisors
    1. the concept is (called lifecycle) as simple as turn-off the TV and restart to fix it!
    2. So, shut down CLEANLY (note,emphasis)!

Managing state :

  1. recursion and message passing to manage state.
    • This is new! = message passing is usually an advanced concept and not ‘generally’ used to manage state, unless - Elixir or MVC model.
3 Likes
6

Chapter 02 : Know Your Elixir Datatypes

Types and pro-cons

DataType Properties Pro Cons
Atoms are integer + constant + NOT garbage collected use = name concepts + constant tables type lookup NOT convert user-input to atom
Arrays = NOT in Elixir! - random access is cheap + traversing O(n) +
Lists NOT arrays! + singly linked-lists + pattern matching on head is O(1) Appending at END of lists or LARGE lists
Maps random access O(log n) => useful for heavily edited data use MapSet when only Keys (uniqueness)
Strings are binaries => efficiently stored + copy is a FULL copy (unlike lists) DO NOT LET processes hold references to long string => may cause memory leaks + NOT use long strings for storing info + DONT concatenate
Tuples fixed length data structure use for tagging data + read chunks of data like csv slow append => don’t edit (if need to, use Maps)
Functions send function to data, NOT data to function
PATTERN matching allows multiple dimension inheritance (sort of) => better than inheritance!!
3 Likes
7

Chapter One: Build Your Project in Layers

This book is about design, and because Elixir heavily uses OTP, we must address how to construct layers around an OTP program.

We recommend the software layers: data structures, a functional core, tests, boundaries, lifecycle, and workers.
Do Fun Things with Big Loud Worker bees

D = Data structures (model the domain)
F = Functional programming (logic)
T = Testing
B = Boundaries (processes)
L = Lifecycles (supervisors)
W = Workers (pools and dependencies)

If you think of OTP as a way to encapsulate data, or even objects, you’re going to get it wrong. Elixir processes work best when they span a few modules that belong together. Breaking your processes up too finely invites integrity problems the same way that global variables do.

Question: what is an “integrity problem” here? I’m not sure I can envision the problem the author is concerned about.

Example, we’d rather wrap a process around a chess game as a standalone component than have each piece in its own process, so we can enforce the integrity of the board at the game level.

Begin with the Right Datatypes

The “data” layer has the simple data structures your functions will use.

When the data structure is right, the functions holding the algorithms that do things can seem to write themselves.

Functional Logic

  • It must not have side effects, meaning it should not alter the state of its environment in any way.

  • A function invoked with the same inputs will always return the same outputs.

Boundary Layer

The boundary layer deals with side effects and state. This layer is where you’ll deal with processes, and where you’ll present your API to the outside world. In Elixir, that means OTP.

  • The machinery of processes, message passing, and recursion that form the heart of concurrency in Elixir systems

  • An API of plain functions that hides that machinery from clients

We typically call the collective machinery a server, the code that calls that server an API, and the code that calls that API a client. In OTP’s case, the server in that boundary layer is called a GenServer, which is an abbreviation for Generic Server.

  • Keep your functional core separate

Testing

  • test the functional core
  • test the API, as though you are a client

Plan Your Lifecycle

  • “supervisor” vs “lifecycle” – it’s about more than just handling failure.

Supervisors are about starting and stopping cleanly, whether you have a single server or a bunch of them. Once you can start cleanly and detect failure, you can get failover almost for free. When a customer support person says “Did you try turning it off and on again?”, they are using lifecycle to recover from failure, whether you’re working with a TV or a desktop computer program. They are making a good bet that shutting things down cleanly and starting with a known good state is a powerful way to heal broken things.

Workers

The workers are the different processes in your component. Generally, you’ll start with a flat design having a single worker and decide where you need to be more sophisticated. As your design evolves you will possibly see places that you need to add workers, for cleaning up lifecycles or for concurrently dividing work. Connection pools are workers; tasks and agents can be as well.

Do Fun Things

  • Data
  • Functions
  • Tests

Big, Loud, Worker-bees

  • Boundaries
  • Lifecycles
  • Workers
3 Likes
8

Know Your Elixir Datatypes

  • All other layers depend on the data layer.

In Elixir, the data is the good stuff.

  • Functional languages generally support fixed-lenth lists like Elixir’s tuples.

Primitive Types

  • Booleans
  • Floats
  • Integers
  • Atoms
  • References

Numbers

  • Integers and Floats
  • Floats are estimates
  • Prefer integers or decimals over floats when you can.
  • Store money as an integer in cents.
  • Use integer division if possible
  • Use make_ref() to get a globally unique identifier – use these to identify things instead of numbers.

Atoms

  • Atoms are for naming concepts
  • Atoms are different than strings internally – two different strings with the same contents may or may not be the same, but two different atoms with the same contents are the same object.
  • Atoms should be used for things with a small set of possible values – don’t exhaust the atom table!

Lists

  • In Elixir lists are singly linked, meaning that each node of a list points to the next node. That’s extremely different than arrays. Arrays make random access cheap, but traversing lists takes longer.
  • A list with n elements is actually n different lists.
  • Lists are built head-first and accessing them by the head is extremely efficient.
    • Adding an element to the head is O(1)
  • Avoid copying lists if you can
  • When you’re dealing with very large datasets, data of indeterminate size, or data from external sources, you’ll want to use streams.
  • Random access in lists is slow

Maps and Structs

  • A struct is actually implemented as a map
  • All structs have a __struct__ field that plain Elixir maps don’t have.
  • defstruct is a macro
  • You can use @enforce_keys to force the specification of one or more fields when creating a new struct.
  • Random access in maps is O(log n), much better than O(n) for lists.
  • Updating a map is O(log n)
  • Any data that will be heavily edited should be a map if possible
  • Maps work well with Elixir pattern matching.

Pattern Matching

  • Polymorphism – behaviors that work different for the same data structure.
defmodule Animal do
  defstruct type: "", legs: 4
end

def speak(%Animal{type: "dog"}), do: "Woof"
def speak(%Animal{type: "cat"}), do: "Meow"
  • Inheritance limits extension to a single dimension. Often, you may need to be able to invoke logic across more than one dimension. Even if you have thousands of clauses, pattern matching used in this way is fast. Matching a map or struct is O(log n)

Map Traps

  • You cannot count on the ordering in maps – if you need to enforce order, use a list
  • Keyword lists are lists of two-tuples: an atom key and any type for a value.
    • better for function options because they allow duplication and support syntactic sugar
  • If you know the keys in advance, upgrade to a struct
  • MapSet is a collection of values of any type that supports ==

Strings

  • prefer strings for user-defined text and atoms for naming concepts in code.
  • two kinds of strings: charlist, binary
  • https://elixir-lang.org/getting-started/binaries-strings-and-char-lists.html
  • A bitstring is a fundamental data type in Elixir, denoted with the <<>> syntax. A bitstring is a contiguous sequence of bits in memory.
  • A binary is a bitstring where the number of bits is divisible by 8. That means that every binary is a bitstring, but not every bitstring is a binary.
  • A string is a UTF-8 encoded binary, where the code point for each character is encoded using 1 to 4 bytes. Thus every string is a binary, but due to the UTF-8 standard encoding rules, not every binary is a valid string.
  • The string concatenation operator <> is actually a binary concatenation operator.
  • A charlist is a list of integers where all the integers are valid code points. In practice, you will not come across them often, except perhaps when interfacing with Erlang.
  • single quotes for charlist
  • charlist is basically just an array of ascii numbers
  • Strings with double quotes are not the same as charlists
  • Prefer strings over charlists – i.e., avoid single quotes

String Traps

  • the BEAM shares long strings across processes, and lets them go after all references are cleared – it’s extremely important to refrain from letting processes hold references to large strings for longer than needed.
  • editing or finding a character are O(n), so don’t use long strings to encode information
  • string concatenation is slow

Tuples

  • immutable (like all elixir datatypes) fixed-length data structures
  • efficient for accessing and pattern matching
  • The position within the tuple means something
  • good for chunks of data with the same structure, like CSV files and database query results

Tuple traps

  • When you find yourself having trouble remembering which element of the tuple goes in which position, it’s time to switch to a map.
  • Appending to tuples is slow
  • If you find yourself editing tuples, you should prefer maps
  • If you find yourself iterating through a tuple via an index, switch to a list

Functions as Data

  • Elixir is a functional language, which means that functions are data, too
  • Don’t send the data to the function – send the function to the data
  • Functions can be sent into processes as part of the message, just like other datatypes

When To Leave Elixir

  • When you’re doing number-crunching – i.e., working with a lot of arrays that you need to update frequently and randomly.

Final Word

Remember, the BEAM is a powerful toolkit – closer to an operating system than most programming language runtime environments.

3 Likes
9

Start with the Right Data Layer

  • In functional programming, functions can’t update data in place, they must create new copies that transform data step by step

  • Example of the data structure used to represent a tic-tac-toe game

    • three-tuple of three-tuples, or list of strings?
    • deep data structures are hard to update, so use flat data structures
    • maps can use tuples as keys!
iex> board =
... %{
... {0,0}=>"O",{0,1}=>"",{0,2}=>"",
... {1,0}=>"",{1,1}=>"X",{1,2}=>"",
... {2,0}=>"",{2,1}=>"",{2,2}=>"",
}

iex> board[{1,1}]
"X"
iex> Map.put(board, {1,0}, "0")
  • If you want to write beautiful code, you need to design the right data structures that consider your primary access patterns.

Immutability Drives Everything

  • functional programming means that the same inputs will give you the same outputs
  • This is why functional languages are so good at concurrency. Multiple processes can access the same data without having to deal with the data changing out from under them.

New Facts Don’t Invalidate Old Facts

  • Object-oriented data structures change over time. Functional data structures are maps of stable values over time. Functional programs do this automatically. Changing anything means creating a new copy, and your data structures will reflect these new realities.

Write Data Structures Functionally

  • Example of a how to represent a bank account.
account:
  %{
    account_number: String,
    account_holder: %User{},
    balance: Int,
    transaction_log: [strings],
}

This structure works OK in many languages, but it is not a functional data structure.

Processes Are Not Data

  • We have built something that works just like an OOP variable that answers the question “What is the current balance?”
  • A much better question is “What is the balance at a specific time?”
    • To answer that question, we can store an initial balance and all of the changes represented in our transactions.
    • We can get all transactions since the beginning of time, or if this becomes a performance problem, all transactions since a checkpoint.
account:
  %{
    account_number: String,
    initial_balance: Integer,
    account_holder: %User{},
    transactions: [%Transaction{}],
}
transaction:
  %{
    change: Integer,
    inserted_at: DateTime,
    note: String,
%}
def balance(account_number, date_time), do: ...
  • balance becomes a function that computes a balance at a point in time based on adding all of the transactions, each with a change that has positive or negative values. We can then start with a balance and reduce over the transactions to get a balance. There’s no ambiguity. It’s completely deterministic.

Mastery Quiz Example

  • picking the nouns out of our description gives us a good start toward the structure of our data

lib/mastery/core/template.ex:

defmodule Mastery.Core.Template do
  defstruct ~w[name category instructions raw compiled generators checker]a
end

  • We use the sigil ~w to create a list of words. Though you usually see () characters with this sigil, the [] characters work perfectly fine. The a modifier means the statement will create a list of atoms instead of strings.

  • We can use the data structure to:

    • Represent a grouping of questions on a quiz
    • Generate questions with a compilable template and functions
    • Check the response of a single question in the template

  • We have a known set of fields of disparate types

    • that structure screams map

Templates Generate Questions

lib/mastery/core/question.ex:

defmodule Mastery.Core.Question do
  defstruct ~w[asked substitutions template]a
end
  • Templates generate questions, and questions are instantiations of those templates.

User Answer with Responses

lib/mastery/core/response.ex:

defmodule Mastery.Core.Response do
  defstruct ~w[quiz_title template_name to email answer correct timestamp]a
end

Quizes Ask Questions

lib/mastery/core/quiz.ex:

defmodule Mastery.Core.Quiz do
  defstruct title: nil,
        mastery: 3,
        templates: %{ },
        used: [ ],
        current_question: nil,
        last_response: nil,
        record: %{ },
        mastered: [ ]
end
  • We haven’t written any code yet, but we have a pretty good idea of how our program will work, just by looking at the data structure of the quiz.

Start with the Right Data

  • examined how choices of data structure might change access patterns and impact the complexity of the code we write
  • introduced simple principles to keep data structures flat and saw that functional data structures are generally slower
  • functional programmers prefer many versions of a value over time rather than continuously mutating a single value
3 Likes
10

Build a Functional Core

  • The functional core is the business logic
  • A functional core is a group of functions and the type definitions representing the data layer, organized into modules.
  • The core doesn’t access external interfaces or use any process machinery your component might use.
  • In Elixir, that process machinery is the GenServer, and those bits are banished to the outer bands of our architecture.

The core presents a clear, stable interface to any external code.

  • Pure Functions

    • a pure function returns the same value given the same inputs each time you run it
    • the core doesn’t have to be completely pure
      • timestamps
      • ID generation
      • random number generation
    • a functional core gets much easier to manage if the same inputs always generate the same outputs.

  • When you group like functions together based on the data with the sole purpose of managing that kind of data, Elixir code becomes easier to code.

  • Some of our modules have only data and a constructor, and that’s OK.

    • Think of a constructor as a convenience function to instantiate a piece of data.
defmodule Mastery.Core.Response do
  defstruct ~w[quiz_title template_name to email answer correct timestamp]a

  def new(quiz, email, answer) do
    question = quiz.current_question
    template = question.template
    
    %__MODULE__{
      quiz_title: quiz.title,
      template_name: template.name,
      to: question.asked,
      email: email,
      answer: answer,
      correct: template.checker.(question.substitutions, answer),
      timestamp: DateTime.utc_now
    }
  end 
end
  • We’re using __MODULE__ instead of typing the full name of the module because that code defaults to the current module, and protects us from refactoring code whenever we reorganize the project.

Edit to a Single Purpose

  • Our rule of thumb is to use processes only when we need them to control execution, divide work, or store common state where functions won’t work.

  • We’ll need to compile templates as users create them.

defmodule Mastery.Core.Template do
  defstruct ~w[name category instructions raw compiled generators checker]a

  def new(fields) do
    raw = Keyword.fetch!(fields, :raw) struct!(
      __MODULE__,
      Keyword.put(fields, :compiled, EEx.compile_string(raw))
    )
  end
end
  • Since we’ll need templates to create questions, let’s add an alias to make it easier.
defmodule Mastery.Core.Question do
  alias Mastery.Core.Template
  
  defstruct ~w[asked substitutions template]a
end
  • These are the things a question needs to be able to do:
    • We need a constructor called new that will take a Template and generate a Question.
    • We need a function to build the substitutions to plug into our templates.
    • As we build substitutions, we’ll need to process two different kinds of generators, a random choice from a list and a function that generates a substitution.
    • We need to process the substitutions for our template.

lib/mastery/core/question.ex:

defp build_substitution({name, choices_or_generator}) do
  {name, choose(choices_or_generator)}
end

defp choose(choices) when is_list(choices) do
  Enum.random(choices)
end

defp choose(generator) when is_function(generator) do
  generator.()
end

Functions are Data

  • Anywhere you can pass some data as an argument, you can pass a function instead.
  • The BEAM even serializes functions, just like other types.

Joe Armstrong, one of the creators of Erlang, used to say we’re always taking the data to the code, which is really hard, when we could take the code to the data, and that’s much easier.

Name Concepts with Functions

  • When we have a concept in our code that needs a description, it’s tempting to reach for a comment.

    • Instead, think about whether there’s a way to name the concept with code.
    • A new variable or a function with a descriptive name is better than a comment because those concepts get checked by the compiler and comments don’t.

  • Easier if you have single purpose functions

Compose a Quiz From Functions

  • as programmers, our first job is to be understood – whether you’re communicating to your future teammate or future you.

Build Well-Named Functions

  • programming is about naming things well
  • don’t be afraid of long names
def tax(amount, city, state, sku), do: ...

vs

def compute_cart_tax(amount, city, state, sku), do: ...

vs

def compute_cart_tax_in_cents(taxable_cents, city, state, sku), do: ...

Shape Functions for Composition

The pipe operator is one of the best things about Elixir:

  • Try to string together a pipeline of transformations using |>.
  • Fallback to with/1 when you need to embrace failure.
  • To shape code that’s difficult to compose, use tokens

When functions in your module also return the module’s struct, you’re built to pipe. Then complex multipurpose functions break down into pipes of single-purpose functions.

Use Tokens to Share Complex Context

  • Think of a token as a piece representing a player on a board game.
  • Plug.Conn is a token
  • Ecto.Changeset is a token
  • Pipelines of functions transform these structures, tracking progress through a program.
  • Our Quiz is a token. It will track the composition of new quizzes and track a user through answering questions.

Build Single-Purpose Functions

  • Building a single-purpose function to add templates to a quiz makes sense:
def add_template(quiz, fields) do
  template = Template.new(fields)

  templates =
    update_in(
      quiz.templates,
      [template.category],
      &add_to_list_or_nil(&1, template)
    )

  %{quiz | templates: templates}
end

defp add_to_list_or_nil(nil, template), do: [template]
defp add_to_list_or_nil(templates, template), do: [template | templates]

We can beautifully generate a new test like this:

Quiz.new(title: "Basic math", mastery: 4)
|> add_template(fields_for_addition)
|> add_template(fields_for_subtraction)
|> add_template(fields_for_multiplication)
|> add_template(fields_for_division)

Each step moves our token with a simple transformation.

Build at a Single Level of Abstraction

Each line of a function or method should be at the same level of abstraction.

Sometimes, code written to a single level of abstraction is longer.

It’s worth it because the most complex logic is what we’re optimizing

Move Tokens Through Transformations

Keep the Left Margin Skinny

You can tell a lot about a programmer by scanning code. For Elixir, this is especially true. When scanning Elixir, look for:

  • long pipelines
  • short functions
  • skinny left margins.

Skinny left margins mean decisions are often made in pattern matches instead of control structures like if, cond, and case.

Skinny left margins make single concept functions much more likely, and simplify tests.

Try Out the Core

  • IEx is a great tool to sanity check our code as we go.
3 Likes