Book Club - Concurrent Data Processing in Elixir

Concurrent Data Processing in Elixir is now content complete and I finally found the time I’ve been looking for to dedicate behind reading the latest Elixir books. The Programming Crystal book club proved to be as effective as it was fun; I’d like to do the same for this one.

Over the next few days I shall be reading the book and sharing my experience with everyone, along with any “code as side effects”. The format will certainly be different from the Crystal one, as I already am experienced with Elixir, but will surely have much more codes produced.

I will start reading it from tomorrow and will be thrilled to have any co-readers to talk with!

8 Likes

Corresponding tweet for this thread:

Share link for this tweet.

1 Like

Looking forward to this one as well Mafinar - I definitely want to read it at some point myself!

I’m sure @svilen will be interested in hearing your thoughts too :nerd_face:

3 Likes

Going to read the first chapter today. I spent all my i-learn-as-i-please coins behind OCaml and Crystal in the past few days.

Did a quick scanning though. I wholeheartedly agree with the table of content. You start with Task, then slowly expand those capabilities with more capable boxes. I never got to work with (or even look into) Broadway so there’s that.

Task is one of my favorite things in Elixir. I look forward to read that chapter later today!

3 Likes

I just bought it, but it might take a me a little while to start. I have to finish the current project I’m working on first.

3 Likes

Chapter 1 - Easy Concurrency with the Task Module

Off to a great start. This chapter tells us ways to use Task. I loved how the chapter is organized, especially in terms of assumption of reader experience. As it starts, the author nicely explains the things like, how to start a REPL, create a supervised mix project etc, then after a great explanation of Synchronous vs Asynchronous tasks (I never get tired of those, they always enlighten me somehow) and Process, we get a very nice explanation and use cases for Task. And it ends with Supervisors! Everything you need to know should you attempt to learn this useful module of Elixir!

Two of my favourite functions: yield and async_stream were there! The explanation of linking and supervision too was pretty satisfactory. At the end, there was a section was dedicated to the “Let it Crash” philosophy, and thank you- WE DO HANDLE ERRORS! it’s just that our defence mechanism is different.

Most examples of this chapter was designed around solving a real-world (ish) problem, not just random “Hello World”-s and “Good bye Reality”-s, so each example and improvements had context and story, this is something I really like about code examples, you get to relate and often feel like, “Okay, so what if I did this…” and then see that getting answered in the next example. I wish the results were depicted via test cases but I also understand this is a personal preference of mine and is not shared by many people, so that’s not even a nit.

All in all, a great chapter that makes me wish I had it handy when I was learning about Tasks, Agents etc (Agent’s not covered in this chapter but I learned and used them together so…).

5/5

6 Likes

I will be reviewing the next two chapters during the weekend but I did skip chapter 2 (for now) and moved to chapter 3 and I must say, this book explained GenStage wonderfully. This book is great. Larger discussion and queries will follow soon but just wanted to post in praise of the awesome explanation style of @svilen

4 Likes

Good highlights @mafinar and I will be writing my highlights to the first chapter and share here after…

5 Likes

Back after a long slumber. Been a little busy with stuff. Going to review Chapter 2 tomorrow. Good thing I skipped it earlier :slight_smile:

3 Likes

Chapter 1 - Easy Concurrency With the Task Module

  • Multi-tasking is the concept that one can split their attention into more than one or more activities simultaneously; the concurrent execution of multiple tasks (also known as processes) over a certain period of time.
  • Multi-threading is the ability of a central processing unit (CPU) (or a single core in a multi-core processor) to provide multiple threads of execution concurrently, supported by the operating system.
  • Multi-core brought parallelism and allowed tasks to run simultaneously; this architecture enabled concurrency and parallelism by supporting two or more CPUs on a single machine.

First, running code on a multi-core does not automatically make it efficient. The language needs to handle concurrency and parallelism for its own, instead of demands to an external service or solution, and the Erlang Virtual Machine(BEAM) has that for its own.

And it will use the Open Telecom Platform(OTP) Behaviours and Concepts to build concurrent and fault-tolerant applications, recover from failures, use back-pressure to deal with limited system resources, and how to handle errors, and prevent crashing.

It talks about the difference in run code synchronous and synchronous, and identifies the processing data when it makes sense to run each of them.

Task Module

It will learn how to start tasks and different ways to retrieve results.

The entire chapter will mostly show examples to use the public functions in the Task Module and how to start and retrieve the data after finishing the execution. And it will keep these in a roll and show us how to prevent, or better choice depends on our necessity.

Most cases have a timeout to run the Task and change the time or set it as an infinity. And show the %Task{} struct, like:

  • Owner - is the PID of the process that started the Task;
  • PID - is the identifier of the Task process itself;
  • Ref - is the process monitor reference.

After the Elixir 1.10 version was implemented, the async_stream function to create the entire stream pipeline process to not just run asynchronously but laziness, and has some optional parameters, it could be passing, like:

  • max_concurrency
  • ordered
  • on_timeout

Process link, no_link, and monitor:

  • Link - a linked process will be linked to another process; when this process dies, it will die the process related.
  • No Link - a no-linked process when died, die alone, and don’t tell anyone.
  • Monitor - a monitor process will monitor some process; when the process going die, it will notifier the process watching him.

Task.Supervisor

It will monitor/supervise the Task was starting in your application, prevent these tasks from being linked to your application process, and die together with some Task processes.

And explain how to use the Task Supervisor and how to add the Application Supervisor Tree.

It shows the many ways to the specification child process.

How to isolate crashes to specify the strategy to restart the child processes, and has three different values, like:

  • :temporary
  • :transient
  • :permanent
6 Likes

I really can’t read PDFs, so just got handed the dead tree edition. It feels surprisingly light!

I am continuing my reading of this book, starting with Chapter 2 (Not skipping it anymore) and with more detailed experience.

Super excited!

5 Likes

It looks great Mafinar! If it’s not a huge book I will have to move it higher up my reading list! :003:

2 Likes

~150 pages packed with useful information.

3 Likes

How many Elixir books have you read now? Perhaps you could recommend them in order to be read? I really need to get back into reading…

1 Like

I have read most of the books on Elixir that was published by Manning and PragProg, except for the newer ones like Nerves (Not into that domain), LiveView (I have the book bought though).

Here are my top 3:

  1. (and always will be number 1) Elixir in Action
  2. Phoenix in Action (Very underrated book. And it helped me learn Phoenix and I loved the way Phoenix was introduced)
  3. Designing Elixir Systems with OTP and Metaprogramming Elixir (It’s a tie)

I do dream of writing a book on Functional Algorithm Design with Elixir (Or something fancy along that line) in a decade or two so naturally (and shamelessly so) that book will be my number 2 when (actually a big if) it gets published :slight_smile:

6 Likes

Number one for me will be always Elixir for Programmers book or the Elixir for Programmer 2 video course. This was the one that made the click to switch from OOP to FP.

Also I love the way that @pragdave organizes is code, because it resembles my Resource Action Pattern approach.

I also love the Elixir in Action book, but just as a second book to read after the PragDave one.

5 Likes

I hadn’t read Programming Elixir book. But heard a lot of good things about it.

3 Likes

I started reading Chapter 2 today. First half of it right now (Before the jobber app began).

I was thinking of mentoring a few folks on Elixir, at least to keep my Elixir claws sharpened since I don’t do that at work. I would say this chapter did a really good job explaining GenServer. And while I just skimmed through the latter half (aka Jobber, a GenServer based job scheduling app that touches DynamicSupervisor and Registry a little) I would be keen on reading it with a pedagogical mindset.

This chapter will be good material if one is equipped with the GenServer section on Elixir Official Guide and Elixir School, at least the first half. The latter half would need a little more intuition around supervisor. But in this book’s defence, it is targetted for folks with some Elixir experience.

I will finish the latter half of Chapter two tonight and will be back with a more detailed review.

3 Likes