Turns out this is a bug with the mix task, it simply generates the wrong code for umbrella phoenix apps:
With the help of the wonderful people at Elixir’s Discord chat, we raised this ticket.
Hopefully, it will eventually be fixed.
In the meantime, if you are creating a Phoenix umbrella app, you can try the theoretical workaround provided in the ticket created. However, since I am not sure this is the way the Phoenix team is going to go with fixing this, I would personally still wait for their fix, or at least for confirmation they are going to implement it this way.