Interview with Jonn Mostovoy Co-Founder at Serokell

Hayden - Nice to meet you, firstly I’d like to thank you for taking the time to complete this interview, I really appreciate it! I’ve taken some time to research your background & experience, as well as Serokell, but for the sake of context, would it be ok for you to introduce yourself & the company in your own words?

Jonn - Thank you for having me! I started programming because in my school there were a lot of people who were attending an extracurricular club called “fortech”, which was later renamed to “progmeistars”. The first language I used to build real systems was PHP, bundled with easyPHP bundler for Windows. I started selling my software while studying in high school, then got hired into a software development company https://very.lv, which was making a social network for professionals for the Russian-speaking market. We had to scale from zero to tens of thousand unique users to a million. It was an instrumental experience in industrial programming. I first started learning Haskell while at very.lv, because my colleague looked at my PHP code and said that I should check it out. Afterwards I worked in banking software using C# and Linq, which enabled me to use some of the aspects of functional programming in an industrial environment. When I got tired of the “bloody enterprise”, I switched gears to startups. Having a decent experience with solving a wide range of problems in Erlang language, I caught a wave of mass exodus of node.js projects switching to Erlang, thanks to Facebook hype. I managed to position myself well in the market and that’s how I started to get my first senior / tech lead positions. Then, over the course of three years, I saved enough money and practiced enough Haskell to implement my dream. With a friend of mine with whom we met on #nixos channel in IRC, we have co-founded a Haskell consultancy and an outsource company called “Serokell”. We have implemented several interesting systems: a secure linux distribution, a bitcoin exchange, and then we got hired by IOHK to implement the settlement layer of their cryptocurrency and bootstrap their genesis block. It was a ton of fun and we’ve done it successfully under quite a time pressure. Now our primary focus is biotech and data science, but involvement in blockchain projects still remains rather great. We regularly perform security audits of smart contracts and develop a huge chunk of the Tezos ecosystem -- from Tezos Agora (the main coordination website for everything Tezos, with blockchain integration) to ergonomic programming languages for Tezos Smart contracts. As you can imagine, Erlang and Elixir are very much used by us and me personally. Our product, https://pont.ee, has an Elixir backend, which coexists with Haskell workers, and I have recently implemented a verifiable credentials framework in Elixir as a pet project.

Hayden - Thank you, that’s a great introduction! I’ve read some of the case studies on your website, it seems you have some recent success stories of building successful crypto-focused platforms in Haskell, you may have seen our interview with Kamil Skowron in which he discussed building a Crypto engine in Elixir, could you tell us a bit about why you opted for Haskell? How do you feel about approaching a similar project in Elixir? Jonn - Yes, I’ve seen the interview! It’s nice and succinct. Not many know, but the first version of Cardano settlement layer could have been implemented in Elixir! That said, most of our projects are full stack, and especially something like CSL, where we want to run the node, the wallet middleware both on servers in headless mode, and on the clients’ laptops with UI attached. After performing a TRL review (technical readiness level), we have decided that Haskell would be a correct choice for such systems’ topology. We also took into account the fact that the volume of purchased tokens by the time we have started the development was already insanely huge, so compile-time guarantees for the ledger state maintenance was amongst the key factors. That being said, we’ve participated in amazing projects that use Elixir in the blockchain space, for instance, we’ve helped to optimise and refactor POA’s Ethereum block explorer! For things like that, Elixir and other BEAM languages work really well.

Hayden - Interesting to hear the thought process behind choosing a specific language.. seeing as you offer a range of functional programming expertise as a USP at Serokell, what does your criteria generally look like for hiring?

Jonn - It’s a tough one! We rarely look for unicorns, we normally hire Haskellers who can learn and hope they will be OK with working on non-Haskell projects. We also hire polyglot developers who are willing to read and write some Haskell, this way we can onboard customers that need, say, Rust or Go easily. Finally, as I mentioned, https://pont.ee is built with Elixir, and since we also use it as an internal tool, we hire BEAM developers to develop it and that allows for rotation of Elixir developers to work for our customers when such need arises. Our HR department is always busy and they are heroes for dealing with the amount of technologies we use daily. Hayden - How do you assess that in an interview process? Are there any challenges related to doing that remotely?

Jonn - We have a system of automated and semi-automated test tasks, carefully tailored to our business needs and heavily inspired by Matasano hiring strategy. The interview process then is a dialogue between the tech lead that shall be responsible for the new team member and said team member to figure out their preferences and understand the best fit for the new team member. We are remote-first (essentially, “remote” as in GitHub 1.0) since conception. Nothing is an unbeatable challenge to go remote for us. As a matter of fact, I met the co-founder of Serokell for the first time when we were submitting papers related to starting the company in Estonia. That being said, we know that some things about remote working cause friction, but we help our team members to cope with it.

Hayden - I noticed from your Github that you have been interested in Elixir for some time, how were you introduced to it? What’s your overall impression of the language, compared to others you’ve worked with?

Jonn - I have been introduced to Elixir by Yurii Rashkovski, with whom we were working as contractors for the same company all the way in 2012/13. We figured we’d use Elixir for some of the work we were doing back then and I really liked it, especially the metaprogramming bit. People who were doing metaprogramming in Erlang know how tricky it is to get parse transforms right, but with Elixir, metaprogramming is just code. The fact that you can have a simple for loop that generates definitions from, say, a CSV file is mind-blowingly amazing! Elixir also had a more normalised and modern standard library, so it was really a no-brainer to start using it. I would say that Elixir in 2012 was in a similar position as Gleam is now, so I’m very hopeful for Gleam, the statically typed language on top of the BEAM virtual machine. Later on, when I got hired as a developer in Yurii’s cryptocurrency exchange SaaS startup, I also got to contribute some patches to Elixir stdlib. Current pretty printing implementation is based on my initial implementation of this feature, for example. Fun fact, the module used to be called wadler.ex because the algorithm for pretty printing was based on Phil Wadler’s improvement of John Hughes’s document algebra. My overall impression of Elixir is that it in a stellar sweep addressed most, if not all, pain points of writing in Erlang. It’s overwhelming success is very warranted. Big part of it is José's social skills. As Joe Armstrong said during a “fire alarm break” at EUC 2013 workshop day, instead of imposing his own vision and driving, José enables contributions and acts as a moderator for a prolific community of contributors. Of course, I’m paraphrasing… Joe certainly had a way with words.

Hayden - Love hearing the quotes from Joe, he was a genius that was sadly taken too soon. It seems he anticipated the success of Elixir, what’s your take on the popularity of the language since you first discovered it? What do you think is in store for the future of Elixir?

Jonn - Actually, we’ve recently sat down and talked about it at length with Saša Jurić, but to sum it up, it’s great that it’s very popular these days. The only drawback is that seemingly some people don’t understand the underlying concurrency model and the ideas behind the OTP. I have a feeling that Elixir is such a great language with such a vast it becomes non-essential, somehow. I think that the future of Elixir is bright and I expect both data science and machine learning to come into play big time and web-facing application servers to improve by reducing persistence boilerplate and making stuff like LiveView a default with low entry barrier. I would love to see toolchains to make native applications in Elixir, but that feels a little bit science-fictiony right now. Finally, I think that ergonomic integrations with Gleam are going to be a major selling point for using Elixir as a glue for complex systems with workers written in Gleam, but that’s a pure speculation.

Hayden - I’ll be sure to check out that interview with Saša, do you have any recommendations for someone that is just starting out in Elixir?

Jonn - At a risk of sounding boring, I would say that if the goal is to be top-tier Elixir specialist, going through Learn You Some Erlang first and then learning how the underlying concepts translate to Elixir by working through Saša’s “Elixir in Action” is the best trajectory. That said, I personally spend quite a bit of time on elixir-language slack asking ridiculously silly questions about Phoenix and Ecto, so an alternative trajectory would be to explore the concepts that are relevant to Phoenix on demand, as you build your first application. I don’t have any book recommendations, but if the person has the time, working through Phoenix, Ecto and -- perhaps -- postgrex documentations and studying the code, should be good enough.

Hayden - Great advice! Can you tell us about any cool projects that Serokell have coming up? Or is it all top secret?

Jonn - Sure! As I’ve mentioned, we’re working in biotech right now. We’ve already implemented an awesome tool with a modern UX that allows scientists and other stakeholders to be more effective while choosing which versions of active compounds to pursue in cancer research. It’s open source and we’re entering phase two of its development right now, to make it even better and more convenient for the stakeholders to use. We’re also expanding the scope of our projects to other aspects of biotechnology engineering.

We’re also rebooting our on the Indigo programming language on a mission to make it truly the most secure and ergonomic way to implement Tezos smart contracts. It’s already rather capable, but only its “smaller brother” called Morley is used in production due to maturity considerations.

On the Elixir side of things, https://pont.ee is completed and if anyone is interested in streamlining their remote work, we offer it as a service, including working on custom integrations and optional consultancy about remote work processes. And gosh, can we consult on that! Tens of thousands of hours were spent in Serokell thinking, working on and tailoring the processes aiming for performance and happiness of team members without compromise.

Finally, we have a lot of non-commercial projects. For example, we have a whole team that develops dependent haskell. This team is currently working on making it possible for type variables to be bound not only in patterns but also in lambdas. In Elixir’s terms, we already know how to bind types from `@spec` to variables and use them in function bodies, but not lambdas. Solving this requires building an alternative to ScopedTypeVariables compiler extension, which, quoting Vlad Zavialov, the head of compiler development department at Serokell “will establish the necessary foundation for further extensions to scope [that are] needed, and will be used in conjunction with the new quantifiers of dependent haskell”.

Concurrently, this team is also working on improving the dependency analysis of Haskell declarations to accept more all the correct instances, instead of just most of them. To learn more about it, check out this ZuriHac talk! Also, we are automating homework checking for a course that the co-founder of Serokell reads in ITMO, the leading University in the world as far as competitive programming is concerned.

Hayden - How would an experienced engineer go about applying for a job at Serokell, do you have any specific openings that you would like to share?

Jonn - We’re always hiring, so shooting an E-Mail at jobs@serokell.io with a CV and an informal motivation letter can never hurt. Here are our current open positions: - Polyglot engineer - Senior Haskell engineer - Site reliability engineer You can expect a single test task that can be done in up to eight hours and aims to model some part of work in Serokell, be it libraries and technologies used in case of Haskell / Rust test task or typical problems solved in case of Nix / SRE test task. Successful completion of said test task results in a single short interview, where you can ask any questions you have about us and discuss some technical subjects with a tech lead from Serokell.

Hayden - Perfect, thank you so much for your time & expert insight, I look forward to sharing this with the community.