Release after the year of development: fierce debug to the New Year’s clinking of glasses and 40 days of the search for the solution

Talk video

Talk presentation

Usually, the last working day of the year means people raiding the shops and making last-minute purchases of presents for their loved ones, liquor and ingredients for all kinds of winter holiday foods. But that is a normal people story — people who don’t release the infrastructure update, that had been developed during the whole year 2017, on the year’s final week. And that is exactly what we did. We had 3 new services, a new stack of video streaming platform, new team members, and new hardware of all kinds. Not that I didn’t realize that it’s a bad idea, but if you’re taking responsibility for the result and pull anchor, you’ve got to be ready to accept the fate prepared for you by an iceberg.

And so it’s 31.12.2017, two days of fruitless troubleshooting of failing streams are over, and you’re looking at Champaign in a glass and thinking how to dig into this black box. The troubleshooting took days more and ended in a non-stop one-week long Go vs Node hackathon, which resulted in the complete rewriting of video streaming core, which was found out to be the problem.