Exploring MACH Principles [eng]

Hi everybody. Nice to see the Ukrainian community, and let's start. Today we will talk regarding architecture with MASH principles. A couple of words about me: I'm working as a fractional CTO with US-based startups. It's actually, I think, five, maybe six years since I started working as a CTO. During this time, I changed several hats; I worked as an architect, as a DevOps engineer, as a cloud engineer. So, it was a different project, but in every project at every startup, my main role was related to architecture. So during this talk, I would like to share with you my experience from startups. And I would like to share with you my experience from the startup ecosystem. Also, I'm volunteering for the Ukrainian community, sharing my experience with the Node.js stack. I'm the lead of Google Developer Group Cloud Kiv and an expert in GCP.

So, I love and know how to make developers and businesses happy. And one of the reasons why I'm telling this is because architecture, which is the most complicated when you make a mistake, can typically be driven in the right way when I work as a fractional CTO. Also, I would like to share with you some of my principles. For example, any problem must be solved at the right level. And during this talk, I will highlight on different levels ideas for one or another principle. We will start, of course, from the business, which is typically on the top of the pyramid. We will investigate how APIs act as a layer of interfaces. These APIs, based on GraphQL, REST, or WebSockets, or whatever will be represented. On the next level, a deeper level, we will talk regarding code. And, of course, the last layer is our infrastructure. You might say that I missed the hardware layer because it's also important, but during this talk, we will be focusing on a cloud-native approach. So because of that, we think that we're not touching this level because it's not the responsibility of our team, and it doesn't make a big difference for splitting or not splitting infrastructure or networking and the hardware layer. As I said, I would like to share with you some of my principles. I demonstrated to you how, for example, I highlighted something on the right level.

But the idea I will show you during this talk is that your architecture requires clear principles. And much is one possible list of principles for your architecture. Maybe you will like it if you have the same environment as I do because, in startups, time to market is a key factor in decisions. So let's start with the first principle, and the first principle is microservices: Microservices as a mindset. Of course, I will not repeat for you all the ideas of microservices described in some human book because it will be obvious. If you haven't read this book, I strongly recommend you read it. But I will try to say that when you're working with your architecture at any state of your project, you can utilize the microservices mindset. Why? Microservices, in a simple way, are breaking down applications into small deployable pieces. And for a typical web project, you very often will do the same. You will split API and web part. If it's a mobile application, you will split your data in Firebase, and you will have a shippable artifact at the Apple Store or Google Play Market.

So let's check how, for the business mindset, this microservices focus is transforming. It's transforming actually into future thinking. We have modular design because we will split our system into different individual features. We will isolate these features because it's real microservices. We will know that we will be able to isolate data to be sure that data consistency is related to only one microservice. And moving this business thinking regarding web features, we will see big benefits. Because you will see prototyping and research and development as key factors in business decisions. But for that, you need to change your mindset and work with business not as an engineer but as a partner. And make on the table your thoughts.

So, here are some real examples. Typically, when I talk with business owners or product owners, I will ask, 'If you had only one week to implement this feature, how would this feature look?' It's a very good question because businesses typically want a fast time to market. To conduct user research, they will use a Figma prototype. You might think it's not crucial for architecture, but you would be wrong. This prototype actually demonstrates the user flow, which can be easily mapped to features and related microservices.

The next consideration is how we will represent this microservices mindset in our API. Our API represents our data and how we interact with it. If you follow the microservices approach, you will create documentation. Moreover, you will create a contract that you will not break. You will follow versioning and even establish a service level agreement. Typical languages for API contracts include Swagger, GraphQL, and API Blueprint. However, they are not human-readable; they are computer-readable. To make your API more accessible for your engineers, describe it in documentation. I encourage you to use Redocly, a cool Ukrainian product. They have an open-source Redoc with a similar UI, but Redocly is much more powerful. It includes not only a documentation page but also features like workflows and the ability to check that you haven't broken your contract. Give it a try.

Moving on, the next layer is code organization. With a microservices mindset, you will start with a monolith because the monolith-first approach is the de facto standard. However, the system you build will be organized in a modular fashion. When you need to remove a feature or scale related code, it will be easy for you to delete or move the required module to a specific microservice. As you evolve your architecture on the code level, you will see that this modular approach, this thinking in features, will be very easy to maintain and adapt to a microservices approach.

Last but not least, when you work with your code organization depending on the needs of your project, you can organize your code and structure it as a Lambda function. These three dots mean that there are a lot of options for how you can ship your code. I want to highlight the hybrid approach. Instead of using real microservices that are deployed and run in Kubernetes, you can just move something, turn off your functionality to Lambdas, and it will be good enough to create your application in a way that both developers and business will be happy.

Next is infrastructure. When we talk about getting into microservices, we need to highlight that third parties from a microservices mindset are the same entities. They form the basis of our infrastructure, much like microservices. Let's check, for example, that our microservices should follow some SLAs to provide business trust in the engineering team that we are following our ownership. Integration points are very similar to what microservices should have. They can be accessed via API or even via SDK. The main idea is that third-party services are actually part of your infrastructure. Yes, you don't own them, but you can manage them. Configure them in a way that suits your product requirements.

When we talk about third parties, I need to provide you with some examples. One of my favorite examples is using third-party services to create cool products. Services for authorization or user credentials, for instance, are important. I had a talk about that around two years ago at Fever Days. So, I encourage you, please don't recreate user management. It will be buggy and have fewer features than if you integrate with Okta, Firebase, Auth0, or AWS Cognito. There are also several newcomers on the market with cool functionality, but I haven't worked with them, so I cannot recommend them. All in all, it's definitely something you can trust to create your authorization. As a cool example, I had this spring is using for chat messaging. It's a typical feature for a mobile application when you need to add the ability to interact with users between each other and create some chat.

Instead of creating your own feature and writing all the required code, you can just integrate with GetStream. They actually have two products: Stream for your newsfeed and Chat for one-to-one or group communication. By using this third-party solution, you will definitely be able to implement this feature and receive feedback from your product owners in one day. Really, you need to highlight that all this talk will go from business to API to code and then to infrastructure. So, we're moving on to the next principle. What is hidden under the 'A,' but it's not architecture, is API first.

In this part of my talk, I've named it 'API First as Newsfeed,' and I'm going to show you how to do that. I believe that any cloud engineer or architect nowadays knows the 12-factor app. The 12-factor app is one of the shared principles for engineers, DevOps engineers, and backend engineers. But Kevin Hoffman made a very good addition to these 12 factors several years ago. He actually added three additional factors, and one of them is API first. This principle refers to development—putting the API in the first phase before you create any UI or consumer application. And by 'create,' I mean not only developing the code but also designing it. So, when you think about some feature, you start thinking from your API's perspective. This is a mindset changer. Similar to how microservices help you think about features, APIs help you think in the main language.

Because let's think, if you need to talk with the business in a shared language or the main language, the API is the perfect place for that. Not documentation, not product requirement documentation. There are these. The API is the right place for doing that. What benefits will it provide? First, this knowledge of how to make requests to your API. The business department will be able to retrieve or change data without any UI. Also, it will be very easy to talk with the business about term collisions or terms that are not very specific. During conversations with the business in the main language specific to APIs, you will actually talk in DSL, which we love when we use Python or JavaScript to describe our entities.

The next thing is API responsibility. When we talk about traditional development, we add API functionality on request. Or maybe we say that we don't have the capacity from the backend side and ask to implement it on the client side. As a result of that, much functionality does not work very well. For example, if we forget to add filtering in our API, then later, when we add it in a rush, we might forget to add indexes to our database, or even more, it will not be covered in the right performance tests. So because of that, I'm promoting the idea of API first. During implementation and design, think about pagination, filtering, how you will handle input validation, and even more. Think about data aggregation.

What do I mean by data aggregation? For example, we know that for some features, we will need to create business aggregations, such as how many steps I made every day. It will be very strange if the frontend or client, mobile client, needs to download all historical data and perform this data aggregation on the client side. So be careful and think about data aggregation. A very simple example is how many records do we have in the user order list? This typical count on the database is the responsibility of the API, definitely not the responsibility of the client. Moving on, and moving forward with this approach on the code level, first, I would like to encourage you: don't create manual clients for your API. Generate SDKs. If you follow ideas from the previous step, if you create your API with some documentation, in this case, you will be able to generate an SDK. For example, GraphQL has an amazing introspection tool from the GraphQL schema called GraphQL API Navigator. Also, this guy contributed a perfect solution. I even created a Zoom API SDK for use in my project. So even if you have a third party but don't have an SDK, and you only have an API, you can generate a client.

And instead of reading your documentation, use an SDK to interact with the API. The next thing I would like to emphasize and share is to use low code. For example, on the screen, you see Retool, a powerful tool for creating internal dashboards and admin systems. I don't think I will ever create custom internal tools with a technical stack typically used for frontend development, like React or Vue. Because Retool or similar solutions with low code allow developers to receive data via a low code solution and represent them with drag and drop. It shows perfect results for internal use. Let's move on to the next, and of course, it will be infrastructure.

When we talk about APIs, we should remember that we want to have several APIs. And at some point, we want to hide or protect an API via a specific key or whatever. In this case, an API gateway would definitely be needed to add to our infrastructure as it answers many questions. For example, if you use AWS and need to implement WebSockets, instead of creating a WebSocket server, you can create Lambdas to work with the WebSockets. It might sound strange because WebSockets as serverless for many engineers looks like an oxymoron, but it's not. The AWS API Gateway is responsible for keeping the WebSocket connection. Then in Lambdas, you can listen to events regarding connection, disconnection, or emit events. It's a very powerful tool. Another case of API gateway from GCP is APIG. APIG provides you with the ability to create a paid API and manage not only rate limits, as most API gateways provide, but also billing for the client. So, if you're thinking about creating a SaaS solution, define your API with an API-first approach, create documentation, deploy it, and protect it with APIG API gateway. And you're ready to start selling your SaaS and charge per API query. Who knows, maybe it will be your hot AI startup, as that's a trending topic nowadays.

Moving on to the next topic, cloud-native. It's obvious because I just shared ideas of using API gateways from two top cloud providers. Cloud-native means that applications are designed to run in cloud infrastructure. Of course, you can say it can be on DigitalOcean or Heroku, but you'll receive the most benefits if you run your application on the top three cloud vendors: AWS, GCP, or Azure. So, I recommend you read this book because so often during interviews or conferences, I hear that serverless is only about functions. It's not true. Serverless is about containers. And this book can prove this idea. Now, let's go to the first layer, the most important one: business. Using cloud services and third parties to implement business features is one of the key factors to understand whether the business is following cloud-native principles or not. Why is it important? Because cloud services provide a significant boost in time to market. Just a friendly reminder, when you're doing something on-premise and you don't have enough resources, you need to buy hardware, pay for it when it will be shipped, set it up, connect to the internet or internal network, do all these things, and only then will you reap the benefits of the work. With public clouds, you're able to eliminate this time-wasting. And because of that, CloudNative is one of the key decision-makers for businesses when implementing certain features. But we should not forget to get into fine-tuning operations!

Because if you make some trivial mistakes in the configuration of your cloud, for example, pushing your client credentials to some public sources, you will have some problems. But anyway, the last but not least, business feature should remember that instead of hiring DevOps and backend engineers, businesses should focus on hiring cloud engineers. Cloud engineers are individuals who know how to set up and manage infrastructure, including CI/CD, and implement features in this infrastructure. This is one of the most important changes during the last several years, where DevOps engineers are evolving into cloud engineers, and backend engineers are doing the work of cloud engineers, going to the next level on API.

In this cloud-native approach, you will, of course, be focused on accessing cloud services. In another way, on the screen, you have a typical illustration of how an API needs to sign a URL or create signed cookies to provide access to S3. Look, we don't host our protected files; we only generate an API-signed link, and based on that, the cloud will be responsible for verifying this signed URL. If it was signed correctly, provide access to the file.

The next thing I need to highlight that will be different with the right implementation of the cloud-native approach on the code level is actually looking at tracing and open telemetry. An entity that is part of the landscape of the Cloud Native Computing Foundation is one of the solutions that I strongly recommend using in your application. Even more, if you don't use all of that, at least you need to use tracing and explicitly demonstrate trace IDs in your API. Later, when you need to do troubleshooting based on this trace ID in your code and your logs, what you should collect and make right routing for storage, you are able to find logs specifically for this request. It helps a lot for bug reporting and improving your application.

I'm going to the last but not the least step. Cloud-native expects that you use infrastructure as code. Even if you use click ops, because not everything, including most third parties, can be configured via infrastructure as code. When we say infrastructure as code, of course, we mean not only Terraform; we mean Pulumi, we mean a lot of other similar tools. But Terraform is still the top one choice for managing your infrastructure. So, click ops is a solution that you should avoid, but if you're doing that, use Terraform import to manage your infrastructure as code.

I'm going to the last and the most not interesting point is headless. It looks like I am almost out of time. So, headless is actually an architectural approach where the frontend is decoupled from the backend, allowing each to operate independently. And actually, most nowadays applications are designed in such a way. Of course, Next.js, for example, going to page.js PHP array for rendering stuff on the backend side. But the idea of headless is actually not to say that we have an API and we don't work with the representation of the website. We're using some modern stack like React.

The main idea for businesses with this approach is that businesses create and own content because headless doesn't relate to content. Many businesses right now are transitioning from CMS without the headless approach or even from digital asset management to new platforms. The term for that is re-platforming, and it's much more powerful than refactoring. With this approach, when you finish data transfer, you will receive the benefit of a modern stack, and you will unblock using headless to create a repeatable user experience on different platforms because you will receive access via API. On this screen, you can see the most popular headless solutions nowadays.

I've worked with Strapi and Sanity, and I can tell you that the most interesting part is actually not in choosing one of the CMS but choosing a pair of static site generation and your CMS. Why? Because when you have content, and this content will not change so often, as we see in API-based applications like Uber, for example, when talking about some e-commerce, you will have a lot of pages without any change for months. And in this case, generating content is one of the things that you want to do to create your application in a perfect way. For that, you, of course, need to deploy on demand. And the trigger for your deploy will be not only you need in your CI/CD but also some changes in your content management system, headless content management system. So, following, we discussed all four principles. We checked how the implementation of each of these principles is aligned with the layers of your architecture. And this is the time for your questions.