Home

What are some best practices for building applications at scale?

When building applications at scale, we need to understand that we are not building for the scale of today. We have to plan for the scale of tomorrow. We are making our platform future-ready by identifying disruptive products and services we can provide to our merchants, but people usually just follow whatever the current hype is in the technology world. When you keep seeing a new buzzword, eventually you feel like you’re out of sync with current technology if you’re not using it. I hear a lot about AI/ML, and ML is definitely in use here, but you need to have specific use cases for AI. And you can’t implement a design pattern just because that’s the trend. You have to figure out what the right design pattern is based on your requirements. Building applications at scale also requires careful thinking about how you can create reusable components. You end up building a lot of components if you don’t make reusable ones, and all of them require testing. Every time you create a component, you have to test it for the functionality, as well as for the non-functional aspects, like the scale security. You also have to put a lot of focus on the architecture of what you are building, because the traditional way of thinking may not work. When we are designing a solution, we often try to fit the end result into the technology we have, rather than defining our end result first and then fitting the technology to that. It has to be reverse-engineered. Otherwise, you will have to change the technology when your requirements change. It is also important that you leverage the tools that are most relevant to that problem, rather than going with what you already know. If the architect has more experience with databases, for example, the solution ends up being inclined towards databases. That is a hindrance, so we have to remove our bias toward the tools we know when building applications for scale. We need to put the end result into perspective, establish what is required, architecturally build that, and then we can choose what technology will solve that particular problem, whether we have used it before or not.

69 views
2 comments
2 upvotes
Related Tags
Anonymous Author
When building applications at scale, we need to understand that we are not building for the scale of today. We have to plan for the scale of tomorrow. We are making our platform future-ready by identifying disruptive products and services we can provide to our merchants, but people usually just follow whatever the current hype is in the technology world. When you keep seeing a new buzzword, eventually you feel like you’re out of sync with current technology if you’re not using it. I hear a lot about AI/ML, and ML is definitely in use here, but you need to have specific use cases for AI. And you can’t implement a design pattern just because that’s the trend. You have to figure out what the right design pattern is based on your requirements. Building applications at scale also requires careful thinking about how you can create reusable components. You end up building a lot of components if you don’t make reusable ones, and all of them require testing. Every time you create a component, you have to test it for the functionality, as well as for the non-functional aspects, like the scale security. You also have to put a lot of focus on the architecture of what you are building, because the traditional way of thinking may not work. When we are designing a solution, we often try to fit the end result into the technology we have, rather than defining our end result first and then fitting the technology to that. It has to be reverse-engineered. Otherwise, you will have to change the technology when your requirements change. It is also important that you leverage the tools that are most relevant to that problem, rather than going with what you already know. If the architect has more experience with databases, for example, the solution ends up being inclined towards databases. That is a hindrance, so we have to remove our bias toward the tools we know when building applications for scale. We need to put the end result into perspective, establish what is required, architecturally build that, and then we can choose what technology will solve that particular problem, whether we have used it before or not.
6 upvotes
Anonymous Author
I got a bit confused – is focus on 'buildings applications at scale' or 'building applications that scale'. Both are slightly different, and I would like to distinguish. By 'building applications at scale' I mean - 1. You have a large application to be built, with a large team as fast as possible. 2. You have a lot of applications to be built with a large team, and you are churning out applications at scale.  By 'building applications that scale' I mean the application is built for horizontal scaling. In other words, a 'cloud native application'. May take on 'building applications at scale'. 1. Divide into micro applications. Such that each micro application should not need a team of more than 20 odd developers.  Build teams which can work concurrently. 2. Obviously, micro applications got to interact, so interfaces must be defined clearly (becomes critically important as concurrent teams are working). Adopt API first approach. Define interfaces and interactions between the applications using API. You may want to use api blueprint (https://apiblueprint.org/) to define the interfaces and use the likes of apiary (https://apiary.io/)  to host API mocks. This reduces friction between teams and requirements of communication are well documented. Inter-application seams become more seamless. 3. Needless to say, this kind of development essentially indicates a microservice-based architecture.  Each application is developed as a microservice with a definite set of requirements. Development of such microservices can happen fast and concurrently.  Advantage with microservices is also that deployment time reduces drastically.  4. Microservices for back-end is well-understood design pattern, however how about front-end. If it is one big application, it's difficult to divide UI development like microservices and still achieve a coherent user interface. It's a challenge. However, do check out design pattern of 'Micro Frontends'.  5. I am not a great fan of reusable components. If the requirement of the component in itself is reusable, then the component automatically becomes reusable. For instance, a component which establishes a connection to cache service. This is naturally reusable component and does not change service to service. Generally, when we try to build a re-usable component, we spend a lot of time in generalizing it so that it fits in various situations. Analysis becomes paralysis. Eventually, the component becomes heavy and buggy.  6. Heroku team had come out with 12 Factor application concept. Kevin Hoffman adds few more to it in 'Beyond the Twelve-Factor App' for building applications that scale on the cloud. Excellent suggestions. I believe with 12 Factor++ concept we can build applications that scale and also build applications at scale (by adopting API first, microservices, micro-frontend, and automation).
3 upvotes