How I Scaled a Golang App to Handle 1.5M Requests Per Minute
I’m a Golang Developer who loves writing Go code. I switch to Golang when we used to create directories like these:
$GOPATH/
│── src/
│ ├── github.com/
│ │ ├── username/
│ │ │ ├── project/
│ │ │ │ ├── main.go
│ │ │ │ ├── package1/
│ │ │ │ ├── package2/
│── bin/
│── pkg/Nowdays, we don’t need them and go.mod has made things much easier and better.
Alongside this journey I wrote many lines of code and sometimes I messed things up and sometimes I made things better.
Recently, I have started to realize that in the past three years I have been using certain rules as my foundational structure of my projects and It has paid off well.
I achieved around 25k request per second which I built the project bare bones and then my team extend it. I ran the application on Ubuntu 24.0 server with 8GB RAM and 6 Cores CPU.
Of course we will run most of our applications with horizontal scale and as microservices but this single instance made me think about the patterns I am observing in my workflow to create any new Golang project!
The application will handle Bank interest rates calculation and other financial use cases.
My project setup is: Gin, Query Builder, PostgreSQL, RESTful APIs and Not using any dependency injection package. Let’s see how we achieve this
1. Data and Database
No ORMs, Lots of Gain!
Yes, ORMs make things easy for us but they will use lots of type reflection and at the end the performance issues with ORMs is very noticeable.
At first I was using Squirrel for writing queries and the Golang SQL driver, but after some toughts I decided to write a Query Builder. Over time, I have improved it, and now we have a very fast way to execute database queries, with syntax sugar that is quite nice.
query, args := querybuilder.
Insert("users").
Columns("first_name", "last_name", "phone_number").
Values(user.FirstName, user.LastName, user.PhoneNumber).
Returning("id").
ToSQL()By using Query Builders, I have seen a performance boost of over 30% in our write-heavy application. (By the way, I’m using Prometheus, and Grafana for these metrics)
Async Patterns
Keep in mind that not everything needs to be handled at the time of the request.
In every application a good amount of task and workflows can be processed in the background and doesn’t need to make user wait until the process is finished.
Using an async processing pattern will allow us to reduce API response time.
Caching and Miss-Storm
Using cache is always a good choice, but handling the miss-storm of cache absent and ensuring fresh data is a more important topic.
We should Build a method to warm up caches instead of allowing user trigger cache build up.
One way to handle miss-storms is by using hard and soft expiration times. The hard expiration is longer (1 Day) while the soft expiration is much shorter (1 Hour).
We set both hard and soft expiration keys in the cache server and the application read data from cache server always checking the soft expiration time. If the data has expired, it will be updated witha reasonable margin like 30 minutes ahead and hits the database, So next request will receive updated version of data.
Another caching method is using Russian Doll Cache method. Russian Dolls are kind of interesting where each doll has another one inside itself. So we break our cache into smaller pieces and each one has a chunk of data which can be revalidated and updated separately.
Always Select What You Need
Another database tip is to always select only what you need from tables not all!
Selecting * can significantly reduce read query performance and If you need convincing, check out this article: “avoid select even on a single column tables”.
Be Aware of Your Database Capabilities
Database is very important and whatever database you are using, you need to understand their pitfalls and strengths. we have to know database pitfalls and pros to use them wisely.
For example, PostgreSQL offers many kinds of indexes and can even be a replacement to Elasticsearch.
Race Conditions
Race conditions are common in high-traffic systems, so we must be mindful of them. The solution is to use locks to protect shared data during operations and leveraging Go’s concurrency features, such as channels and goroutines to improve performance.
2. Cronjobs Are Easy to Use but Can Easily Bloat System’s Performance
Cronjobs aren’t inherently bad, but always consider that they can keep your system involved for an uncertain amount of time. Even beginner mistakes, like not handling race conditions or overlaps, can negatively impact the system.
In many situations, we can use a cronjob, but almost every case can be handled more efficiently with an event system, queue system, or etc.
3. Observability, Observability, Observability
Not having a good monitoring system and structured logs will obscure our sight.
We need to see what’s going on and how much time each part of code takes to run. Only by seeing the problem we can prepare and apply the solution. A black box is just for hiding things, not solving them.
4. Bonus
- CDN is a must, not a choice! Always use other people’s servers to serve your content faster :)
- Use Golang goroutines wisely by managing them through a supervisor and a combination of channels to handle the maximum number of running goroutines
- Database connection pooling is very important.
- Database indexing is crucial, but NOT everywhere
- Dont forget about Materialized Views and CTEs
If you like this article please Clap 👏 and follow me to get more of these Golang stories. Also read:
