Over the last few months, we have been testing the API server’s ability to deploy several services. Personally, I have launched deep learning related services, and the server has processed a lot of data.
There are many myths about API servers on the internet. Problems with API servers can get even more complicated when your service uses particular environments, for example: WAS, Cloud, DB, ect. Therefore, the reason for writing this post is that you may have encountered or solved performance problems in a different way than what you have learned while you received your degree.
Before we continue, I have to explain our API server’s testing environment. I think that these factors also affect the server because there are some services and infrastructures that are already built into the companies of many services. Here are the environments we tested.
- Tomcat 8
- Mongo DB
- Spring Boot 2.1.0
- ElasticCache r4.large
- AWS EC2 r4.large * 2
- nGrinder (perfomance testing tool)
My company has 300 employees and most of the services use this stack of programs.
As we went through the performance tests, we realized the following three facts.
1. Using external components will slow the entire performance of API servers
Based on the experiences got from the tests, the biggest impact on API servers is the number of calls to external services. If your service calls external components or services, you should reconstruct it. Sometimes changing the data structure, or making major architectural changes to the service, may be necessary to reduce its number of calls.
If Redis or mongoDB has a replication function, applying read from the ‘slave prefer option to the read operator also helps improve performance by distributing the load of the read and write operations.
- If you are using MongoDB, apply the secondary read preference.
- If you are using Lettuce(redis driver), apply the slave read preference.
Lastly, if data changes do not happen frequently, I suggest that you apply a cache.
2. It is difficult to optimize reactive frameworks for production
You can find many posts about improving API’s performance by using a reactive framework, for example, Project Reactor and RxJava. We tried to adapt this framework. Reactive stack is well known for processing many requests at the same time. For instance, if the API service includes some IO blocking, it should help programmers fix the complexity and difficulty of concurrent programming.
However, when we changed the DB driver, the Redis driver went to the reactive library, and our API server’s performance was severely reduced. In fact, the result of the performance test was much worse than traditional request-per-threads model on our environment. You can see our results at the bottom.
We had to study these results. Perhaps we did not see the performance improvements that we wanted to because the infrastructure and the underlying stack had to be transferred to the supported environment.
In other words, the performance test results may be different than what you were told about reactive frameworks. The only way to be sure is to try it yourself.
3. Check out AOPs
Spring uses AOP in several components, and its logic is executed whenever there is an API call. If the bottleneck is found in Spring itself after profiling, you should look at the AOP component and an annotation. In particular, performance degradation can be seriously incurred if many time consuming operations are performed while recording logs. These include random value generation and continuous object creation.
I noticed that AOP spent most of its time on tomcat. This bottleneck resolved after removing time consuming methods while logging.
By the end of these several tests we got the results which are shown below. Through several modifications of the source code, we were able to achieve improvements that satisfied our company’s performance standard.
Conclusion
Bottlenecks are not always obvious. A broad range of observations and modifications are sometimes needed to find a solution. These include changes to the source code, configuration, and infrastructure. It will be helpful to carry out your own test with a well structured test plan.
