Time flies super fast. We just realise that we have operated our business for more than two years already! Our service has been improved all along the time starting from small MVP service to the robust system that is currently handling thousands of concurrent connections per second every day with 99.9% uptime. Thanks to our engineer team for making this happens. I must say that it is super fun working you guys!
Although our techs are not that fancy but we really have learnt a lot through these years. And we think that it is a good time to start sharing our experience to the community from what we learn so far. Hopefully that the information we gave would be helpful for you all more or less.
Let us start our first engineering blog with the simple but important stuff like the technology stack.
Background of our services
Although there are a bunches of products that we have delivered to the market so far, but by overall, our main business is still to provide a tool to let brand or agency to create a unique kind of interactive rich media ads featuring new technology like 360°, VR, AR and 3D that perform better than the static ad.
Anyway we know that the ad hosting is a super heavy stuff. The traffic could easily bring down the server at any point of time if it is not prepared for that scale of traffic well. That’s why we also decide to help hosting all the content and also help handling the ad serving to make sure everything is working perfectly without a downtime. Only thing that the advertiser/publisher needs to do is to copy the ad tag provided by us and paste it to serve through any channel they want including through the standard DSP.
And here is the key challenging. Since we have to handle the massive amount of traffic, every service we provided have always been designed with scalability in mind. It is so crucial here to make the service as robust as possible handling any scale of traffic without a downtime or any delay.
With our current system, we can now easily serve the 360°/3D ad at 1 million ad per minute, and of course, at the reasonable cost!
This blog will reveal the technology stack that we are using right now as well as giving you a reason why we are using them. Let’s start!
Technology Stack Overview
A picture is worth a thousand words, right? Here is the infographic indicating the main technology we are using to make it easier to see the overall information.
Now let’s drill down one by one.
Since we are having a quite limited number of the engineering resources here. Whatever that could help us reducing the work, it is always worth it. That’s why we have decided to not doing the infrastructure layer ourselves but hosting our services on the existed IaaS and PaaS out there in the market instead.
Our computation servers are split into 3 parts:
- Main computation, for example, website, ad serving and stat collecting are running on Heroku.
- All the AI services are being run on Google Compute Engine because it provides the significantly better computation power than Heroku, and also slightly cheaper. Every service is containerized and deployed on the production with docker.
- Some part of the computation that could be split into a module and does not utilise the CPU much, we manage to run in on Google Cloud Run instead which could significantly reduce our cost (sub $10/mo).
Every single part deployed is scalable with a single click which is the main reason why we choose to use these techs in our stack.
Content Distribution Network (CDN) is one of the most important component in any online service these days, our system is also no exception. There are a lot CDN services to choose out there but at the final what we choose is Cloudflare for many reasons.
We have designed our system to leverage the CDN benefit as much as possible. Right now more than 97% of the incoming traffic are cached in CDN level which could help us reducing server cost a lot. And also, our users and customers are so happy with the serving speed from the CDN’s benefit.
Web Front-End Technologies
We currently have so many products launched to date. Each one is developed separately with different technology based on the requirement since we believe that there is no superman tech that fits them all. So before we start any project, we would do the feasibility study and choose the right tech for both short and long term.
For the web front-end (client side rendering), React is used to wrap web component into a virtual DOM which help us a lot of scaling the project without messing our code. MobX is used to do the state management though. jQuery is still being used in the legacy code but the we have not used it anymore for the newly initiated project.
And since our main product are rendered in 360° and 3D, WebGL is unavoidably picked to be one of our system’s core component.
Gatsby is being used to compile some part of website to a static html page to reduce the server load and speed up the load time. While Webpack and Parcel Bundler are being used to bundle everything in a compacted single file.
Programming Language and Framework
As same as the front-end technologies, there is no one language that could do all the jobs. We just need to pick one that fits the specific task best. Here is the list of language we are using:
- Our website and core service are developed with Ruby on Rails leveraging a lot of caching system in middleware level. Let’s say over than 90% are cached and only 10% of the traffics are sent through the controller architecture.
- Some server service that requires better CPU utilisation is implemented in Node.js since it could handle the traffic much more efficient than Ruby on Rails and help us reducing the server cost with significant amount.
- Some part of Node.js code is implemented in Coffeescript.
- We are also running couple of AI services. All of them are implemented in Python 3 along with Tensorflow and Pytorch framework.
- Some part of code that requires super heavy processing power is implemented in C++ 11 to utilise every single processing clock. These codes are also written with asm.js/WebAssembly compatible.
- We also provide the 360°/VR Player on Android and iOS. Those libraries are written in Java (Android) and Swift (iOS)
- Our Ad Network product is developed in Unity-native with C# language.
- CSS is mostly written in Scss.
- WordPress is chosen to run our blog (the one that you are reading right now). So we have to develop something using PHP from time to time as well.
So far although there are plenty of programming languages here but we could say that it is super normal. If you are a software developer yourself, you would know that it is true.
At the end of the day, the end user would never know or even care which programming language are we using. It is all about the quality and user experience that we serve to them that they actually could notice.
Pick one that fits with your team and also do the job that you want best and everything would be good!
Background Processing Framework
We use Sidekiq to do the background job for Ruby on Rails while Kue is being use to do the background job for Node.js.
Database and Cloud File Storage
RDBMS and NoSQL have their own advantages and couldn’t be replaced each other. That’s why both of them are being used together in our system.
The relational database management system that we are using is PostgreSQL 11 while the main NoSQL database used is Redis. MongoDB is also used in some small task which sharding is not even required yet.
When talk about the files, they were previously stored in Amazon S3 but due to the performance issue, they finally had been migrated to Google Cloud Storage since late 2018. And we could say that it was one of the best decision we have ever made so far! Slightly expensive than S3 but super happy with it tho.
We always have an internal metric to measure the performance of product we have launched. The main analytics service that we are currently using right now is Google Analytics. We also sometime use Mixpanel. It depends on which kind of metric are we going to measure.
Anyway, our ads statistic collecting system is implemented internally to make sure everything is working as we expected.
Sentry is used to do the error reporting on both server and client side.
Stripe is chosen to do the job. Reason? It is great!
Slack is the main tools we use to communicate between people in our company. We previously used Workplace by Facebook for a year but it did not work well. Or to be exact, it did not work at all… After we switched to Slack, the efficiency of collaboration has been far far better.
When it comes to video call, Google Meet is the major tools we choose to do the job.
For the project/task management system, we have tried multiple tools all along, for example, Google Docs, Google Spreadsheet and Trello. Actually every single one of that list worked pretty well but when the project and team was staring to grow, we need the more managed system. That’s why we are using (or testing) Asana, and so far it is pretty good tho. In this area, I think there is no the best tools that fits every single team. You just need to choose one that fits your team best. And the only way to know is to try it one by one. Which …, to be honest, we are still finding the best one for our team.
For the version control, of course, we use Git! Most of our codes are checked out on bitbucket and gitlab’s private repo.
Software engineering is not just about to “make it happens” but it is also about “with as less as resources possible”. And resources here is a function of “time to market”, “performance” and “cost”.
It is non-sense at all choosing the brilliant tech stack that take a while year to deliver your product to market. There is no chance that you would win over your competitor. And it is also non-sense at all to pay a million dollar for server cost to get 100K income, agree?
You just need to choose the tech stack that fits your team and the situation best. Focus a lot on business and adapt things with your engineering skill!
This blog just gave you an eyes on our tech stack which let us go through these years. Might not be the best tech stack compared to the others but yeah, we may not even have a chance to write this blog if we chose the other choice. =)
We just share ours. Now please feel free to share yours! Love to read!