Many projects rely on cloud infrastructure. Over time, as the infrastructure grows, so do the accompanying bills. This can often lead to the need for AWS cost optimisation.
Software Planet Group were recently approached by a client who ran an AgTech logistics management portal. The SaaS-like solution used a microservice-based architecture running on top of Amazon AWS. Their key areas of expenses were:
- Amazon EC2
- OpenSearch Service
- Amazon RDS
Other services in use included S3, EKS and ECR.
Because their daily bill reached $110, this resulted in over $3.3k in monthly expenses on infrastructure alone. As a result, the client wanted to determine if these expenses were both justified and efficient, which led them to task us with an AWS infrastructure audit.
During the AWS audit, our DevOps specialists thoroughly analysed the solution’s architecture — in addition to other factors — and subsequently came up with the following checklist:
- Shut down any unused services
- Remove unused resources
- Archive any resources that are not being used at the moment
- Review lifecycle policies
- Buy reserved instances for permanently used services
This resulted in a reduction in the monthly bill from $3,030 to $1,695.
How To Optimise an AWS-based Infrastructure
This initial checklist gave rise to the following actions:
1. Check if there are any unused EC2 instances still running (i.e. burning the customer’s money) and shut them down.
Surprising though it may seem, this actually happens quite often. Sometimes, teams will launch instances for testing purposes, before forgetting to shut them down. Other times, as modules and services become deprecated, their functionality is no longer available.
This was the case for the client, and so 4 unused EC2 instances were promptly shut down for good.
2. Remove any unused EBS volumes.
Similarly, when you no longer require an EC2 instance, it is important to make sure that it’s disposed of appropriately. When deleting the instance, any EBS volumes belonging to it should also be removed. If a PersistentVolume (PV) is employed — in our case, a dynamically provisioned piece of storage in the Kubernetes cluster — you should delete the PersistentVolume of any service as soon as the service is removed from the cluster. Otherwise, the storage remains reserved however unused, and Amazon will include it in the final bill.
3. “Archive” instances that are not expected to be used in the near future.
In order to safely archive an instance, you can take a snapshot of the EBS volume, disconnect the volume from the EC2 instance and finally delete it. Volume snapshots are usually cheaper than actual EBS volumes. For example, in the Ohio region, a gp2 volume costs $0.10 per 1 GB per month, while a snapshot costs just $0.05 per 1 GB per month. Five cents may not seem like massive savings, but when paired with huge quantities of data, this figure quickly becomes less trivial.
4. Archive snapshots of EBS EC2 instances.
On occasion, it may also be useful to store snapshots for a while. This can be done at a reduced cost when you choose to archive them. With that in mind, if you are at the end of your active development phase and you have a number of snapshots which might at some point become useful, but there are currently no plans to use them in the near future, then you should probably use the snapshot archiving function. The price for storing these snapshots will be lower. Just remember that when restoring them, you will need to retrieve the snapshots from the archive first.
Note too that there is a small charge for retrieving the data from this long-term storage: $0.0125 per GB/month of stored data and $0.03 per GB retrieved. You will also be charged for a 90-day period at minimum.
* — Price per GB/month of gp2 type storage in the us-east-2 (Ohio) region, as of 7-Oct-2022.
** — Prices for the us-east-2 (Ohio) region and gp2 type for EBS Volumes.
*** — Price per GB/month of stored data and $0.03 per GB retrieved. You are charged for a 90-day period at minimum.
5. Delete snapshots that are no longer needed.
While snapshots are cheaper, there is also no need to keep them for no reason. During the development process, they are often created to enable the dev team to quickly and securely return to the original state of the instance — should any changes they apply be unsuccessful. These snapshots are often forgotten. As a result, over time, they can also accumulate and increase your storage fees.
6. Identify if there are any detached elastic IPs.
There is a well-known trick with elastic IPs — if the elastic IP in your account is not attached to a working instance, because Amazon still “holds” this for you, you will actually be charged for it, according to Amazon’s current tariff. On the other hand, if the IP is attached to an instance, then you will not be charged any fees.
7. Set up life cycle policies for builds in the Elastic Container Registry (ECR).
By default, all builds that fall into the ECR are saved, which over time can cause excessive expenses for storing unnecessary information.
8. Change S3 storage type for certain files.
S3 offers different storage types that will vary in price depending on the expected frequency of accessing the data. If your software solution needs to store large files which will rarely be deleted, there could be a potential savings opportunity.
9. Configure lifecycle policies in S3.
Once again, when incorporating appropriate policies, it is possible to make some savings by deleting outdated files or automatically transferring them to a cheaper form of storage.
10. Check if changing from gp2 in EBS to gp3 could make additional savings.
When it comes to the Elastic Block Store, storage size matters. For a gp2 volume type up to 160 GB in size, it is possible to reduce the costs per use by changing the storage type to gp3, a move which will also come with increased performance. For gp2 type volumes over 160 GB, however, this is unfortunately no longer as straightforward. There are plenty of nuances to be considered, as in certain cases, performance may be reduced with a gp3 volume.
In terms of pricing, for example, in the Ohio region, gp3 costs $0.08/GB/month, while gp2 costs $0.10/GB/month. Note that the gp2 type is suggested by default when creating new instances and will often remain unchanged.
11. Buy reserved instances for permanently used services.
Reserved instances (RI) may be likened to a monthly gym membership — the less flexible your visiting hours, the cheaper the cost.
Reservation enables you to save up to 72% of the tariff cost on demand. The actual discount will depend on the conditions of the reservation. Some common factors that will affect the discount for all reserved services are:
- the period for which the reservation is made (1 or 3 years);
- the type of payment (no prepayment, partial prepayment or full prepayment).
There are also individual factors. For example, EC2 enables you to exchange instances of one type for another type in the future without having to go through the process of selling via the AWS Marketplace.
For the top 3 expenditure items — EC2, OpenSearch Service and RDS — our DevOps specialists, working alongside the development team, determined the resources which would be used in the long term. Based on the outcome of this analysis, they were able to identify the most optimal RI parameters.
The Results of Our AWS Audit
Step by step, our team analysed the platform’s infrastructure, whilst applying the above mentioned changes. This resulted in a reduction in the monthly bill from $3,030 to $1,695.
In the graph above, each column represents daily expenses. This shows:
- a gradual decrease in the total cost beginning on April 13th. The results were achieved by identifying items for savings without sacrificing project performance.
- Two spikes on May 20th and 25th, which were a significant surge in expenses as we purchased the Reserved instances.
- The purple colour shows the costs of EC2. EC2 Reserved instances — which cover part of the working instances — were purchased on the 20th. Their maintenance fee was charged once at the time of purchase. After the 20th, only EC2 instances that were not covered by the reservation were billed on the graph.
- To conclude the results of our AWS audit, a similar situation occurred on May 25th. As you can see, the turquoise colour represents the OpenSearch Service (a search project based on older versions of Elasticsearch and Kibana that was forked by Amazon). Note that there are no OpenSearch Service charges in the daily cost charts after the 25th.
We hope this case has shown the importance of AWS cost optimisation in stopping your infrastructure expenses from spiralling. A little foresight goes a very long way.
Need an infrastructure audit?
Our DevOps and technical specialists will help you identify the available options to reduce your AWS costs.
Request a Quote
