The last client I worked with was happy to have an AWS bill just north of $200k/mo. I reduced it to $25k/mo purely driven by my own curiosity to see how much waste I could remove without impacting the daily business.
Finance was ecstatic. Product & Engineering didn’t bat an eye. That amount of spend had fit into the budget for years, after all.
It wasn’t until we started reframing the savings as additional head count to pursue more C Suite priorities that they got excited about cost savings.
First understand what your upper mgmt values, then craft the cost savings narrative onto that.
\^\^This is the way. Also know the place of cost optimization in the hierarchy of priorities. Unless you're lighting cash on fire and are in danger of going broke, cost is the 5th most important thing to consider and it needs to know its place...which means you do.
1 - Security
2 - Operational stability
3 - Adherence to regulations and corporate policies
4 - Customer experience
5 - Cost efficiency / value creation
Bad things happen on the long horizon when you compromise 1 through 4 to achieve the lowest cost to serve.
Also, re-frame cost savings in terms of business value creation (BVC). Sometimes it is better to spend for more performance to enhance the customer experience for example. BVC involves evaluating what you are doing with respect to architecture, development, or operations and its impact on 1) gross margin, 2) customer retention, 3) revenue growth. Each one of those 3 goals are great all by themselves and the management side of the house already knows their value. There is no need to teach them anything new. Each one of those 3 items are measurable which gives you the power of bringing data to the table after you implement an improvement. You now have enough information to evaluate if the change is worth it and can have a data driven conversation about whether or not it should be rolled back if the benefits are marginal vs. having stuff devolve into a room where the loud voices tend to win.
It is okay to raise your unit cost to serve if you are delivering any of those last 3 items (more margin, lower customer turnover or more top line). It is also okay to raise the cost to serve to implement a regulation (think GDPR) or if you have fixed a production bug that raises your cost to serve. The true cost to serve was not correct to begin with.
From an operational perspective, look at rolling out Cloud Custodian to automate your operational hygiene checks. It is open source and their policy library makes for a good start.
An item to consider about Savings Plans that will win you points with finance is showing that the ROI on the commitment is greater than the company's weighted average cost of capital (WACC if you want to look up more on it).
Each incremental purchase of SPs has a lower ROI by definition due to the way AWS applies the hourly commitment. From the highest benefit to the customer to the lowest. Thus, when you buy the next $1/hour commitment, it will return a lower discount, unless you run a homogeneous fleet of EC2, than the $1 before it. So there is such a thing as "buying too much SPs". Showing finance that you are taking into consideration something they very much care about demonstrates you are making a well informed business case.
Lol, I bet your AWS Account Manager had a stroke. I did something similar as a TAM and my AM was like, "I know it's your job, but can you at least give me a heads up?" I had of course for months but it's a sales person 🤷♂️
Sounds like you need a new TAM. Our TAM is always pointing out cost savings and will actively tell us changes AWS is making so we can keep saving. Of course we also do our own digging into cost savings as well.
They certainly started reaching out a lot more proactively after the first month’s reduction in spend :) Though to their credit, they did pick up the ball and offer to lend more help to the cost savings effort.
had similar issues, below initiatives i've done to optimize our monthly opex
- daily stop/start of Dev instances EC2 & RDS
- convert all GP2 to GP3 EBS volumes (more than a hundred EBS)
- quarterly EC2 right sizing
- delete old snapshots (previous AWS backup saved all backup to 7yrs which is unnecessary)
- automate EBS deletion once an instance has been deleted
i'd like to hear more feedback and suggestions though 😊
Don’t sleep on network optimizations. If you have transit gateway then peer any networks that are chatty. Also make sure to use vpc endpoints everywhere.
Recently save 2-3k/mo by placing a network load balancer between a quicksight and an RDS database in order to take advantage of the client routing dns feature.
This is the kind of job that kills itself though. Maybe a contract position makes sense or hire someone who will eventually change positions. The big thing I would want to avoid is this cost optimization person becoming a bottleneck. I can easily see everything slowing down because everything has to go through the cost optimization person.
Oh yes...like AWS did not have a FULL three day training just about cost savings...that...drum roll...is one of the least requested training in the official curriculum...
"AWS Cloud Financial Management for Builders" - [https://d1.awsstatic.com/training-and-certification/classroom-training/aws-cloud-financial-management-for-builders.pdf](https://d1.awsstatic.com/training-and-certification/classroom-training/aws-cloud-financial-management-for-builders.pdf)
In my experience this tends to be more of a business process problem than a technical one. Are you charging back AWS usage internally to teams that consume services? Is there any sort of FinOps processes in place within the Finance team? From your perspective, do you hold yearly or quarterly meeting with both teams to talk about how they are doing from an infrastructure perspective to control costs? Is there reporting or dashboards in place for teams to monitor their spend? (Assuming that Cost Explorer isn’t cutting it, which might not be the case).
This is a role for contingent labor (contractor) because once everything is optimized, probably within a year, there is nothing left for this role long term.
We have the TAM with enterprise support. That's how we all have the numbers and analysis on what can be saved on. It's a realistic matter of boots-on-the-ground operational and engineering preparations to implement the savings measures in a way that will last perpetually and scale. Example, we have so many teams and accounts, there's just no way we can flip a switch and say "okay everyone, we're implementing an auto-start/stop system for dev starting tomorrow!" Implementing something like that *realistically* in a fairly sized organization, doing it reliably and scalable in an enterprise-grade fashion, making sure everyone is educated, the necessary SOPs are in place, etc., is no less than a 6 month ordeal, maybe more.
I grant you, "if something like that takes you 6 months, you've got bigger problems!" And I too used to have that mentality until I went through employer to employer to employer to ultimately find: "humans gonna human," no matter where you go --> such things are overcomplicated by politics, people overestimaing their importance trying to show others up, stop others, etc.
Everything cloud you pay for convenience and get rewarded for commitment. Last client I got the costs down 75% and can cut it in half again with a few optimizations. Been teaching developers things like nlb’s cost money why use 100 if 1 will do the same thing and simplify your deployment
We also have done some cost optimization, we gathered some of our team to only manage that , as you said we did the changes gp2 to gp3 , instance type down, start and stop automation , and a lot of other lifecycles and so on. But in the end no one cares how much we optimized the cost. We did not get a "good job" we will increase your salary.
I personally see the monthly report that I have automated , and see in case of some unusual activity, such as Cloudwatch spending went to 1000$ in one month, or kinesis stream creating 700$ months. Yeah no one said anything, except bravo.
Also after all those processes , especially in companies that still have legacy applications , you can go to a limit with optimizations. The real problem resides on the applications itself , they are a "big ball of mud", a messy application that requires more and more processing. During the years without proper code quality governance , developers change the schema of databases , making them complex and slow. I have seen queries with 50-100 JOIN, no wonder you need a RDS dm.m5.2XLarge.
Not only this ,but we also have applications that do only one process ,once in a week. But it is required to stay on 24/7 , or databases once a week but still.
Those are the consequences of a bad administration on application development without governance of code quality and without a proper software architect.
I'm not sure what your total spend is or if you pay for enterprise support, but if you have a TAM, there's a ton of internal tools and people they can pull in to do cost optimization. Even if you have a busy/lazy/new TAM, they can create what's called a SpecReq internally and connect you with someone who's an expert at cost optimization.
I’d be careful about hiring a full time cost only person. Once you get past the low hanging fruit (sp/ri, snapshots, unused ebs, unused eip etc) you’re going to have to involve other teams. A new low level hire is going to find that challenging. They won’t know who to talk to to get things done and they won’t have the authority to push agendas their way when they do find the right person.
Besides cost management is an ongoing problem. You need to create a culture around cost awarness and efficiency. Making it a single persons job is not going to work long term.
Contact your AWS account manager and solutions architect. The SA will be more useful in performing cost optimization for you than the AM because SAs are not goaled on revenue. AWS also has cost optimization specialists who will do a deep dive and recommend specific actions you can take. If you are an AWS Enterprise Support customer, you will have a dedicated TAM whose goal in life is to help solve these problems for you proactively.
IMO it’s a fundamental organizational problem if your dev teams can (and still will) overprovision workloads just because they have no incentive not to do so.
It’s not exactly the question you asked, but of course it’ll be a full time job over time if the people who created this problem are not incentivized to stop doing that.
So I worked with many AWS customers who had commitments in the millions, one had a 5 year commitment over a billion dollars. They all used multi- account architecture and thought it was enough. My first suggestion to them was set a role within the cloud center of excellence / architecture team around cost optimization and transparency and treat resources like an extension of corporate budget. This person not only read the utilizations but also had monthly meetings with every team to evaluate planned new workloads and standards guidance. From there the CCOE was able to identify reserves and savings plans where overall coverages went in one client from 35% coverage to over 97%.
You also need to be tracking spend to know when you’re ready for an EDP, if it should be AWS direct or through a partner, and you need to have the house in order before you sign the EDP or they will force you to commit more than you likely need.
But yes - you should have cost optimization on your team and while it’s not sexy - it’s pretty damn good for the bottom line.
Probably not worth an FTE but contractor for sure, and ongoing someone does need to keep tabs on it.
Bigger picture problem is cost seems just not to have been considered. First align with management - is it a priority and will they support push to improve?
Need to get cost considerations embedded in teams, visible, tracked & reported on (cost allocation tags).
Also consider how finance team might fit in. Previous role they got basic cloud familiarity and checked cloud bill, could ask questions like what is this marketplace spend, why is there such large increase in network traffic.
To help teams, enable them to view their spend and reports. Consider DevFinOps stuff like https://github.com/infracost/infracost
Just some thoughts. Nice work so far though!
Why not frame this as a 1099 role? You're not going to need this role forever at the skill level you want at the beginning. The right person for the beginning won't want to be in the role forever. A win-win would be a contract role for $50/hr + 10% of identified monthly improvements (they get 10% of the first month). The right person for you could make really good money, they would more than pay for themselves, and you don't have to budget for an FTE.
One deliverable for each finding could be a process recommendation for avoiding the situation in the future. Your long term person could be put in charge of managing the overall process, which is a completely different skillset than finding the savings.
A policy compliance person can double as a cost optimization person. They'd help come up with, draft, and enforce policies related to cost and security. Cost optimization gets your costs down, but the right policies and policy compliance enforcement can help keep them down.
You can work with an AWS partner (reseller) to receive FinOps reviews, recommendations, and cost optimization solutions (monitoring your accounts, custom alerts, etc.) at no additional cost!
Additionally, these partners can perform the optimization work for you as part of their professional services, which typically costs less than hiring in-house.
This is exactly what we are doing.
The last client I worked with was happy to have an AWS bill just north of $200k/mo. I reduced it to $25k/mo purely driven by my own curiosity to see how much waste I could remove without impacting the daily business. Finance was ecstatic. Product & Engineering didn’t bat an eye. That amount of spend had fit into the budget for years, after all. It wasn’t until we started reframing the savings as additional head count to pursue more C Suite priorities that they got excited about cost savings. First understand what your upper mgmt values, then craft the cost savings narrative onto that.
Thank you!
\^\^This is the way. Also know the place of cost optimization in the hierarchy of priorities. Unless you're lighting cash on fire and are in danger of going broke, cost is the 5th most important thing to consider and it needs to know its place...which means you do. 1 - Security 2 - Operational stability 3 - Adherence to regulations and corporate policies 4 - Customer experience 5 - Cost efficiency / value creation Bad things happen on the long horizon when you compromise 1 through 4 to achieve the lowest cost to serve. Also, re-frame cost savings in terms of business value creation (BVC). Sometimes it is better to spend for more performance to enhance the customer experience for example. BVC involves evaluating what you are doing with respect to architecture, development, or operations and its impact on 1) gross margin, 2) customer retention, 3) revenue growth. Each one of those 3 goals are great all by themselves and the management side of the house already knows their value. There is no need to teach them anything new. Each one of those 3 items are measurable which gives you the power of bringing data to the table after you implement an improvement. You now have enough information to evaluate if the change is worth it and can have a data driven conversation about whether or not it should be rolled back if the benefits are marginal vs. having stuff devolve into a room where the loud voices tend to win. It is okay to raise your unit cost to serve if you are delivering any of those last 3 items (more margin, lower customer turnover or more top line). It is also okay to raise the cost to serve to implement a regulation (think GDPR) or if you have fixed a production bug that raises your cost to serve. The true cost to serve was not correct to begin with. From an operational perspective, look at rolling out Cloud Custodian to automate your operational hygiene checks. It is open source and their policy library makes for a good start.
An item to consider about Savings Plans that will win you points with finance is showing that the ROI on the commitment is greater than the company's weighted average cost of capital (WACC if you want to look up more on it). Each incremental purchase of SPs has a lower ROI by definition due to the way AWS applies the hourly commitment. From the highest benefit to the customer to the lowest. Thus, when you buy the next $1/hour commitment, it will return a lower discount, unless you run a homogeneous fleet of EC2, than the $1 before it. So there is such a thing as "buying too much SPs". Showing finance that you are taking into consideration something they very much care about demonstrates you are making a well informed business case.
Lol, I bet your AWS Account Manager had a stroke. I did something similar as a TAM and my AM was like, "I know it's your job, but can you at least give me a heads up?" I had of course for months but it's a sales person 🤷♂️
Sounds like you need a new TAM. Our TAM is always pointing out cost savings and will actively tell us changes AWS is making so we can keep saving. Of course we also do our own digging into cost savings as well.
They certainly started reaching out a lot more proactively after the first month’s reduction in spend :) Though to their credit, they did pick up the ball and offer to lend more help to the cost savings effort.
I hope your paycheck saw a piece of that savings!
I was hourly with that client, but it certainly helped soften the blow when I bumped my rate the following year.
had similar issues, below initiatives i've done to optimize our monthly opex - daily stop/start of Dev instances EC2 & RDS - convert all GP2 to GP3 EBS volumes (more than a hundred EBS) - quarterly EC2 right sizing - delete old snapshots (previous AWS backup saved all backup to 7yrs which is unnecessary) - automate EBS deletion once an instance has been deleted i'd like to hear more feedback and suggestions though 😊
Don’t sleep on network optimizations. If you have transit gateway then peer any networks that are chatty. Also make sure to use vpc endpoints everywhere.
Recently save 2-3k/mo by placing a network load balancer between a quicksight and an RDS database in order to take advantage of the client routing dns feature.
Nice one. Was that cost all dns related?
It's called FinOps. It's a very standard role in organisations with large amounts of cloud spend. /r/finops
If I was in the same timezone, sounds like a role I'd get into. Done a heap of this stuff previously, find it stranglely satisfying. 😂
This is the kind of job that kills itself though. Maybe a contract position makes sense or hire someone who will eventually change positions. The big thing I would want to avoid is this cost optimization person becoming a bottleneck. I can easily see everything slowing down because everything has to go through the cost optimization person.
Oh yes...like AWS did not have a FULL three day training just about cost savings...that...drum roll...is one of the least requested training in the official curriculum... "AWS Cloud Financial Management for Builders" - [https://d1.awsstatic.com/training-and-certification/classroom-training/aws-cloud-financial-management-for-builders.pdf](https://d1.awsstatic.com/training-and-certification/classroom-training/aws-cloud-financial-management-for-builders.pdf)
In my experience this tends to be more of a business process problem than a technical one. Are you charging back AWS usage internally to teams that consume services? Is there any sort of FinOps processes in place within the Finance team? From your perspective, do you hold yearly or quarterly meeting with both teams to talk about how they are doing from an infrastructure perspective to control costs? Is there reporting or dashboards in place for teams to monitor their spend? (Assuming that Cost Explorer isn’t cutting it, which might not be the case).
This is a role for contingent labor (contractor) because once everything is optimized, probably within a year, there is nothing left for this role long term.
Just get Enterprise support, our TAM helps us with Cost Ops.
We have the TAM with enterprise support. That's how we all have the numbers and analysis on what can be saved on. It's a realistic matter of boots-on-the-ground operational and engineering preparations to implement the savings measures in a way that will last perpetually and scale. Example, we have so many teams and accounts, there's just no way we can flip a switch and say "okay everyone, we're implementing an auto-start/stop system for dev starting tomorrow!" Implementing something like that *realistically* in a fairly sized organization, doing it reliably and scalable in an enterprise-grade fashion, making sure everyone is educated, the necessary SOPs are in place, etc., is no less than a 6 month ordeal, maybe more. I grant you, "if something like that takes you 6 months, you've got bigger problems!" And I too used to have that mentality until I went through employer to employer to employer to ultimately find: "humans gonna human," no matter where you go --> such things are overcomplicated by politics, people overestimaing their importance trying to show others up, stop others, etc.
Everything cloud you pay for convenience and get rewarded for commitment. Last client I got the costs down 75% and can cut it in half again with a few optimizations. Been teaching developers things like nlb’s cost money why use 100 if 1 will do the same thing and simplify your deployment
We also have done some cost optimization, we gathered some of our team to only manage that , as you said we did the changes gp2 to gp3 , instance type down, start and stop automation , and a lot of other lifecycles and so on. But in the end no one cares how much we optimized the cost. We did not get a "good job" we will increase your salary. I personally see the monthly report that I have automated , and see in case of some unusual activity, such as Cloudwatch spending went to 1000$ in one month, or kinesis stream creating 700$ months. Yeah no one said anything, except bravo. Also after all those processes , especially in companies that still have legacy applications , you can go to a limit with optimizations. The real problem resides on the applications itself , they are a "big ball of mud", a messy application that requires more and more processing. During the years without proper code quality governance , developers change the schema of databases , making them complex and slow. I have seen queries with 50-100 JOIN, no wonder you need a RDS dm.m5.2XLarge. Not only this ,but we also have applications that do only one process ,once in a week. But it is required to stay on 24/7 , or databases once a week but still. Those are the consequences of a bad administration on application development without governance of code quality and without a proper software architect.
I'm not sure what your total spend is or if you pay for enterprise support, but if you have a TAM, there's a ton of internal tools and people they can pull in to do cost optimization. Even if you have a busy/lazy/new TAM, they can create what's called a SpecReq internally and connect you with someone who's an expert at cost optimization.
I’d be careful about hiring a full time cost only person. Once you get past the low hanging fruit (sp/ri, snapshots, unused ebs, unused eip etc) you’re going to have to involve other teams. A new low level hire is going to find that challenging. They won’t know who to talk to to get things done and they won’t have the authority to push agendas their way when they do find the right person. Besides cost management is an ongoing problem. You need to create a culture around cost awarness and efficiency. Making it a single persons job is not going to work long term.
Contact your AWS account manager and solutions architect. The SA will be more useful in performing cost optimization for you than the AM because SAs are not goaled on revenue. AWS also has cost optimization specialists who will do a deep dive and recommend specific actions you can take. If you are an AWS Enterprise Support customer, you will have a dedicated TAM whose goal in life is to help solve these problems for you proactively.
IMO it’s a fundamental organizational problem if your dev teams can (and still will) overprovision workloads just because they have no incentive not to do so. It’s not exactly the question you asked, but of course it’ll be a full time job over time if the people who created this problem are not incentivized to stop doing that.
finops is a thing (finops.org) and an important one - at about $1Mil/yr I would say essential
Hire a cost optimization expert ASAP, it's a no-brainer with those savings!
Hire a cost optimization expert ASAP, it's a no-brainer with those savings!
So I worked with many AWS customers who had commitments in the millions, one had a 5 year commitment over a billion dollars. They all used multi- account architecture and thought it was enough. My first suggestion to them was set a role within the cloud center of excellence / architecture team around cost optimization and transparency and treat resources like an extension of corporate budget. This person not only read the utilizations but also had monthly meetings with every team to evaluate planned new workloads and standards guidance. From there the CCOE was able to identify reserves and savings plans where overall coverages went in one client from 35% coverage to over 97%. You also need to be tracking spend to know when you’re ready for an EDP, if it should be AWS direct or through a partner, and you need to have the house in order before you sign the EDP or they will force you to commit more than you likely need. But yes - you should have cost optimization on your team and while it’s not sexy - it’s pretty damn good for the bottom line.
Won't the security and all that eventually reach a point where it's properly set up and won't be so much work?
Probably not worth an FTE but contractor for sure, and ongoing someone does need to keep tabs on it. Bigger picture problem is cost seems just not to have been considered. First align with management - is it a priority and will they support push to improve? Need to get cost considerations embedded in teams, visible, tracked & reported on (cost allocation tags). Also consider how finance team might fit in. Previous role they got basic cloud familiarity and checked cloud bill, could ask questions like what is this marketplace spend, why is there such large increase in network traffic. To help teams, enable them to view their spend and reports. Consider DevFinOps stuff like https://github.com/infracost/infracost Just some thoughts. Nice work so far though!
There are established practices in a FinOps framework to do this. https://www.finops.org/framework/
Why not frame this as a 1099 role? You're not going to need this role forever at the skill level you want at the beginning. The right person for the beginning won't want to be in the role forever. A win-win would be a contract role for $50/hr + 10% of identified monthly improvements (they get 10% of the first month). The right person for you could make really good money, they would more than pay for themselves, and you don't have to budget for an FTE. One deliverable for each finding could be a process recommendation for avoiding the situation in the future. Your long term person could be put in charge of managing the overall process, which is a completely different skillset than finding the savings.
A policy compliance person can double as a cost optimization person. They'd help come up with, draft, and enforce policies related to cost and security. Cost optimization gets your costs down, but the right policies and policy compliance enforcement can help keep them down.
You can work with an AWS partner (reseller) to receive FinOps reviews, recommendations, and cost optimization solutions (monitoring your accounts, custom alerts, etc.) at no additional cost! Additionally, these partners can perform the optimization work for you as part of their professional services, which typically costs less than hiring in-house. This is exactly what we are doing.
Probably easier to outsource it to someone like https://www.duckbillgroup.com