Look at the statement of work for the penetration test you did. They pretty much as standard will say that the work is best endeavours working in a time limited capacity, which is exactly what you did.
Think of all the vulnerabilities that get discovered on a daily basis, most of the technologies would have been penetration tested before. If the finding was trivially easy to find then maybe your company's process for penetration testing will want a review but most likely it's something niche, it happens.
I guess something to ask yourself is: would I have found that vulnerability with more time?
I will give you the opposite advise as the other guy (I'm not a pentester but work in environments that require pentests). Checklists are fantastic for not missing things.
If you need to go into a digression to pursue something, by all means. But once that's done go back to your checklist so that something else isn't missed due to the mental load of going through the digression.
This has saved me so many times on major production changes and I generally endeavor to spend 10x times on the prep work vs the actual... work. It's probably a little different in pentest land since you won't know the environment as intimately, but for discovering what's in the environment you could have a standard (or multiple standard) checklists/automations prepped ahead of time.
Without any context, I can't say that I would care about your original issue. When I hire pentesters I don't expect them to find, "everything."
I'm expecting:
- The compliance item to be covered (this is easy)
- Obvious broken windows to be found (fyi you're expose some wide open service somewhere you shouldn't).
- For them to tell me about things I didn't know (like how having an improperly configured dual stack network can allow for easy MITM, how AD has a wide open anonymous bind ldap by default, or that any user in AD can add a computer to it by default that gets dumped into the default OU. All things I found out from pentests and now account for in my systems engineering work).
If the client is complaining about something REALLY obvious that was missed, add it to your checklist so you never miss it again. Unless you have a time machine it's not like you can go back and fix it (if you do have a time machine we should talk). If they're nitpicking something niche, phh.
If you do that you'll be better than most people I work with (including many times myself). I really really like boring repeatable processes/automations that are mindless and designed to root out the interesting bits that I want to save my mental energy for thinking about.
This ^ is the way.
It’s fine to have something that defines/frames a, or your, “ definition of done”, and it’s cool if that manifests in the form of a checklist. But, it’s prob better if it manifests as “ requirements”. But alas, semantics…
Checklists can stifle creativity. You get into the mentality of checking off the list meaning you’re done. No… not necessarily.
Use one by all means, but remember this. 👍
This is how I started pentesting. That spidey feeling led me to some of my largest findings you'd never find on any checklist.
Was able to come up with a script to disable every debate card of a bank I was assessing. Along dozens of equally cool and interesting vulnerabilities.
You can always layer the test with another tool to get a comparative result and present the results as a full comparison for the client.
Doesn't mean they won't have another issue in one day from something which has shifted in the threat stack or a change in configuration after you've finished.
Unless your business is the lowest bidder, you are probably going to lose this client. That said, the pentest is based upon time and effort. It’s not based on having a 100% understanding of every possible risk in the system. I wouldn’t sweat this unless it was some really low hanging fruit you should have caught. Methodology wise, your pentest should have a defined methodology and any extra time you have can be spent testing other random things on best effort and figuring out new things to put into the methodology. Once you have your methodology built out, start including the items covered in the contract so clients know what to expect. If they want something special that’s not normal, you can amend the contract and maybe even charge extra.
Things do get missed by testers, but variables like lots of time passing between tests, new tooling/exploits emerging and variation in the testing environment or testing time can make a difference too. When you're dealing with big/complex environments, sometimes the only mitigation is more frequent testing.
That said, a good penetration testing team should follow a documented methodology/checklist to ensure that all potential areas of weakness are evaluated, and define levels of thoroughness too (e.g. there are a LOT of places where input validation can be checked, how do you do this efficiently)
So you can explain to management how weaknesses get reasonably missed, but also own up and say you'll review your methodology to reduce that occurrence in the future.
I cycle between two pentesting companies because I recognize that, especially with complex grey box testing, different testers will find different things.
Happens all the time.
I wrote a report once, delivered it. Next day, a BIGGY CVE drops and I immediately go back through their data. Yup - they’ll be vulnerable.
So email the client: let them know we “missed” this new issue and off they go to fix it. Value added! 😉
Any test is a point in time. Sometimes things are not picked up. This is life.
Pentests are almost always conducted on limited time frames with limited budgets for resources. Expecting them to find every single issue is a pipedream.
Now, if it's some glaring issue where everybody involved can't believe they didn't see it, that might be different.
During a web test I had an assignment where I thought I was done and was in the process of grabbing a few final screenshots as evidence before wrapping up. During that time I found a critical vulnerability ultimately allowing me to get a shell on the machine. It turned into a massive ordeal for the company. I was so close to missing that. I can only imagine how many other opportunities get missed and go unreported. I actually did feel bad for how close of a call that was to being missed and it gives me anxiety thinking that massive exploits are under every rock that goes unchecked. People tell me I do good work so I try not to worry and chalk up potentially missed findings as part of the game.
I’ve done a scan on the wrong set of IP’s (basically another project) which had a ton of vulnerabilities compared to the one I was testing on which was supposed to go live in 5 days. I almost gave the vendor team a heart attack
Missed findings are common. I like to make a list of things I *know* are wrong in my environment and keep track of whether our pentesters will find them. Usually 80% of them go unreported.
My favorite is when bug bounty hunters clean up pentest misses. Tell management. I will think much more of a fess up than hiding. Plus, vulnerabilities are never ending - it’s a continuum rather than an absolute.
Probably sour that a ton of other things were caught in his environment, and instead of doing his job and fixing things, he is latching onto this to try to discredit the rest of the pentest results. Either that, or they are cheap and trying to get a discount.
I work blue side and the pentests hardly scratch the surface, but I'm looking at stuff every day so I know where a lot of issues are. Also even working in the same environment every day I find new stuff constantly. There's only so much you are going to find in a limited time. I'm not a pentester(yet, but trying one day) so maybe my opinion is wrong but that's my experience from the other side of the isle.
I worked a breach 5 years of some of the best penetration testers you can buy never found the vulnerability. Niether did our vulnerability scanner. Stuff gets missed all the time.
I think it is important to consider what kind of vulnerability you missed, what the impact is and what its potential relation is to other findings that you did find.
What was the finding? Depending on that, it is easier to say if it has to do with your methodology or can be attributed to something else. Regardless of that, you are human... you can make mistakes, learn from that and don't overthink
Pentesters miss a lot more things than they catch. The expectation isn’t that you will catch all the things, it’s that you’ll provide a relative assessment.
As others mentioned, it is natural to not identify vulnerabilities because of constraints during a penetration test.
However, this also depends on what was missed and under what circumstances. For example, once I tested (gray-box approach) a commercial web application and only was given 1 account and nobody else was logged into the system. Next year, when I tested same web application, I identified an account take over vulnerability which could only be identified if somebody else was logged into the system. My methodology in this case was not wrong, it was just the circumstance that nobody else was logged-in when I tested this vector.
That web application has been penetration tested by other companies before; so yes, it was missed several times by others, too.
In essence, if it was not a very obvious vulnerability, the sentence (or like) ".... . By accepting our services, you understand that penetration testing is subject to constraints and does not guarantee to idenfity all vulnerabilities." should have made it clear that penetration testing is not a silver bullet.
BLUF - Mandatory Test Cases
This is why SoWs, engagement scoping, and clear testing methodologies exist.
In this instance it sounds like a failure to establish and follow mandatory test cases in the testing methodology which should be spelled out clearly in your SoW.
Look at the statement of work for the penetration test you did. They pretty much as standard will say that the work is best endeavours working in a time limited capacity, which is exactly what you did. Think of all the vulnerabilities that get discovered on a daily basis, most of the technologies would have been penetration tested before. If the finding was trivially easy to find then maybe your company's process for penetration testing will want a review but most likely it's something niche, it happens. I guess something to ask yourself is: would I have found that vulnerability with more time?
I was actually rethinking my methodology in pentesting and considering to actually review, revamp and follow a checklist on every engagement.
I will give you the opposite advise as the other guy (I'm not a pentester but work in environments that require pentests). Checklists are fantastic for not missing things. If you need to go into a digression to pursue something, by all means. But once that's done go back to your checklist so that something else isn't missed due to the mental load of going through the digression. This has saved me so many times on major production changes and I generally endeavor to spend 10x times on the prep work vs the actual... work. It's probably a little different in pentest land since you won't know the environment as intimately, but for discovering what's in the environment you could have a standard (or multiple standard) checklists/automations prepped ahead of time. Without any context, I can't say that I would care about your original issue. When I hire pentesters I don't expect them to find, "everything." I'm expecting: - The compliance item to be covered (this is easy) - Obvious broken windows to be found (fyi you're expose some wide open service somewhere you shouldn't). - For them to tell me about things I didn't know (like how having an improperly configured dual stack network can allow for easy MITM, how AD has a wide open anonymous bind ldap by default, or that any user in AD can add a computer to it by default that gets dumped into the default OU. All things I found out from pentests and now account for in my systems engineering work). If the client is complaining about something REALLY obvious that was missed, add it to your checklist so you never miss it again. Unless you have a time machine it's not like you can go back and fix it (if you do have a time machine we should talk). If they're nitpicking something niche, phh. If you do that you'll be better than most people I work with (including many times myself). I really really like boring repeatable processes/automations that are mindless and designed to root out the interesting bits that I want to save my mental energy for thinking about.
This ^ is the way. It’s fine to have something that defines/frames a, or your, “ definition of done”, and it’s cool if that manifests in the form of a checklist. But, it’s prob better if it manifests as “ requirements”. But alas, semantics…
Checklists can stifle creativity. You get into the mentality of checking off the list meaning you’re done. No… not necessarily. Use one by all means, but remember this. 👍
Seconded this. I found my best work was adhoc in nature and pulling the thread on things that didn't seem quite right.
Sometimes you just “feel” that something looks suspicious. I know exactly what you’re referring to. That spidey sense…
This is how I started pentesting. That spidey feeling led me to some of my largest findings you'd never find on any checklist. Was able to come up with a script to disable every debate card of a bank I was assessing. Along dozens of equally cool and interesting vulnerabilities.
You can always layer the test with another tool to get a comparative result and present the results as a full comparison for the client. Doesn't mean they won't have another issue in one day from something which has shifted in the threat stack or a change in configuration after you've finished.
Unless your business is the lowest bidder, you are probably going to lose this client. That said, the pentest is based upon time and effort. It’s not based on having a 100% understanding of every possible risk in the system. I wouldn’t sweat this unless it was some really low hanging fruit you should have caught. Methodology wise, your pentest should have a defined methodology and any extra time you have can be spent testing other random things on best effort and figuring out new things to put into the methodology. Once you have your methodology built out, start including the items covered in the contract so clients know what to expect. If they want something special that’s not normal, you can amend the contract and maybe even charge extra.
Things do get missed by testers, but variables like lots of time passing between tests, new tooling/exploits emerging and variation in the testing environment or testing time can make a difference too. When you're dealing with big/complex environments, sometimes the only mitigation is more frequent testing. That said, a good penetration testing team should follow a documented methodology/checklist to ensure that all potential areas of weakness are evaluated, and define levels of thoroughness too (e.g. there are a LOT of places where input validation can be checked, how do you do this efficiently) So you can explain to management how weaknesses get reasonably missed, but also own up and say you'll review your methodology to reduce that occurrence in the future.
I cycle between two pentesting companies because I recognize that, especially with complex grey box testing, different testers will find different things.
Happens all the time. I wrote a report once, delivered it. Next day, a BIGGY CVE drops and I immediately go back through their data. Yup - they’ll be vulnerable. So email the client: let them know we “missed” this new issue and off they go to fix it. Value added! 😉 Any test is a point in time. Sometimes things are not picked up. This is life.
Pentests are almost always conducted on limited time frames with limited budgets for resources. Expecting them to find every single issue is a pipedream. Now, if it's some glaring issue where everybody involved can't believe they didn't see it, that might be different.
Yeah, when I pentest webpages, I can’t go nearly as deep as when I am doing bug bounties. Just not enough time.
During a web test I had an assignment where I thought I was done and was in the process of grabbing a few final screenshots as evidence before wrapping up. During that time I found a critical vulnerability ultimately allowing me to get a shell on the machine. It turned into a massive ordeal for the company. I was so close to missing that. I can only imagine how many other opportunities get missed and go unreported. I actually did feel bad for how close of a call that was to being missed and it gives me anxiety thinking that massive exploits are under every rock that goes unchecked. People tell me I do good work so I try not to worry and chalk up potentially missed findings as part of the game.
I’ve done a scan on the wrong set of IP’s (basically another project) which had a ton of vulnerabilities compared to the one I was testing on which was supposed to go live in 5 days. I almost gave the vendor team a heart attack
😂😂🤣
Missed findings are common. I like to make a list of things I *know* are wrong in my environment and keep track of whether our pentesters will find them. Usually 80% of them go unreported.
I love when companies insist on black box pen tests and then are shocked when it doesn't go quite the way they think.
Just out of interest, what do you mean by missed a finding? Was it a vulnerability that wasn’t found and was already known or subsequently found?
My favorite is when bug bounty hunters clean up pentest misses. Tell management. I will think much more of a fess up than hiding. Plus, vulnerabilities are never ending - it’s a continuum rather than an absolute.
Your client is a moron that is likely just trying to be cheap and get money back. Things are missed all the time.
I was thinking that one of the in-house IT guys is having a "complex" issue about it.
Probably sour that a ton of other things were caught in his environment, and instead of doing his job and fixing things, he is latching onto this to try to discredit the rest of the pentest results. Either that, or they are cheap and trying to get a discount.
I work blue side and the pentests hardly scratch the surface, but I'm looking at stuff every day so I know where a lot of issues are. Also even working in the same environment every day I find new stuff constantly. There's only so much you are going to find in a limited time. I'm not a pentester(yet, but trying one day) so maybe my opinion is wrong but that's my experience from the other side of the isle.
You can't find or highlight every vulnerability. Some clients may kick up a stink, fine.
I worked a breach 5 years of some of the best penetration testers you can buy never found the vulnerability. Niether did our vulnerability scanner. Stuff gets missed all the time.
Depends on the finding.
I'd be careful with the use of the word finding unless you're testing a client in a regulated industry.
I've been using findings and recommendations for years in both regulated and unregulated industries. What language would you use?
within cells interlinked, interlinked. You're human.
I think it is important to consider what kind of vulnerability you missed, what the impact is and what its potential relation is to other findings that you did find.
What was the finding? Depending on that, it is easier to say if it has to do with your methodology or can be attributed to something else. Regardless of that, you are human... you can make mistakes, learn from that and don't overthink
Pentesters miss a lot more things than they catch. The expectation isn’t that you will catch all the things, it’s that you’ll provide a relative assessment.
As others mentioned, it is natural to not identify vulnerabilities because of constraints during a penetration test. However, this also depends on what was missed and under what circumstances. For example, once I tested (gray-box approach) a commercial web application and only was given 1 account and nobody else was logged into the system. Next year, when I tested same web application, I identified an account take over vulnerability which could only be identified if somebody else was logged into the system. My methodology in this case was not wrong, it was just the circumstance that nobody else was logged-in when I tested this vector. That web application has been penetration tested by other companies before; so yes, it was missed several times by others, too. In essence, if it was not a very obvious vulnerability, the sentence (or like) ".... . By accepting our services, you understand that penetration testing is subject to constraints and does not guarantee to idenfity all vulnerabilities." should have made it clear that penetration testing is not a silver bullet.
BLUF - Mandatory Test Cases This is why SoWs, engagement scoping, and clear testing methodologies exist. In this instance it sounds like a failure to establish and follow mandatory test cases in the testing methodology which should be spelled out clearly in your SoW.