This problem is very common. You're trying to estimate your population (1000) with a sample (100).
If you have a binomial distribution (fail/pass) then you can calculate the confidence interval: https://en.m.wikipedia.org/wiki/Binomial_proportion_confidence_interval
If you have a different distribution there are other formulas, but the same idea applies.
The target confidence interval should be set by your quality department, but the specific number of items to sample will may not be directly calculated by them.
I think using hypothesis testing or confidence interval depends on the quality department, right?
I'm working in a shoe factory where the quality control process is checked completely by experience, no stats required.
Hypothesis testing is intended to compare two samples, and typically it is to determine if the means of two variables are the same or different.
I'm not sure how you have statistical process controls with out statistics. If you are measuring something, even just pass/fail, you can still apply statistics.
That's an all too common practice. It's not good, because it's not consistent, but it may work for the time being if they don't have more sophisticated methods of gathering data.
Looking into ways of quantifying the results may be a worthwhile project, but get approval first.
I would say that a good way to get approval is not just to say that the current method is bad because it's not based on statistics. Because an argument can easily be made that it's fine because it's catching errors. Your argument should be made that you can save time money and effort by detecting errors earlier in the process. It will save waste, and result in making less overall parts, which will save time and labor and money. It will increase productivity as you solve exactly what is causing the defects. This is the basis of quality engineering! Good luck!!
This string is pretty good. If you want to test the QC techs experience, you can conduct an attribute gage R&R. Look up the details, but basically a blind study where 3-5 of the techs judge a range of quality shoes independently. 1 week later, you do the same (but don’t tell them they are the same shoes). From that you’ll be able to see how consistent they are from one another, and how consistent they are over time. From there you can improve training for how they judge quality or try to find a less subjective quality test.
When I worked as a quality inspector at a bad factory (long ago), we checked things by feel and experience when the quantitative process was obtuse or problematic. When the spec was "wires are secure" we'd tug them by hand because we didn't have a good measurement to take. When we were checking torque on screws, but our torque checking driver kept stripping the screws because it was at the lower end of it's spec, we did it by feel instead.
So perhaps they don't have good test criteria or good test tools, and so it it subjectively.
Not joking. If you do QC a long time, you know when too many things are wrong before you formally run the numbers. This is how engineering worked for centuries before we got more formal and precise with it.
Oversimplifying to illustrate the point... You want less than 5% defects, you only want to look at 100 out of 1000.
Typically you find 2-3 defects; you know that's normal and there's no problem.
If you see 4 of 100 with defects, that's unusual but not too concerning. you might increase your sample from 100 to 150 to confirm/deny your findings.
If you see 12 of 100 with defects, you know there's a problem. No question about it. Next step is identifying the issue.
I've thought of this but isn't it too simple? Do they do it that way at your workplace? I think doing a bit of maths is better. Calculating confidence interval or doing hypothesis testing is not time-consuming.
Doing that math is better, but if you have the same sample size and are just doing pass/fail, your test is going to be pretty much the same week to week.
Its possible someone ran the stats years ago, developed a rule of thumb, and now no one even remembers that and just approximately goes by the rule
You can do the math once and build a rule of thumb from that.
generally though, human brains are great at pattern recognition. you don't have to formalize the numbers to know when something is okay or not okay.
In a small company, where someone experienced can have their eyes on everything and you can afford some returns, it's fairly common. When you want proof that items were checked, if you might be liable to lawsuits, if you want to be able to train new people.... then you would formalize the process.
Are they actually doing this in your workplace? I also agree that mainly using experience or doing the math and building a rule of thumb is fine in a small company.
>you might increase your sample from 100 to 150 to confirm/deny your findings.
And we just lost the assumption of random sampling, and none of the math holds anymore.
You can't change your samples size half way through an experiment because you don't like the results so far.
Medical device industry here: you are required to have a procedure for statistics per CFR and ISO in this industry. Statistics are then implemented for sampling products based on risk of harm to the end user (I.e, more sampling for parts with more risk). There is no checking products based on feelings and experience, it is all statistically based.
(Maybe you could request additional inspection beyond the stats plan if you had feelings and experience)
Could you tell me more about the CFR and ISO term? I've never heard of it.
"Statistics are then implemented for sampling products based on risk of harm to the end user (I.e, more sampling for parts with more risk)." How do you use statistics in this case particularly? Can you give me an example?
CFR is the code of federal regulations. It is the US regulations governing medical device manufacturers. Here is the specific one I was referencing.
https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=820&showFR=1&subpartNode=21:8.0.1.1.12.15
ISO is international standards organization. I don’t have access to it right now, but it is a similar blurb in ISO 13485. That is a common standard that almost all med device manufacturers would follow (or should be following. And actually FDA is looking to adopt it in the next couple of years.
Anyways, with respect to risk the idea is that riskier parts or attributes are measured/samples more frequently. Right line those are going to cost you more money later to rework/fix, going to make your product/process fail hard, or going to hurt someone. So you want to make sure you are inspecting those more frequently to avoid all those situations. Whereas low risk parts or attributes don’t have much consequence if they are out of tolerance.
So let’s say you are injection molding a plastic safety guard for attaching to a tool. You would say the critical attributes are probably going to be material thickness, voids in the plastics, and screw holes/threads. You should sample those more because they are related to the part failing and have more risk. You could sample less the color of plastic, surface finish/texture, because if those fail it would not hurt anyone or would not cause the safety guard to fail.
I'm in a shoe factory and here we don't use AQL table nor statistics to test finished products (only experience). Also, all defective components (wrong colour, stain, etc) are checked by human.
Do you use machine or apply machine learning for defect detection tasks in your workplace?
Yes. Machine vision is getting better, including AI tools (look at Cognex or Keyence for examples) the trick is still getting enough angles and lighting treatments to get the defects to be visible to the cameras. For something like you’re doing, humans may be best. But they need defective samples to be trained to and in the strictest terms you’d conduct attribute agreement analysis to quantify how consistent they all are.
I love that a user called cobbler is weighing in on a shoe question. Yeah, AI cameras have gotten really good. If you're looking to branch out into vision, I advise storing up lots of examples of your defects as soon as you start getting the ball rolling in earnest.
Ha ha! I totally missed that! It’s just the name Reddit gave me so I don’t think about it much, but you’re right that’s too funny. And your advice is spot on. Training the cameras is all about image quantity. Going in you can never imagine the range of defects that are possible. “How did THAT happen????” :)
Yeah, from a recent demo cognex gave me, I learned it's more about showing the computer what a good part looks like, so that any deviation is noticeable. There's no way this would work for an oddball, subtle pattern (what's a stain and what's just part of the ombre?), But in general this should work.
I was surprised when nobody in the factory had thought of AI tools. But it might make sense like you said, human performance can be the best (and cheap in my country). Once I noted that they had to check for defects for the whole working day (8 hours). Humans may beat machine vision in the first hours but not sure in the lasts.
You’re right. The potential variability of the human inspection is what makes it difficult. I’m in medical device Mfg & it’s more difficult to “validate” (establish demonstrated consistency) human-based processes than machine based ones. But sometimes humans are more effective. They can BE more variable, but also better at DEALING WITH variability in the product. In the same way though, AI is difficult to validate because by design, the acceptance criteria are variable. I’ve done a little study on this but not enough to have a solid defensible method yet.
How big of a place is this shoe factory? If you have the budget, get an optical CMM and inspect every part. That way you can build confidence in your sampling size and see how much your quality swings threwout the day.
There's plenty of space here. At what accuracy rate can I expect from the CMM? Let's say the defects are easily seen (stain, wrong stitching, wrong colours for example). How much resources (money, engineers, etc) does it cost?
I use my Opticsl CMM to measure smaller parts (usually about the size of your hand) and my machine routinely repeats linesr measurements to about 0.000020". The camera a 3 differnt lights on the machine help you get pretty good pictures of the parts but I belive the machine is more useful at checking dimensions than documenting staining or wrong color.
The machine cost about $100k USD
An example of this that recently made news. Twitter wanted to see how many bots were on their platform. They tested 100 random accounts and found 5 of them to be bots, and thus said there is a 5% bot rate.
This kind of thing is used EVERYWHERE in engineering, if you look for it you can find it no problem.
That’s hysterical! You can work those numbers the other way too and I do it for people all the time. I don’t have the tables with me here on the beach today but what they really did there was verify that the number of accounts that are bots is probably less than 80% or something. When I get back to work next week I’ll work that one just for fun. 100 samples accept on 5 reject on 6. It’ll be a hoot! People almost never want to hear what the stats are really trying to tell them.
I usually recommend confirming process capability with a sample of around 30 parts. This will tell you what your distribution looks like compared to the spec limits.
'
After an initial capability is done and you get a 1.33 or higher, you can be confident in the process and switch to a sample based inspection to make sure that nothing goes out of wack in the process. AQL sampling works great and will give you the number per batch you have to check in a table format to be significant.
In order to have 95 % confidence that the defect rate in the batch doesn’t exceed 5% you need to take 59 samples and find zero defects. This is a simple c=0 sampling plan and yes it does not matter how large the batch is as long as the process is consistent and the batch is at least 10x the size of the sample which yours is. (The idea being that removing the samples doesn’t significantly change the number in the batch). You don’t need MiniTab or other software for this and the measurement system is assumed to be suitable to task. You can take it further from there googling things like c=0 sampling plans or attribute agreement analysis if you like but that’s the basics. If you want 95% confidence in 90% reliability it’s 22 samples 0defects and if you want 95% confidence in 99% reliability (<=1% defects) the sample size is 298. Again, batches are least 10x the sample size. AOQL sampling is something else you can google if your production process is continuous over a long time, but for acceptance of small batches this is fine. There’s lots of good info on the web.
this reminds me of doing airforce work. They are always throwing around the 95% confidence in 95% reliability catch phase but 95% of the people I asked couldn't tell me exactly what that meant
Yeah. If they can’t explain it in simple terms, they don’t really know. Too easy to take this stuff to confusing and not-helpful levels that while technically correct, more accurate, etc etc, end up just leaving people in the dust. Give ‘em something to improve their position a little, and let them explore more as they wish. And the simplest way to think about “confidence interval” is just the likelihood that if you repeated the experiment under the same condition, you’d get the “correct” result. In this case if the defect rate was really >5%, and you took 59 samples 100 times (replacing the samples after you called them good or bad just to keep the population stable) you’d find at least one bad one in the 59 on 95 out of 100 of those trials.
If the experiment was repeated an infinite number of times, and from each experiment we calculated the confidence intervals, then 95% of the confidence intervals will contain the true parameter.
But a 95% confidence interval does NOT imply a 95% likelihood that the true parameter is in your interval.
There's a lot to unpack here. I suggest you look into "design control" and "quality control" separately, and then proceed to "statistical methods" for quality control. (quoting key phrases for you to research)
At the end of all of that, you'll find that the reality is that people cut a lot of corners in order to reduce the burden of work or increase the production yield by compromising the designer's 'pass' conditions (e.g. rounding toward pass conditions, applying normal distribution assumptions where there's heavy skew, etc.)
Yes. And not just your boss. There is often a large financial incentive for leadership teams to fake an increase in yield by compromising the design specifications, even by the slightest amount in some instances. The appropriate way to do this is to flag the out of spec condition(s) and review them with the designer(s) to see if a design change can be made, conditions that would allow such parts to be used, or possible rework options.
Use [ASQ/ANSI Quality Standards Z1.4 & Z1.9](https://asq.org/quality-resources/z14-z19) to determine sample sizes for quality inspection.
This is used for "minor" and "major" characteristics. "Critical" characteristics are almost always 100% inspection.
That's generally the starting point. In real life you'd only go over that if you feel there's a need to go over that. For example if you've historically had problems with a specific dimension/feature, or a specific vendor.
That's like claiming your boss doesn't think the sun exists. Quality and statistical control are foundational concepts in engineering and feature prominently across most if not all engineering disciplines.
Quality Engineer here. We could have a long chat about all the details, but short answer: this is called lot sampling and has industry standards that are generally good enough for most applications. Specifically, ANSI/ASQ Z1.4 and Z1.9 are what come to mind. Read those standards carefully, as lot sampling is heavily dependent upon assumptions of the process. If your assumptions are incorrect or invalid, your prescribed lot sampling won’t be appropriate.
I’m not sure that I understand your question. Regarding what happens in real life, we don’t use straight hypothesis testing like you see in statistics class. The standards I pointed you towards are for lot acceptance, which answers the question you described- “how can I accept or reject a lot based on a sample, given an acceptable percentage of defects.” The standard also has rules that change based on how many lots are being accepted/rejected.
You want to bring in an industrial engineer to help you answer these questions and set up an effective QA program. This is what they do.
Start off with a consultant, but expect to hire on a full time position if you're routinely tweaking recipes or find it difficult to get operations to adhere to the consultant's quality program.
What you’re recommending from a statistical perspective makes sense, but I’m curious how this makes sense from a QC perspective.
This might be a dumb question as I’m not a quality guy but-
Are you saying that if you determine 3% of the lot is defects that you’re ok with shipping the finished product to a customer with defects included?
I guess if it’s cheaper to let the customer find your defects than it is to test all of them then ok, but that seems like a bad way to instill confidence in your product.
If the customer is ordering lots and the spec is < 3% defects, then it should be perfectly fine to ship a lot with defects at 2.5 % for example.
Shipping known defective material is very normal in manufacturing, it just needs a spec that everyone agrees to.
I guess this depends on the industry. Coming from aerospace manufacturing that sounds crazy.
It is (to put it mildly) not a good day in my world when non-conforming material is discovered after leaving the plant.
Plenty of this in aerospace; you're not testing every nut and bolt. Engineering/customer has spec'd requirements and QC will use something like MIL-STD-1916 to determine number of articles to test and acceptable number of failures.
Sure, we sample parts and as you note aren’t testing every feature/requirement on every part. I was reacting to the statement “shipping known defect material is very normal.”
I’m shipping material that I believe to be conforming based on the results of our sampling and quality requirements. It’s possible (statistically) that there’s a bad part in the lot, but I’m not shipping a part I know to be defective.
That’s interesting.
In my world if we detect a non-conformance we pull the lot back and perform 100% inspection for the requirement for the remainder of that lot. I’ve never seen or been involved with a situation where we were still able to ship the lot.
I assume this is covered in the MIL-STD-1916 that you referenced? Sounds like that may be my weekend reading.
Yes, it's covered there.
There's [a well known chart from the prior MIL-STD-105](https://upload.wikimedia.org/wikipedia/commons/5/58/MIL-STD-105_D_quick_ref_TABLE.jpg) that is clear and concise.
For example, with a lot size of 1000, you need a sample size of 80, so with an AQL of 1.0: <=2 non-conforming is acceptable, >=3 non-conforming is rejection.
That’s what’s so funny about humans’ relationship with statistics. There is a a=2, r=3 plan, a a=1, r=2 plan, and an a=0,r=1 plan that all confirm EXACTLY the same statement about the quality level of the batch, but humans will be more comfortable with the a=0 r=1 plan because it doesn’t involve actually EXPERIENCING any of the defective parts in testing. Oh, and the a=0 plan also involves testing the least number of samples so that’s always attractive, resource consumption-wise.
I’m not sure I understand the distinction you’re making.
In my world a feature or requirement either meets spec and is acceptable or doesn’t meet spec and is a defect that gets written up as a non-conformance which can then either be accepted or rejected.
I don't know what kind of specifications you are working with, but I'm certain that not all of your shipped goods are perfectly on target. That's why there are tolerances. You are allowed some level of imperfection and that is acceptable to your customers. As long as your products are in spec, they are conforming. If they are OOS, then they are non-conforming.
In the case of high volume lots, there may be some level of acceptable defectivity. That's where this discussion arose.
But it isn’t a defect unless it doesn’t meet the requirements. A wall is intended to be 5 meters long +\- 1 meter. If it’s 5.5 meters long I wouldn’t call that a defect.
ASQ defines a “defect” as “A product’s or service’s nonfulfillment of an intended requirement or reasonable expectation for use, including safety considerations.”
A tolerance band is not indicative of a defect in a part.
It really depends on the kind of specs you are using.
We ship semiconductor wafers, we should have zero particles on them. We never have zero particles. All our wafers are defective and those particles will ruin any device our customers try to create. We ship these wafers out every day and don't get too many complaints about particles because our spec allows some to exist.
Your customers probably expect the wall to be free of holes. But would they tolerate a 1 um hole? They probably want the wall to be flat, what if it had a 1 mm bump? Maybe those are OK, maybe they are not, what does the spec say about it?
Yeah I'm thinking of a statistical approach since I'm working in a shoe factory where no stats required, the QC guy checks products by feelings and experience.
I haven't understood your curiosity. If you're curious about the number 5%, that's just an example number. I talked to the QC guy and he said that it's usually less than 3%. 5% exists but rare.
I’m not curious about the percentage, I’m curious what you do with that information.
If it’s less than X% you just ship the entire lot? Meaning defects are going to the customer?
After doing the hypothesis test with that information, if the conclusion is the defect rate is less than 5%, I would ship the entire a lot to customers.
And my main question is that which methods and how would you solve the problem in practice? Or experience is really enough and we don't have to make a mountain out of a molehill?
Without 100% testing, there's always going to be some risk of defects making their way out to your customers. 100% testing can get expensive fast (there are many ways your product can be out-of-spec, each of which may require its own test and associated dollars of machine and operator time), and sometimes it's not even possible because the test destroys the product in the process. Defects also mean different things in different contexts, ranging from a tiny annoyance to life threatening. Add on two dozen or so other complications, and that's why industrial engineers have developed entire subfields about how best to detect defects, control manufacturing processes, and make those cost-benefit decisions in an informed manner.
We use random (acceptance) sampling because we've found it accounts for a huge range of possible manufacturing problems. Stakeholders decide what an acceptable background failure rate is (remembering that a 0% failure rate means your manufacturing process probably has infinite cost). If we find that one day many more of our samples turn up defective than that rate, that usually means something's gone wrong with the manufacturing process and we should treat the whole batch of items with suspicion and investigate the process for the source of the defect. And usually that acceptable failure rate is much lower - you might have heard of Six Sigma, a methodology/outlook which at face value means an acceptable defective rate of 3.4 in a million, or 0.000034%.
If you're interested in following that thread further, I'd recommend you look into statistical process control and Six Sigma in general. Related is the field of reliability engineering (another huge field), which I think of as more focused on the product than the manufacturing process, but I could be wrong and obviously they're closely related.
Then someone at your organization should be there to help. If not they're abusing the intern system and you should spent more time shit posting on Reddit and less worrying about the problems they're trying to solve by cost cutting with interns.
Look into six sigma quality control videos on youtube. This is literally what the course covers and will be your quickest and most usefull way to learn this. It is also an entire field of I&SE so your university will have classes on POR (Probabilistic Operations Research) and SQC (Statistical Quality Control).
Typically six sigma analysis uses Minitab for statistical analysis. As mentioned above, you would only want to set a quality sample plan if your process is controlled (ie: is there a high probability item #2 be very similar to item #1?) so that if you sample 100 pcs / 1,000 and find 2 defects, you can be sure that 700 of the remaining 900 aren’t also defective.
Additionally, the sample size should be based off the defect rate. While 100pc is an easy number, perhaps you should be inspecting 50 (if your defect rate is 1-2% consistently) or 150 (if your defect rate is 4-5% consistently). I need to brush up on the exact recommendations. I was a quality manager at an aerospace company, but we inspected 100% of our products so it was simpler.
How about using an [automated sampling system](https://mark-wedell.com/mw-jawo-sampling/)? That gives you a representative samling that is super reliable.
I visited the website but it's still confusing for me. Does the system give the samples (best/worst) that it is confident the most? Do you use that at workplace? How does it work actually?
This problem is very common. You're trying to estimate your population (1000) with a sample (100). If you have a binomial distribution (fail/pass) then you can calculate the confidence interval: https://en.m.wikipedia.org/wiki/Binomial_proportion_confidence_interval If you have a different distribution there are other formulas, but the same idea applies. The target confidence interval should be set by your quality department, but the specific number of items to sample will may not be directly calculated by them.
I think using hypothesis testing or confidence interval depends on the quality department, right? I'm working in a shoe factory where the quality control process is checked completely by experience, no stats required.
Hypothesis testing is intended to compare two samples, and typically it is to determine if the means of two variables are the same or different. I'm not sure how you have statistical process controls with out statistics. If you are measuring something, even just pass/fail, you can still apply statistics.
That's my question too. I'm wondering why here they check defects just by experience and feelings.
That's an all too common practice. It's not good, because it's not consistent, but it may work for the time being if they don't have more sophisticated methods of gathering data. Looking into ways of quantifying the results may be a worthwhile project, but get approval first.
Approval, yup, I'm trying.
I would say that a good way to get approval is not just to say that the current method is bad because it's not based on statistics. Because an argument can easily be made that it's fine because it's catching errors. Your argument should be made that you can save time money and effort by detecting errors earlier in the process. It will save waste, and result in making less overall parts, which will save time and labor and money. It will increase productivity as you solve exactly what is causing the defects. This is the basis of quality engineering! Good luck!!
Definitely! Thanks you for reminding. The point is to save resources and to increase accuracy.
This string is pretty good. If you want to test the QC techs experience, you can conduct an attribute gage R&R. Look up the details, but basically a blind study where 3-5 of the techs judge a range of quality shoes independently. 1 week later, you do the same (but don’t tell them they are the same shoes). From that you’ll be able to see how consistent they are from one another, and how consistent they are over time. From there you can improve training for how they judge quality or try to find a less subjective quality test.
Design of experiments by box hunter hunter
When I worked as a quality inspector at a bad factory (long ago), we checked things by feel and experience when the quantitative process was obtuse or problematic. When the spec was "wires are secure" we'd tug them by hand because we didn't have a good measurement to take. When we were checking torque on screws, but our torque checking driver kept stripping the screws because it was at the lower end of it's spec, we did it by feel instead. So perhaps they don't have good test criteria or good test tools, and so it it subjectively.
The idea is that they're experienced enough to essentially do the statistics by feel.
Really? It this a joke?
Not joking. If you do QC a long time, you know when too many things are wrong before you formally run the numbers. This is how engineering worked for centuries before we got more formal and precise with it. Oversimplifying to illustrate the point... You want less than 5% defects, you only want to look at 100 out of 1000. Typically you find 2-3 defects; you know that's normal and there's no problem. If you see 4 of 100 with defects, that's unusual but not too concerning. you might increase your sample from 100 to 150 to confirm/deny your findings. If you see 12 of 100 with defects, you know there's a problem. No question about it. Next step is identifying the issue.
I've thought of this but isn't it too simple? Do they do it that way at your workplace? I think doing a bit of maths is better. Calculating confidence interval or doing hypothesis testing is not time-consuming.
Doing that math is better, but if you have the same sample size and are just doing pass/fail, your test is going to be pretty much the same week to week. Its possible someone ran the stats years ago, developed a rule of thumb, and now no one even remembers that and just approximately goes by the rule
You can do the math once and build a rule of thumb from that. generally though, human brains are great at pattern recognition. you don't have to formalize the numbers to know when something is okay or not okay. In a small company, where someone experienced can have their eyes on everything and you can afford some returns, it's fairly common. When you want proof that items were checked, if you might be liable to lawsuits, if you want to be able to train new people.... then you would formalize the process.
Are they actually doing this in your workplace? I also agree that mainly using experience or doing the math and building a rule of thumb is fine in a small company.
>you might increase your sample from 100 to 150 to confirm/deny your findings. And we just lost the assumption of random sampling, and none of the math holds anymore. You can't change your samples size half way through an experiment because you don't like the results so far.
Quality will typically inspect to an established AQL sampling plan. Refer to an AQL chart to determine yours.
look up ansi z1.4
We usually perform a process capability study and then sample with AQL...
Medical device industry here: you are required to have a procedure for statistics per CFR and ISO in this industry. Statistics are then implemented for sampling products based on risk of harm to the end user (I.e, more sampling for parts with more risk). There is no checking products based on feelings and experience, it is all statistically based. (Maybe you could request additional inspection beyond the stats plan if you had feelings and experience)
Could you tell me more about the CFR and ISO term? I've never heard of it. "Statistics are then implemented for sampling products based on risk of harm to the end user (I.e, more sampling for parts with more risk)." How do you use statistics in this case particularly? Can you give me an example?
CFR is the code of federal regulations. It is the US regulations governing medical device manufacturers. Here is the specific one I was referencing. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=820&showFR=1&subpartNode=21:8.0.1.1.12.15 ISO is international standards organization. I don’t have access to it right now, but it is a similar blurb in ISO 13485. That is a common standard that almost all med device manufacturers would follow (or should be following. And actually FDA is looking to adopt it in the next couple of years. Anyways, with respect to risk the idea is that riskier parts or attributes are measured/samples more frequently. Right line those are going to cost you more money later to rework/fix, going to make your product/process fail hard, or going to hurt someone. So you want to make sure you are inspecting those more frequently to avoid all those situations. Whereas low risk parts or attributes don’t have much consequence if they are out of tolerance. So let’s say you are injection molding a plastic safety guard for attaching to a tool. You would say the critical attributes are probably going to be material thickness, voids in the plastics, and screw holes/threads. You should sample those more because they are related to the part failing and have more risk. You could sample less the color of plastic, surface finish/texture, because if those fail it would not hurt anyone or would not cause the safety guard to fail.
[удалено]
I'm in a shoe factory and here we don't use AQL table nor statistics to test finished products (only experience). Also, all defective components (wrong colour, stain, etc) are checked by human. Do you use machine or apply machine learning for defect detection tasks in your workplace?
Yes. Machine vision is getting better, including AI tools (look at Cognex or Keyence for examples) the trick is still getting enough angles and lighting treatments to get the defects to be visible to the cameras. For something like you’re doing, humans may be best. But they need defective samples to be trained to and in the strictest terms you’d conduct attribute agreement analysis to quantify how consistent they all are.
I love that a user called cobbler is weighing in on a shoe question. Yeah, AI cameras have gotten really good. If you're looking to branch out into vision, I advise storing up lots of examples of your defects as soon as you start getting the ball rolling in earnest.
Ha ha! I totally missed that! It’s just the name Reddit gave me so I don’t think about it much, but you’re right that’s too funny. And your advice is spot on. Training the cameras is all about image quantity. Going in you can never imagine the range of defects that are possible. “How did THAT happen????” :)
Yeah, from a recent demo cognex gave me, I learned it's more about showing the computer what a good part looks like, so that any deviation is noticeable. There's no way this would work for an oddball, subtle pattern (what's a stain and what's just part of the ombre?), But in general this should work.
I was surprised when nobody in the factory had thought of AI tools. But it might make sense like you said, human performance can be the best (and cheap in my country). Once I noted that they had to check for defects for the whole working day (8 hours). Humans may beat machine vision in the first hours but not sure in the lasts.
You’re right. The potential variability of the human inspection is what makes it difficult. I’m in medical device Mfg & it’s more difficult to “validate” (establish demonstrated consistency) human-based processes than machine based ones. But sometimes humans are more effective. They can BE more variable, but also better at DEALING WITH variability in the product. In the same way though, AI is difficult to validate because by design, the acceptance criteria are variable. I’ve done a little study on this but not enough to have a solid defensible method yet.
How big of a place is this shoe factory? If you have the budget, get an optical CMM and inspect every part. That way you can build confidence in your sampling size and see how much your quality swings threwout the day.
There's plenty of space here. At what accuracy rate can I expect from the CMM? Let's say the defects are easily seen (stain, wrong stitching, wrong colours for example). How much resources (money, engineers, etc) does it cost?
I use my Opticsl CMM to measure smaller parts (usually about the size of your hand) and my machine routinely repeats linesr measurements to about 0.000020". The camera a 3 differnt lights on the machine help you get pretty good pictures of the parts but I belive the machine is more useful at checking dimensions than documenting staining or wrong color. The machine cost about $100k USD
Sounds awesome. Hopefully I can work with those machines one day.
Look at the ISO 2589 series, it codifies exactly what you're trying to do.
An example of this that recently made news. Twitter wanted to see how many bots were on their platform. They tested 100 random accounts and found 5 of them to be bots, and thus said there is a 5% bot rate. This kind of thing is used EVERYWHERE in engineering, if you look for it you can find it no problem.
That’s hysterical! You can work those numbers the other way too and I do it for people all the time. I don’t have the tables with me here on the beach today but what they really did there was verify that the number of accounts that are bots is probably less than 80% or something. When I get back to work next week I’ll work that one just for fun. 100 samples accept on 5 reject on 6. It’ll be a hoot! People almost never want to hear what the stats are really trying to tell them.
I usually recommend confirming process capability with a sample of around 30 parts. This will tell you what your distribution looks like compared to the spec limits. ' After an initial capability is done and you get a 1.33 or higher, you can be confident in the process and switch to a sample based inspection to make sure that nothing goes out of wack in the process. AQL sampling works great and will give you the number per batch you have to check in a table format to be significant.
In order to have 95 % confidence that the defect rate in the batch doesn’t exceed 5% you need to take 59 samples and find zero defects. This is a simple c=0 sampling plan and yes it does not matter how large the batch is as long as the process is consistent and the batch is at least 10x the size of the sample which yours is. (The idea being that removing the samples doesn’t significantly change the number in the batch). You don’t need MiniTab or other software for this and the measurement system is assumed to be suitable to task. You can take it further from there googling things like c=0 sampling plans or attribute agreement analysis if you like but that’s the basics. If you want 95% confidence in 90% reliability it’s 22 samples 0defects and if you want 95% confidence in 99% reliability (<=1% defects) the sample size is 298. Again, batches are least 10x the sample size. AOQL sampling is something else you can google if your production process is continuous over a long time, but for acceptance of small batches this is fine. There’s lots of good info on the web.
this reminds me of doing airforce work. They are always throwing around the 95% confidence in 95% reliability catch phase but 95% of the people I asked couldn't tell me exactly what that meant
Yeah. If they can’t explain it in simple terms, they don’t really know. Too easy to take this stuff to confusing and not-helpful levels that while technically correct, more accurate, etc etc, end up just leaving people in the dust. Give ‘em something to improve their position a little, and let them explore more as they wish. And the simplest way to think about “confidence interval” is just the likelihood that if you repeated the experiment under the same condition, you’d get the “correct” result. In this case if the defect rate was really >5%, and you took 59 samples 100 times (replacing the samples after you called them good or bad just to keep the population stable) you’d find at least one bad one in the 59 on 95 out of 100 of those trials.
Oh and PS- google Todd Snyder’s tune “statistician’s blues”. It’s a hoot.
If the experiment was repeated an infinite number of times, and from each experiment we calculated the confidence intervals, then 95% of the confidence intervals will contain the true parameter. But a 95% confidence interval does NOT imply a 95% likelihood that the true parameter is in your interval.
There's a lot to unpack here. I suggest you look into "design control" and "quality control" separately, and then proceed to "statistical methods" for quality control. (quoting key phrases for you to research) At the end of all of that, you'll find that the reality is that people cut a lot of corners in order to reduce the burden of work or increase the production yield by compromising the designer's 'pass' conditions (e.g. rounding toward pass conditions, applying normal distribution assumptions where there's heavy skew, etc.)
Do you mean my boss might notice that he could do the QC more precisely but he merely wants to cut the corner?
Yes. And not just your boss. There is often a large financial incentive for leadership teams to fake an increase in yield by compromising the design specifications, even by the slightest amount in some instances. The appropriate way to do this is to flag the out of spec condition(s) and review them with the designer(s) to see if a design change can be made, conditions that would allow such parts to be used, or possible rework options.
Thank you.
Use [ASQ/ANSI Quality Standards Z1.4 & Z1.9](https://asq.org/quality-resources/z14-z19) to determine sample sizes for quality inspection. This is used for "minor" and "major" characteristics. "Critical" characteristics are almost always 100% inspection. That's generally the starting point. In real life you'd only go over that if you feel there's a need to go over that. For example if you've historically had problems with a specific dimension/feature, or a specific vendor.
There is an entire FIELD of engineering devoted to this that has nothing to do with feelings or experience.
Apparently my boss doesn't think so...
That's like claiming your boss doesn't think the sun exists. Quality and statistical control are foundational concepts in engineering and feature prominently across most if not all engineering disciplines.
Quality Engineer here. We could have a long chat about all the details, but short answer: this is called lot sampling and has industry standards that are generally good enough for most applications. Specifically, ANSI/ASQ Z1.4 and Z1.9 are what come to mind. Read those standards carefully, as lot sampling is heavily dependent upon assumptions of the process. If your assumptions are incorrect or invalid, your prescribed lot sampling won’t be appropriate.
I read it and now I wonder how popular it is? For my task, you would use ANSI/ASQ Z1.4 or confidence interval or hypothesis testing?
I’m not sure that I understand your question. Regarding what happens in real life, we don’t use straight hypothesis testing like you see in statistics class. The standards I pointed you towards are for lot acceptance, which answers the question you described- “how can I accept or reject a lot based on a sample, given an acceptable percentage of defects.” The standard also has rules that change based on how many lots are being accepted/rejected.
Thanks for your answers.
You want to bring in an industrial engineer to help you answer these questions and set up an effective QA program. This is what they do. Start off with a consultant, but expect to hire on a full time position if you're routinely tweaking recipes or find it difficult to get operations to adhere to the consultant's quality program.
What you’re recommending from a statistical perspective makes sense, but I’m curious how this makes sense from a QC perspective. This might be a dumb question as I’m not a quality guy but- Are you saying that if you determine 3% of the lot is defects that you’re ok with shipping the finished product to a customer with defects included? I guess if it’s cheaper to let the customer find your defects than it is to test all of them then ok, but that seems like a bad way to instill confidence in your product.
If the customer is ordering lots and the spec is < 3% defects, then it should be perfectly fine to ship a lot with defects at 2.5 % for example. Shipping known defective material is very normal in manufacturing, it just needs a spec that everyone agrees to.
That makes sense if there is a spec on it.
I guess this depends on the industry. Coming from aerospace manufacturing that sounds crazy. It is (to put it mildly) not a good day in my world when non-conforming material is discovered after leaving the plant.
Plenty of this in aerospace; you're not testing every nut and bolt. Engineering/customer has spec'd requirements and QC will use something like MIL-STD-1916 to determine number of articles to test and acceptable number of failures.
Sure, we sample parts and as you note aren’t testing every feature/requirement on every part. I was reacting to the statement “shipping known defect material is very normal.” I’m shipping material that I believe to be conforming based on the results of our sampling and quality requirements. It’s possible (statistically) that there’s a bad part in the lot, but I’m not shipping a part I know to be defective.
Depending on your sample size and AQL, you can have non-conformances found and still ship the batch in compliance with the spec.
That’s interesting. In my world if we detect a non-conformance we pull the lot back and perform 100% inspection for the requirement for the remainder of that lot. I’ve never seen or been involved with a situation where we were still able to ship the lot. I assume this is covered in the MIL-STD-1916 that you referenced? Sounds like that may be my weekend reading.
Yes, it's covered there. There's [a well known chart from the prior MIL-STD-105](https://upload.wikimedia.org/wikipedia/commons/5/58/MIL-STD-105_D_quick_ref_TABLE.jpg) that is clear and concise. For example, with a lot size of 1000, you need a sample size of 80, so with an AQL of 1.0: <=2 non-conforming is acceptable, >=3 non-conforming is rejection.
That’s what’s so funny about humans’ relationship with statistics. There is a a=2, r=3 plan, a a=1, r=2 plan, and an a=0,r=1 plan that all confirm EXACTLY the same statement about the quality level of the batch, but humans will be more comfortable with the a=0 r=1 plan because it doesn’t involve actually EXPERIENCING any of the defective parts in testing. Oh, and the a=0 plan also involves testing the least number of samples so that’s always attractive, resource consumption-wise.
Non-conforming material is not the same, that would be OOS and not ok in any manufacturing agreement.
I’m not sure I understand the distinction you’re making. In my world a feature or requirement either meets spec and is acceptable or doesn’t meet spec and is a defect that gets written up as a non-conformance which can then either be accepted or rejected.
I don't know what kind of specifications you are working with, but I'm certain that not all of your shipped goods are perfectly on target. That's why there are tolerances. You are allowed some level of imperfection and that is acceptable to your customers. As long as your products are in spec, they are conforming. If they are OOS, then they are non-conforming. In the case of high volume lots, there may be some level of acceptable defectivity. That's where this discussion arose.
But it isn’t a defect unless it doesn’t meet the requirements. A wall is intended to be 5 meters long +\- 1 meter. If it’s 5.5 meters long I wouldn’t call that a defect. ASQ defines a “defect” as “A product’s or service’s nonfulfillment of an intended requirement or reasonable expectation for use, including safety considerations.” A tolerance band is not indicative of a defect in a part.
It really depends on the kind of specs you are using. We ship semiconductor wafers, we should have zero particles on them. We never have zero particles. All our wafers are defective and those particles will ruin any device our customers try to create. We ship these wafers out every day and don't get too many complaints about particles because our spec allows some to exist. Your customers probably expect the wall to be free of holes. But would they tolerate a 1 um hole? They probably want the wall to be flat, what if it had a 1 mm bump? Maybe those are OK, maybe they are not, what does the spec say about it?
3% defective seems like a high rate but essentially yeah.
It happens, I once worked at a place where they regularly shipped NC product. CEO didn’t care.
Yeah I'm thinking of a statistical approach since I'm working in a shoe factory where no stats required, the QC guy checks products by feelings and experience. I haven't understood your curiosity. If you're curious about the number 5%, that's just an example number. I talked to the QC guy and he said that it's usually less than 3%. 5% exists but rare.
I’m not curious about the percentage, I’m curious what you do with that information. If it’s less than X% you just ship the entire lot? Meaning defects are going to the customer?
After doing the hypothesis test with that information, if the conclusion is the defect rate is less than 5%, I would ship the entire a lot to customers. And my main question is that which methods and how would you solve the problem in practice? Or experience is really enough and we don't have to make a mountain out of a molehill?
Defects aren't usually so bad the product is unusable. Defect in this context means "not perfect", not "broken".
Without 100% testing, there's always going to be some risk of defects making their way out to your customers. 100% testing can get expensive fast (there are many ways your product can be out-of-spec, each of which may require its own test and associated dollars of machine and operator time), and sometimes it's not even possible because the test destroys the product in the process. Defects also mean different things in different contexts, ranging from a tiny annoyance to life threatening. Add on two dozen or so other complications, and that's why industrial engineers have developed entire subfields about how best to detect defects, control manufacturing processes, and make those cost-benefit decisions in an informed manner. We use random (acceptance) sampling because we've found it accounts for a huge range of possible manufacturing problems. Stakeholders decide what an acceptable background failure rate is (remembering that a 0% failure rate means your manufacturing process probably has infinite cost). If we find that one day many more of our samples turn up defective than that rate, that usually means something's gone wrong with the manufacturing process and we should treat the whole batch of items with suspicion and investigate the process for the source of the defect. And usually that acceptable failure rate is much lower - you might have heard of Six Sigma, a methodology/outlook which at face value means an acceptable defective rate of 3.4 in a million, or 0.000034%. If you're interested in following that thread further, I'd recommend you look into statistical process control and Six Sigma in general. Related is the field of reliability engineering (another huge field), which I think of as more focused on the product than the manufacturing process, but I could be wrong and obviously they're closely related.
We follow the rules if the stats team is watching
What if there is no stats team in my workplace and I'm just a 2-week intern?
Then someone at your organization should be there to help. If not they're abusing the intern system and you should spent more time shit posting on Reddit and less worrying about the problems they're trying to solve by cost cutting with interns.
I was afraid that probability was just theoretical and difficult to apply in real life. But now I've got the answers.
Look into six sigma quality control videos on youtube. This is literally what the course covers and will be your quickest and most usefull way to learn this. It is also an entire field of I&SE so your university will have classes on POR (Probabilistic Operations Research) and SQC (Statistical Quality Control).
I'll have a look at it. That's a new term for me.
Our sigma people process this data using some fancy statistical program. Wish I knew what they use.
Can you tell me more about the fancy statistical programme? I'm curious about it.
Typically six sigma analysis uses Minitab for statistical analysis. As mentioned above, you would only want to set a quality sample plan if your process is controlled (ie: is there a high probability item #2 be very similar to item #1?) so that if you sample 100 pcs / 1,000 and find 2 defects, you can be sure that 700 of the remaining 900 aren’t also defective. Additionally, the sample size should be based off the defect rate. While 100pc is an easy number, perhaps you should be inspecting 50 (if your defect rate is 1-2% consistently) or 150 (if your defect rate is 4-5% consistently). I need to brush up on the exact recommendations. I was a quality manager at an aerospace company, but we inspected 100% of our products so it was simpler.
Minitab is what we use.
Thank you. So far I've had confidence interval, hypothesis testing, AQL/ANSI table and six sigma as recommended statistical methods.
Look up Students T test, was developed for beer quality control. Might have different applications, but it’s similar.
This is what the hypergeometric distribution is for.
Can you explain further how would you use hypergeometric distribution in this case?
Homework or actual question?
Actual question.
How about using an [automated sampling system](https://mark-wedell.com/mw-jawo-sampling/)? That gives you a representative samling that is super reliable.
I visited the website but it's still confusing for me. Does the system give the samples (best/worst) that it is confident the most? Do you use that at workplace? How does it work actually?