Hey @chrsmth!
As it turns out, there does exist a tool in JMP to do the calculation @MRB3855 described for you. It just involves a bit of creative usage.
The Nonparametric Demonstration Reliability explorer (DOE>>Sample Size Explorers>>Reliability>>Nonparametric Reliability Demonstration) can easily perform the calculation you need. Just set Max Failures to 0, Alpha to 1-confidence level (in this case, 0.05), make sure the drop-down menu next to Solve for reads Sample Size, and then put 1-p into the Demonstration Reliability box. It should then give you the Sample Size (rounded up the next integer) you need. See the example below:
This explorer is used to determine how many sample sizes you need to demonstrate a given reliability (1-defect rate) while allowing for a maximum number of failures. It is essentially the complement to your situation, where you want to know how many samples you need to demonstrate a given defect rate while allowing for a minimum number of failures. So, in short, it is directly performing the trick @MRB3855 described.
In general, by putting in the complement of the defect rate and one less than the minimum number of defects you would like in the Max Failures entry, you should get the sample size you're looking for in any variation of your situation. Not exactly the direct tool you may have wanted, but if I have described the procedure well enough, this should still answer your question with little effort on your part (you just have to compute 1-p).