Business Problem Example
In the typical PTW analysis you might end up with something that looks like the image below. You estimated where the Independent Cost Estimate (ICE) is, the lowest credible price, where the you and competition will end up on evaluated capability (score), and where everyone will be on price.
Let us say the quantification of that picture looks like the below where A/B/C are the competitors.
| Element/ Company | A | B | You | C |
| Score | 80 | 88 | 90 | 95 |
| Price | $875k | $800k | $1M | $1.2M |
| Score Differential | -10 | -2 | – | +5 |
| Price Differential | -12.5% | -20% | – | +20% |
With the table above we make a few observations:
- Company A has a higher price and lower score than Company B and thus is unlikely to win
- Company C has a small score advantage but has a staggering price premium
- Company B has a healthy discount off our price at what could be perceived as a negligible score differential
Now before we move on be aware that a prudent approach, depending on the complexity and size of the opportunity, might be to do some scenario analysis on you and your competition. The scenario analysis will help you understand the ramifications of say, Company B having a range of scores between 80 and 90. If the probability of a 90 is high then we have to lower our price. If the probability is high of an 80 then we may need to lower our price, but not as much compared to a 90 score. But how much do we need to lower it?
GovPTW GAO Protest Based Product Explanation
Well it would be great if we knew how much a 2 point score premium is worth. Most people would probably correctly guess that a 20% price premium is not worth 2 points on a 100 point scale. In that case we would need to bring our price down, but down to what? Do we go down to $800k or can we get say $850k or $875k with our 2 point premium? That is precisely what the GovPTW Government Accountability Office (GAO) Protest based PTW product we offer helps you do. Let us demonstrate with some concrete examples.
GAO Protest – Tables (Data)
Below is the price/score excerpt from GAO Protest file number B-420443; B-420443.2. In this protest it clearly shows the assigned score and price (yes, the Government actually does assign scores in some instances). The math is simple: A 2 point advantage (scaled to 100) with a 6.5% lower price. RSI-QT (the winner) left money on the table. Also below is the price/score excerpt from GAO Protest file number B-420358; B-420358.2. This is using adjectival ratings which can be converted into a quantitative score and thus a differential computed between it and price. UCOR won with an equal or higher rating compared to all competitors for each evaluated factor. OREP had a lower score and a slightly higher price (losing proposition). The unnamed offerors had a lower score and price, but the prices were lower by just a couple percentage points. Their price had to be lower to entice the Government to bite on a lower score. When we harvest hundreds of these decisions and data points, and run them through a model, we can ascertain the Probability of Win (pWin) at any intersection of price and capability.
GAO Protest – Outcomes Types
Now before we move on let us explore a few issues with GAO Protest data you might be wondering about, and explain how we work through them here at GovPTW.
Is the data flawed because these are protested decisions? That is a very valid question. Let us review the types of GAO protest outcomes. A good reed for more information is the GAO Protest FAQ page.
- Denied – found no merit to the protest
- Dismissed – technical or procedural flaw (such as lack of timeliness or jurisdiction)
- Sustained – protest is upheld
- Split – part of the protest is sustained while others are not
- Granted – usually regarding attorney fees
- Reimburse – usually regarding attorney fees
- Withdrawn – protestor backed out
At GovPTW we only incorporate Denied/Dismissed protest data. If anything, a review by GAO that confirms the procurement was carried out correctly validates the decision, and in turn the data is reliable enough for us to use. Sustained/split decisions have something wrong with them and thus have no value in our dataset.
GAO Protest – Excluded Points
What about Lowest Price Technically Acceptable (LPTA) and Best Value Tradeoff (BVT) evaluations? What do you do with those? Since LPTA makes no tradeoffs on non-price factors, it adds nothing to our data. Therefore we omit it and focus solely on BVT protest data. We further limit the data to only BVT decisions where non-price factors are more important than price. Decisions where non-price factors are equal to or weighted less than price are omitted.
Does every Dismissed/Denied protest have a table like the ones shown earlier? Another great question and the answer is no. Not every Dismissed/Denied protest even has a publicly available write up to review. As of the time of publishing this article, over the past almost 20 year there are over 6,000 protest decisions with a written decision (not just a one word outcome). Of those, over 4,500 do not have a price/score table and thus are unusable for our purposes. Of the approximately 1,500 that remain there are some that are LPTA and/or were not dismissed/denied, which we throw out. Then, some tables show the details of multiple bidders giving us multiple trade off decision data points in the same GAO protest. The result is there are over 1,000 BVT data points to harvest, and growing!
GovPTW Product Explanation
When we assemble enough points together and apply a probability model to it we get our product. Our website’s home page shows the image below but it is worth explaining here.
- Solid color quadrants
- Top left – A higher price (positive price differential being above the X axis) at a lower score. What right minded person would select such a bid? Thus the GAO Protest data and model output shows that quadrant to be a losing proposition, rightfully so.
- Bottom right – A lower price (negative price differential being below the X axis) at a higher score. A no brainer to select these bids right? But wait a minute, you just left money on the table and instead could have bid a higher price and still won.
- Trade off quadrants
- Top right – A higher score is only worth so much price premium. Stay closer to the competition’s price depending on score and your pWin is higher. Fly ‘too close to the sun’ and try to command a price premium that your score and/or Government does not deem worth it, and your pWin falls.
- Bottom left – This is simply the inverse of the top right quadrant. Except, instead of being the ‘lead dog’ with a higher score, you have a lower evaluated score/capability. To differentiate yourself you need to lower your price, but by how much? The product will help you quantify that.
- Now the chart has a clear 45 degree gradient in these two quadrants. This is simply to help visualize the trade off occurring in those quadrants, the actual data and model output is more nuanced than that. The product includes a cell by cell Probability of Win (pWin) estimate. Say your PTW model puts you 15 points higher than the competition. Go to the +15 point marker on the X axis (Score Differential) and look up the Y axis (Price) value. You will see the pWin estimate at various price differential (%) markers (e.g. 5%, 6%, 10%, etc). You can also look left and right to see what the pWin looks like at a 10/11/12 or 18/19/20 point differential. Those are the insights our product provides which you can use to inform your positioning, and your ultimate price.
In the next and final article we will use a theoretical example to show the GovPTW product in a PTW analysis