One of the key features of public procurement law and practice is that the system is designed to produce predictable and transparent outcomes. The expectation is that the stated tender evaluation system should generate an answer in a manner that is protected from inappropriate interference and should be seen as predictable by bidders. It should also be possible for those wanting to contract to understand in advance how the choice of one evaluation model or another would affect the sort of offer that is likely to be accepted. It is strange, therefore, that systems commonly applied should so often operate unpredictably. It is also rather strange that this is the subject of so little comment given that the problems are quite well known.
In this post, Michael Bowsher QC considers two problems arising from the use of a common formula for evaluating price.
Two recent publications have highlighted issues with one of the most commonly used formulae for identifying the most economically advantageous tender (see Moreau, “La question de la régularité de la méthode proportionelle d’évaluation du critère du prix” (2014) Contracts et Marchés Publics 11, and Kiiver & Kodym, “The Practice of Public Procurement” (2014)). These issues are encountered fairly frequently in practice but because they are the consequences of an evaluation process that bidders sign up to at the outset, it is usually too late to raise them in a challenge at the end of the process.
When evaluating tenders by reference to price and quality criteria, it is necessary to apply some means of tying together the two analyses so as to take account of trade-offs between them. This is often done by some arithmetic means which allows quality and price scores to be drawn together in a total score representing the entire evaluation for each bid. To do this, the price expressed in currency units must be translated into a point score and this is often done by use of the following formula.
| Price score | = | Lowest price bid | x | Price weighting |
| Price of tender being evaluated |
This can then be added to the total quality score (also multiplied by the appropriate weighting) to give the total.
Take an example in which the scores are weighted 60% for price and 40% for quality. The following prices and scores are submitted and awarded.
| Price | Price Score ex 60 | Quality Score ex 40 | Total ex 100 | |
| A | 400 | 45 | 38 | 83 |
| B | 350 | 51.4 | 32 | 83.4 |
| C | 300 | 60 | 20 | 80 |
Bidder B wins with 83.4 points.
If the same three bids are submitted in the same tender, but now a bidder D submits a bid with a price of 250 currency units and is awarded a quality score of 15, the price scores and overall totals change as follows.
| Price score ex 60 | Quality score ex 40 | Total ex 100 | ||
| A | (25/40×60) | 37.5 | 38 | 75.5 |
| B | (25/35×60) | 42.86 | 32 | 74.86 |
| C | (25/30×60) | 50 | 25 | 70 |
| D | 60 | 15 | 75 |
Bidder A now wins. Bidder B has put in the identical bid but its bid is marked as being less economically advantageous than that of Bidder A even though neither of these bids has changed. The relative advantage of Bidder A over Bidder B (and vice versa) is determined by an essentially irrelevant factor, namely the existence and score of the bid put in by Bidder D.
Of course, depending on the rules of the bid Bidder B might in the second scenario claim that Bidder D’s bid was abnormally low or failed to meet a relevant quality threshold so that it should not be treated as a valid bid. Whatever the rights and wrongs of this debate, the fact remains that the outcome of the contest between the two highest quality bids turns on the price of the lowest price bid.
There are simpler oddities with this formula. Take three priced bids:
| Scores | |||
| A | 100 | 50/100 | = 50% |
| B | 75 | 50/75 | = 67% |
| C | 50 | 50/50 | = 100% |
One might expect that the middle bid would receive a score of 75%, but because the formula is not a linear function its price score is lower than would probably be thought “fair”. This of course substantially prejudices bids in the middle of any range of bids scored this way. A bidder cannot know whether it is likely to be so prejudiced until the end of the process as it cannot know where in the range its price falls, but in principle it does know that it is exposed to the risk of such prejudice. From the purchaser’s perspective, the likelihood of the formula selecting a “Goldilocks” bid that is “just right” lying mid-range in price and quality is probably reduced by this effect.
This is, of course, not the only formula of this type but I am yet to find one which does not raise some odd outcome in some situation or another.
These issues raise real and pressing issues for contracting authorities planning their procurements and ceiling to predict the outcomes they are looking for. They raise difficulties for bidders trying to establish a winning bid strategy.
From a legal perspective they also challenge the assumption underlying the Altmark 4th criterion that the outcome of a public procurement will reflect the best economic solution and any contract entered into will therefore be free from state aid. How can that be when the outcome of that procurement is not determined by the relative economic merits of the bids but rather by the positioning of another quite separate bid?
In future blogs, Michael and Practical Law will look at problems arising from other common formulae used to evaluate price and consider what this means for tenders more generally.
Hi Michael,
My company recently suffered from problem one above in that the lowest bidder was less than 50% of the winning bid and thus altered the ranking of our bid versus the winning bid. If the third party price was 53% or over the ranking would have changed in our favour. One obvious solution to me is that the least advantageous bid is eliminated on the first stage evaluation, using the formula, and then a direct comparison is made between the remaining bidders. If necessary, the basic evaluation could be repeated if there are more than three bidders but the final choice should be made by direct comparison of two bidders using the formula. This might not cover all scenarios but it does help. Alternatively, exceptionally low bids should be dealt with in a special way e.g. a different formula should be used that is more robust.
Regards,
Tim.
Thank you for a very helpful post. I have encountered this silly situation many times (my company tenders more than once per month). The net effect is that the cheapest bid almost always wins, even when quality scores are really quite low.
Given that in my case we are tendering for complex, mission-critical software systems, this is poor. Quite often, on the occasions when my company loses, the end users of the winning solution are visibly upset to have to use a system which they believe to be substandard.
This is a very useful post and provokes some thought on the matter. I am currently looking at using this model to replace an averaging and standard deviation model used areas of Scotland (I believe), which appears a bit ambiguous.
The model you suggest is one I’m very familiar with and so are the issues. One way around this for us procurement people or purchasers is to pre-determine the type of bids we’re going to receive based on market research and/or to place emphasis on what is important in the solution we are trying to procure. For example;
If I am procuring some specialist IT equipment and services, I might use Porters 5 Forces to confirm our stance in the market place and therefore I know it is a very competitive market and there is good chance there will be less experienced resellers (or substitute suppliers) in the mix. Additionally, in this example I am keen to ensure I am well protected with regards to IT security. In order to protect this I will therefore place greater emphasis on quality over price by using a high quality weighting i.e. 75% or higher leaving only 25% for price. This way it will be difficult for any bidder to ‘undercut’ and get ahead with a poor quality response.
I think It’s important for procurement to engage with suppliers and understand the market before sticking their necks out and seeking a solution.
Assuming all bids are valid and viable, scenario with bidder D makes sense for me. When there is bigger gap between min and max price that means that requirements are not precise enough, otherwise all bidders should calculate more similar prices. In such case it is good that top quality bid A won.
In case gap is small, you can afford to calculate and accept lower quality, as there is smaller risk of scope misunderstanding.
However there is a problem with someone deliberately submitting low price bid to help top quality bidder (which is not legal).
My company came to a similar conclusion a few years ago. We have advocated and lead a couple of Authorities to use a technique we call ‘Real Value For Money’ (RVfM) on a number of multi-million GBP decisions. We believe the method has none of the weaknesses described, so well, above. I attempted to describe them myself last year http://www.cd.qinetiq.com/blog-post/2016/05/09/is-rvfm-a-solution-to-flawed-meat-models/. Yours is much better!
My colleague, Andy White, just held (today) a webinar on the subject attended by over 100 people. It was well worth the 30 minutes of watching. We definitely need to spread the word.
Hi Michael,
“but I am yet to find one which does not raise some odd outcome in some situation or another” – have you heard of MC:MU? Marginal Cost:Marginal utility; used with some insight into the utility aspect (qualitative criteria weighted based on differentiation potential rather than ‘importance’ and (at times) taking risk into consideration – this (in my 20+ years of experience) has never raised an odd outcome. (but, maybe I’ve just been lucky!)