Should linear programming be used to reduce subjectivity in the supplier selection process?

The linear programming technique (Data Envelopment Analysis) objectively determines the weights of evaluation criteria after suppliers are scored as opposed to the traditional process where the buyer pre-determines the evaluation criteria weightings before suppliers are scored

Introduction

In this article, I illustrate, through example, a linear programming technique called Data Envelopment Analysis (“DEA”) to score and rank suppliers. This approach is premised on a mathematical model that objectively assigns evaluation criteria weights after suppliers have been scored on evaluation criteria.

It has been argued that DEA provides fairer and more robust supplier selection results because of its objectivity and its ability to offer a fuller relative evaluation of suppliers against each other.

I personally have never seen or heard of this approach in my procurement career — in fact determining evaluation weights after suppliers have been scored would raise a big red flag for any procurement person ! So I wanted to understand it more.

In this article, I outline the approach, intuition and motivation behind DEA and then consider its usefulness in procurement evaluation.

This article is for people who:

• understand procurement and are interested in a more objective way to score suppliers in an evaluation process
• understand linear programming and are interested to see its application in a procurement process

The dummy scores I will be using

For this section, I will be illustrating a traditional versus DEA approach for supplier selection based on the following made-up scores:

Made-up scores (raw)

I attempted to replicated some supplier ‘types’ in these scores to make them more meaningful:

• ‘Supplier E ’ (all the bells and whistles): Meets all needs very well but at a high cost
• ‘Supplier B’ (the OK incumbent): Scores OK on all criteria with a price similar to current costs
• ‘Supplier G’ (cheap but scored poorly): Least expensive but low confidence that they can actually deliver

For the example, I normalised all scores to between 0 and 1:

Made-up scores (transformed)

There are many ways to treat Price such as not providing a score at all. However, for this example, I have scored Price by indexing against the lowest tendered price ($650,000). For example, the score of Supplier A is calculated as $650,000 / $950,000 = 68% while Supplier G gets 100%, because it quoted the lowest price.

Traditional Approach: Pre-Determined Weightings

The approach

Any fair evaluation process requires buyers to pre-determine evaluation weights before receiving any proposals from suppliers.

Buyers should not choose their weights after proposals come back because buyers may choose weights that favour certain suppliers which makes the process unfair.

The final scores calculated from the traditional approach are a simple sum product of the scores against the pre-determined weight.

The traditional process would rank ‘Supplier D’ as the top supplier.

This makes total sense — why do we need to over-complicate it?

Proponents of the DEA approach have argued that pre-determining weights has its problems:

• Pre-determined weights are too subjective: How weights are determined is quite subjective, leading to lower transparency in the procurement process
• Pre-determined weights can still ‘favour’ bidders: Buyers can still ‘favour’ some suppliers by assigning higher weights to criteria that will benefit only some suppliers

The same authors observe that the DEA approach, on the other hand, provides every supplier

‘…the right to be evaluated according to the set of weights that favor her against the other competitors’

As such, this was deemed preferable to the traditional approach because:

• Weights are more objective: The set of weights are determined by a pre-defined (linear programming) algorithm
• Equal treatment for suppliers: It guaranteed equal treatment of all suppliers
• Full relative evaluation: It allowed a full relative evaluation by accounting for the advantages of every bid

So how does this look in our example?

DEA approach: Optimised Weightings

Overview of the DEA approach

1. Find optimal weightings for each supplier: Find weightings for each supplier that maximises their final weighted score through a linear programming technique
2. Calculate the supplier scores based against the optimal weightings of itself and those of all other suppliers: For example, 8 different scores for Supplier A would be calculated by multiplying the score of Supplier A against the different weightings of Suppliers A through to Supplier H
3. Calculate final score and rank: Average the scores calculated in Step 2 to obtain the final score for each supplier

Step 1 : Determine the optimised weights

The appeal of linear programming is that once a decision problem is represented by a linear programming model, you are guaranteed a mathematically optimal result. You can do this with just a few clicks in Excel’s Solver package for simple problems.

The linear programming model components for applying DEA to supplier selection are as follows:

1. Objective: The supplier score
2. Variables: The supplier evaluation criteria weights (each supplier will have different weights to maximise their weighted score)
3. Constraints: Constraints vary across research papers — I felt that this approach, which allowed buyers to rank criteria rather than not assigning any weighting at all, more reflective of reality (buyers generally have some idea of what is important, even if they don’t know the exact weighting). I also added a constraint to cap the weighting on Price as allowing Price to make up 100% of the criteria is inappropriate in most cases (and make most procurement professionals balk!)

Therefore, the model written for ‘Supplier A’ looks like this:

Linear program model to determine optimum weights for ‘Supplier A’

You can then input the above model and determine the optimum weights ‘w’ in Excel using its Solver package. Doing this for all suppliers gave the following results:

Optimised weights and score for each supplier

The optimised score is higher than the original scores (calculated with the traditional approach) for all suppliers due to Solver finding the optimal criteria weights to maximise each supplier’s scores.

Note, for example, that Solver assigned a 40% Price weighting (the maximum weighting that Price could take) to Supplier G which had a Price score of 100%. If we did not add in this constraint, Solver would have allocated 100% weighting to Price for Supplier G.

Step 2 & 3: Calculate the final score

For each supplier, a total of 8 different scores are calculated based on its score multiplied weighted by the optimal weights of all other suppliers. For example, the 8 different scores calculated for Supplier A are:

1. Supplier A score x Supplier A optimised weightings
2. Supplier A score x Supplier B optimised weightings
3. Supplier A score x Supplier C optimised weightings

…

8. Supplier A score x Supplier H optimised weightings

The final score for A is the average of all 8 scores above — this is the DEA ‘cross-efficiency’ score.

Final DEA — CE score; for example 70% is calculated by the sum product of ‘Supplier A’ scores and ‘Supplier B’ optimal weightings

Comparison of Results

The figure below overlays the final score resulting from the Traditional and DEA approach with the individual criteria scores of each supplier.

Both approaches lead to the same top 4 and bottom 4 so at a high level, the approaches have resulted in similar ranked result.

However, there is a swap in the 1st and 2nd ranking between the approaches. In this hypothetical example, the difference is only 1% but for the sake of comparison which one better aligns with our intuition for which supplier should be the ‘best’?

The intuition of the traditional result

In the traditional approach, Supplier H came 2nd to Supplier D because its scores didn’t perform as well against the pre-determined weights.

However , if the original weighs of Price — 30% and Systems and Resources — 10% were changed slightly to Price — 35% and Systems and Resources — 10% , then ‘Supplier H’ score would be tied to ‘Supplier D’!

This is just a shift of 5% of weighting and arguably quite arbitrary. I doubt that any buyer can confidently say that Systems and Resources — 10% is a much more ‘correct’ weight than Systems and Resources — 10%.

The intuition of the DEA results

In the DEA approach, Supplier H came 1st because it achieved the best performance against the weights that favoured itself as well as the weights that favoured all its competitors. Intuitively, I think of it as Supplier H competes well against the strengths of all other suppliers and therefore has the best all-rounder performance.

Supplier H has a low score for ‘Innovation’ and ‘Systems and Resources’ but this did not matter because other suppliers also scored low in these areas. This means that the optimal weighs for these criteria were generally low across all suppliers — Solver did not allocate high weights to these areas as high weights in these areas would not have maximised the scores of most suppliers.

If other suppliers were strong in ‘Innovation’ and ‘Systems and Resources’ (resulting in a higher weights allocated to these criteria), then Supplier H would certainly not have been ranked 1st. Alternatively, if we specified that these criteria must have higher weights than the others as a model constraint, Supplier H would also not have been ranked 1st.

Final thoughts — is it worth it?

Personally, I agree with proponents of the DEA approach that it is a more objective ranking and selecting suppliers than the traditional method. I also feel that it has a stronger logic for objectively determining the ‘best’ supplier.

However, its key weakness is in the implementation:

• Time and effort to implement: Calculating the optimised weights is cumbersome in Excel (the optimal weights need to be solved individually for each supplier). This is unsustainable for a large number of suppliers but open source programs (e.g. the Python PuLP package) or optimization software can streamline this process. However, this requires some optimization knowledge and, in case of commercial software, added costs.
• Relatively difficult to explain: The intuition of DEA is much less obvious than the traditional approach and is likely to take some time for stakeholders to understand and accept

Overall, this makes the method difficult to ‘sell’ especially when the additional benefits may not be significantly more than the traditional process. Therefore, it’s not surprising that the only references to this approach are in academia.

However, I don’t think this method should be completely discounted — especially when the following apply to a procurement process:

• A desire to have extra rigour, confidence and fairness in the supplier selection process (e.g. highly public, high stakes procurement)
• Evaluation criteria where weights are arbitrary or cannot be determined
• Evaluation criteria where pre-determining weights risks favouring some suppliers over others