In his post "Who gets promoted - Mr. 3.76 or Ms. 3.79?" Mike points out the heart of the issue:
Well, first off, nobody is terrible at everything (and even if they are, who will have the heart to say so?), so the lowest overall scores across your company will probably sit somewhere in the range of 2.0-2.5 (remember, "3" is supposedly average).On the other end of the continuum, you have more hope. After all, who wouldn't want to give their star player all 5's? Still, you're probably looking at overall highs in the 4.0-4.5 range. So suddenly your scale has shrunk from 1.0-5.0 to around 2.0 - 4.5, and 70% of your 40,000 employees fall somewhere between 3.2 and 3.6.
So what should you do? As Mike suggests you could use Behavioral Rating Scales for taking the decision. However, I would also suggest that for a promotion using a rating of a current job is a little useless piece of data.
Lets take the classic example of promoting a salesperson to a sales manager position. The competencies needed for the new role is fundamentally different from the past role's? No matter how well you've done the performance appraisal for the current role, promoting on its basis for the next role is fraught with danger.
One thing you could do before a promotion is assess the person for the competencies for the next role. However that is easier said than done, specially when there are 40,000 employees up for promotion. And what if the person is really not interested in that role? Can you give the person a growth in complexity of the role rather than the hierarchy of the role?
Unfortunately organizations are really not thinking along these lines. One reason is that would mean giving up control in the career development area to the employees.
But, would that really be such a bad thing?