I'm not certain that doing reviews at all is a good idea. Left to myself, I'd look for some way to reward entire teams, and empower them to most or all of what a traditional manager does. I don't think my company is ready for such a radical change, so I and my peers have worked out some techniques that (I think) give us a more effective take on the traditional PR.
Hard work warning: I believe that the only way to do a decent review is to put a lot of effort into it over a long period of time. Ideally, the annual review should summarize dozens conversations and observations that take place throughout the year.
Here are some of the keys as I see them. These are all things that I have applied or that I am currently applying, with varying degrees of success:
Focus the goals - Each developer has a different sub-goal (or two) to focus on for a given time period. For example, each of the four major goals for my group has 10-20 bullet points that it covers. Telling people that they need to "focus" on 50+ behaviors means that they will call out what they will make up a narrative that shows they did everything on the list, rather than trying to improve anything in particular.
Ask each team member to choose only one (two at most) behaviors or sub-goals to focus on, and reword them to apply to the individual's project.
Change over time - The goals should shift throughout the year. Try to change them quarterly as a guideline. Allow people to decide that a certain goal isn't as worthwhile and change it sooner, while others find that they take longer and extend a goal into the next quarter.
Individual ownership - The master list of goals should be built through conversations with the developers. Even if the major goals are handed down from on high (ours were), you should still be able to make them more specific for a development group, and it is imperative that the people in the group have a say in their wording.
Review at least monthly - People, especially busy software developers, will forget long-term things like goals. Having a monthly 1x1 where you review how they're doing really helps keep things on track.
Focus on strengths, not weaknesses - Encourage each person to focus on things that they are either already good at, or that they're passionate about being good at. Focusing on "problem areas" usually means that you're asking people to spend more time on things they don't like doing. A recipe for failure.
Embedded management - I'm less sure that this is essential, but I think it is a big help in my case. Having manager/developers allows us to pair with the people on our teams regularly. It makes it easier to "call bullshit" on exaggerated self-reviews, but it also makes it much easier for us to see the little things that our teammates are doing every day to add value, so we can call them out on the reviews.
These things seem to work well for my team. The alternative was more subjective - gathering feedback from stakeholders at the end of the review period. I'm still doing that, but I pick it up in casual conversation now instead of waiting for the end of the fiscal year. This allows me to call out problems while there is still time for people to fix them, rather than forcing them to defend themselves when it's too late.
