Whether or not it's a problem depends on the specifics of how your overall grading system works. I've always used a system where each member distributes an average of 10 points to the OTHER members and, when you have a really weak member, the opposite is more of a concern (i.e. it's an advantage to the "workers"). That's because:
1) Almost with exception, the non-contributors spread their points out as evenly as the distribution rules will allow. Thus, none of the "workers" gets a score that, by itself, is likely to have a significant impact on his or her grade.
2) the non-contributor will get very few points from the "workers"--thus, they (the "workers") have more points to distribute to each other."
In general, my advice would be to tell the worried students to be honest in their evaluations and let nature take its course. If you don't think justice has been done, then you might want to take an action that corrects the problem--but still treats everyone in the class in the same way. For example, you might consider is tossing out EVERYONE's highest and lowest scores and see if that solves the problem without creating any inequities in other groups.
I hope this helps. Good luck.
Larry