panel. Each individual reviewer can also nominate applicants for discussion for any reason they want.
5) The panel then brings together all of the reviewers to discuss the applicants with the goal of moving the borderline applicants to either the award/don't award pile. Each applicant was discussed in turn (starting with UG->1G->2G). One of the original 3 reviewers summarized the application for the panel, the other 2 reviewers added their comments, and the entire panel then discussed. Ultimately, based on this discussion, the original 3 reviewers could change their scores or comments.
6) After this process, the rank-ordered list of applicants was sent to NSF. At that point, I'm not sure how awards are actually assigned - if it's just based on the z-scores or if other criteria (e.g. maximum number of students per panel) are used.
Some comments on actual reviewers, here:
1) Reviewers included assistant/associate/full professors, a couple of lecturers/non-tenure track faculty, I think one or two scientists from industry, and faculty from a variety of institutions (R1, SLAC, etc.). So there was a broad perspective represented on the panel.
2) Similarly, reviewers had highly diverse expertise, meaning that on average each reviewer would not be an expert in the research area of each applicant. Reviewers were given the option to transfer an applicant to another reviewer on the panel if they felt that it would be a better research fit (I had one or two applicants transferred to me).
3) Reviewers were screened for conflicts of interest - I was not allowed to review or participate in the discussion for any applicant from my university or for which I had another conflict of interest (collaborators, someone who had rotated through my group, etc.). During the discussion, you would be sent to a breakout session to avoid hearing the discussion of applicants with a conflict of interest.
Finally, some comments on what was discussed/what reviewers looked for. Obviously, each applicant posed different research questions and evaluation challenges, so these are just broad strokes comments mostly focused on aspects that may not be obvious.
1) We were instructed to review applications holistically to determine which applicants would have the highest likelihood of becoming impactful scientists. In other words, we were explicitly told to think of this process as more similar to graduate admissions than a NSF proposal review panel - the research statement is only one component of the application
Well, Damn. This makes so much sense now.