Policies & Procedures - 6.2.1 Evaluation Process

- updated 02/08 -

The IEEE requires the Fellow Evaluation Committee of each Society to submit an individual evaluation form for every candidate who has had a nomination submitted to their Society, plus a summary rank ordering of the nominees (with no “ties”). The evaluation procedures recommended here for the ComSoc Fellow Evaluation Committee (FEC) will produce the required result. The methodology described is intended to supplement the rules found in the IEEE policies and procedures documents for standing committees, such as those found at http://www.comsoc.org/about/documents/pp.  Additional FEC recommendations are described here as refinements of the procedure that the ComSoc FEC has used successfully. Considerable flexibility in the application of these recommendations is expected, but a tight schedule is always imperative as there are typically more than 70 ComSoc nominations to consider, and all results must be delivered to the IEEE before 15 June. Suggestions for improvements are welcome and should be directed to the current chair as identified at http://www.comsoc.org/about/memberprograms/fellows.

• Step 1: The Chair sends a FedEx Priority package with every nomination form to each Committee member within two days of receipt of the paper nomination forms from the IEEE. A little over three weeks later the chair receives from each committee member a division of the candidate pool into (nearly) equal quartiles.

This can be especially difficult because so many of the nominees are well qualified; however, unless this is done there will be little hope of producing the absolute ranking ultimately required. Because of the quality of nominees, the members of the lower quartiles are not considered “unqualified” but simply less qualified than their peers. In this respect, our initial quartiles do not have the same interpretation as the final IEEE group definitions.

A number of parallel activities usually take place in conjunction with Step 1. Committee members review the set of IEEE Instructions provided, especially the restrictions in the “Code of Ethics.” To avoid conflict of interest, the IEEE Fellow Code of Ethics requires that any Technical Society Evaluating Committee Member who is a business associate of an IEEE Fellow Candidate being evaluated for ranking in the Society should make this relationship known to the Committee and withdraw from acting on the specific candidate’s rating. Obviously, a committee member in the same organizational direct reporting chain as a nominee needs to withdraw, but, especially in very large organizations, conflict of interest is more difficult to assess. These issues are resolved and relevant conflict situations documented.

The committee is sensitive to the fact that many industry candidates, especially Application Engineers/Practitioners and Technical Leaders (typically about 15% of the submissions) do not have the same type of opportunity to produce an extensive, generally available record of publications as those, for example, in a university environment. Candidates adjudged to be in the AE/P or TL categories are identified and then ranked somewhat separately and against each other. After initial evaluation, the committee also determines if some candidates might be more appropriately (and beneficially) placed in classifications other than those indicated on the application. As will be described, candidate classification is an important ranking influence. Ultimately, all groups are carefully interleaved so as to not penalize any specific group.

As an aid to ranking, various citation statistics are gathered and distributed for each candidate. It is important to consider the citation statistics in combination with the above-mentioned categorization. Publication is important in a university-research setting, but for many practitioners and leaders, publication is not at all a priority and we take this into account. The weight given to citations needs to be considerably different depending upon the candidates grouping. All lists are eventually interleaved during the review process so as to not penalize the strong practitioner or industrial leader for lack of publication, and to simultaneously give university-based candidates proper credit for recognized achievements.

• Step 2: The chair accumulates the Step 1 quartile rankings produced by each committee member, counts quartile rankings for each candidate, and circulates to all committee members a list of candidates ordered lexicographically by quartile-based histogram (so [7,2,1,0], indicating for a specific candidate 7 first-quartile votes and 2 second-quartile votes etc., precedes [4,0,2,4]). This method of ranking serves as an important diagnostic tool; it helps the chair identify an advocate for each nominee, and to organize the process of collecting and disseminating information about the different candidates. Many cultures are represented in the committee and this process makes it easier for a committee member to be heard.

The tabulation by histogram also associates an initial “agreement level” with each candidate. Histograms such as [10,0,0,0] or [0,0,0,10] (meaning all 10 reviewers placed this candidate in the first or last quartile) indicate full agreement among committee members, but a histogram such as [3,2,2,3] indicates that more discussion is needed in order to reach consensus.

As an additional aid, another ranking reminiscent of “grade-point average” is created by assigning weights  of 1 to 4 to each of the candidate’s quartile counts, and then dividing by the number of members who actually rated the candidate. Although it is intended that every committee member become familiar with and rank every candidate, there are usually a few cases (i.e., conflict) where that is not possible. This “GPA” method produces a ranking that is more independent of the number of rankers.

• Step 3: With the tabulated data as assembled by the chair in hand, a teleconference is held a few days later for the entire committee. During this teleconference, advocates for each candidate are selected from among the committee, and this member then has the task of eventually completing the Individual Evaluation Form for that candidate. Collection of any needed additional information is assigned to committee members for distribution by email. During this first teleconference, conflict-of-interest issues are resolved, and a consolidated AE/P-TL list is produced to aid in the second round. At this point, after appropriate discussion, a large segment of the lower rated population is set aside. Unless significant new information appears, this lower group does not receive any further extensive consideration.  

• Step 4: Prior to a second teleconference, the chair receives from each committee member another division into quartiles of just the higher-rated portions of all the candidate pools. The chair integrates this input and circulates the updated rankings to the committee. These second lists are then refined and all lists are interleaved as part of the second teleconference cycle. This is done in such a way so as to not penalize the strong practitioner or industrial leader for lack of publication, and to simultaneously give university-based candidates proper credit for achievements as recognized by others.

• Step 5: Experience has shown that a third teleconference is usually needed to obtain closure and before the final rankings are forwarded to IEEE. After a third teleconference, a single, unified final ranking of all candidates is agreed. The chair produces the Summary Sheet (1 page) and assembles the 70+ Individual Evaluation Forms as received from and reviewed by the committee members.

As a result of the effort to be proactively inclusive of all aspects of our profession, past experience has shown that of the highest rated candidates (i.e., those ultimately rated in the Exceptionally Qualified-EQ and Highly Qualified-HQ categories), typically as many as half may have been considered to have been supported by industrial, application, or leadership credentials.

• Step 6: The Chair forwards the ranking and evaluation forms to the IEEE where the Fellow Committee (http://www.ieee.org/web/membership/fellows/committee.html) produces the final selection.

In addition to the process outlined above, there are some additional issues that should be considered:

• Replies from references are not made available to the Evaluation Committee. However, on individual request basis, additional information on candidates can be requested. Nominators should therefore understand the importance of the individual nomination form in pointing out what the candidate has done, and why these accomplishments have had an extraordinary impact on the profession.
• Nominators should understand that for those candidates where it is appropriate, a measure of publications which have had a significant history of multiple citations may have a positive influence.
• In addition to paper distribution, electronic files of nominations (when they exist) should also be accessible to the committee as backup.
• New committee members should be appointed before the candidate nomination cycle to prevent new members from being inadvertently listed as references or nominators.