Analyzing the ComSoc Conference Portfolio
In this President’s Page, I am pleased to introduce Stefano Bregni, the past Vice President for Conferences (2018–2019, 2020–2021) and currently Director for Conferences Operations (2022–2023) of the IEEE Communications Society, who will present the analysis we are carrying out on our Conference Portfolio. Prof. Stefano Bregni is with Politecnico di Milano, Italy. He contributed to ETSI/ITU-T Standards on network synchronization, and is an author of 100+ papers and of a recognized book on network synchronization (Wiley, 2002). He has served as a tireless IEEE volunteer for more than 25 years. He is or served as General Vice-Chair of GLOBECOM 2022, Technical Program Co-Chair of GLOBECOM 2023 and ICC 2016, Technical Program Vice-Chair of GLOBECOM 2012 and GLOBECOM 2009, Symposium Co-Chair in nine other ICC/GLOBECOMs and Chair of the Steering Committee of the IEEE Latin-American Conference on Communications (IEEE LATINCOM). He received the IEEE ComSoc/KICS Exemplary Global Service Award (2019) and the IEEE ComSoc Hal Sobol Award for Exemplary Service to Meetings & Conferences (2014).
The ComSoc Conference Portfolio
In addition to Publications (Journals and Magazines), Conferences are the main medium for ComSoc Members to find, exchange, and distribute technical content and results of their scientific research. Conferences sustain ComSoc’s overall activities through their financial returns. In fact, every year conferences yield about half of the total revenues in the ComSoc budget, including their immediate direct revenues and the long-term revenue from the publication of their Proceedings on IEEE Xplore.
The ComSoc Conference Portfolio now includes some 40 conferences and various technical events held every year, of which ComSoc is full or partial financial sponsor. The full list is posted on the ComSoc web site including links to the respective sites. The Conference Portfolio is without doubt one of the main assets of our Society. It includes general conferences, covering the entire ComSoc scope of topics, as well as more specialized conferences, focusing on some specific subjects or even narrow topics. It also includes Regional Conferences, aimed at serving specific regions of the world, while retaining full global participation by authors and in technical program composition. ComSoc conferences really interpret the principle of diversity in its broadest meaning, in terms of geographical coverage, technical topics, and the composition of the technical program committee and attendance.
Our Portfolio of Conferences is overall successful and holds a leading position in such a competitive arena. Our conferences have consolidated a reputation of technical excellence and good organization. This is largely due to the dedication of many volunteers, who contribute a significant amount of their time to guarantee this success, and to the professionalism of ComSoc Staff members.
Conference Portfolio Analysis
In 2018, during my first term as Vice-President for Conferences and with Hikmet Sari as Director for Conference Operations, we started a thorough process of consolidation and analysis of the ComSoc Conference Portfolio. In particular, we evaluated the financial and technical performance of all our Portfolio Conferences except Industry Summits and Forums, by focusing initially on a basic set of Key Performance Indicators (KPI) over the last 5 years. The basic set of selected KPIs collected for each conference is the following.
Basic Indicators of Financial and Technical Performance
- Attendance indicators: number of paid registrations, number of complimentary registrations, number of total registrations, percentage of complimentary over total registrations.
- Financial indicators: revenues, expenses, and surplus (absolute and percentage over expenses).
- Technical indicators: number of submitted papers, number of accepted papers, and acceptance ratio.
Since the first data collection, the analysis has provided some significant insight into the operation and performance of our conferences, which is essential for the future planning and strategies. For example, a stable good financial performance (surplus above 25% over expenses in most conferences) and good selectivity (paper acceptance ratio firmly below 40%) of our flagship conferences ICC and GLOBECOM has been confirmed throughout all the periods analyzed. Moreover, INFOCOM has been confirmed as one of the most selective conferences in the ComSoc Portfolio, with a paper acceptance ratio in the range of 15–21%, which is more commonly seen in high-quality journals rather than conferences. In contrast to several excellence peaks, some problems were also identified, such as relatively high paper acceptance ratios or repeated financial losses, but limited to a few conferences. Follow-up actions were carried out to solve problems identified in all those specific cases.
In 2019 and 2020, the Portfolio Analysis was refined by introducing further technical indicators, which go deeper into the technical performance evaluation of our conferences. These additional technical KPIs are the following.
Further Indicators of Technical Performance
- Number of days from the paper submission deadline to acceptance notification (for each conference);
- Number of days from conference end to submission of Proceedings to IEEE Xplore (for each conference);
- Number of unique peer reviewers of papers submitted (for each conference);
- Average number of peer reviews per paper (for each conference);
- Number of IEEE Xplore downloads (for each conference series, time averaged);
- Revenues distributed back to ComSoc for IEEE Xplore downloads (for each conference series, time averaged);
- A citation index of articles published in each conference series (e.g., the Google Scholar h5-index).
While we recognize that this list of technical KPIs is not exhaustive and definitely can be further broadened, we believe that they highlight key aspects of the general understanding of “technical quality,” which are of most interest and significance to our attendees and Members.
Determining the preceding indicators is not as trivial as it may seem at first glance. For example, for any given conference, the number of IEEE Xplore downloads has a long tail and increases with time. Averages over moving time windows are reported by IEEE.
Evaluation of Conference Impact: h5-Index Analysis
As everyone reading this probably knows already, the h-index of a publication is the largest number h such that at least h articles in that publication were cited at least h times each. The h5-index, then, is the h-index of only those articles that were published in the last five complete calendar years. We consider the h5-index as computed by Google Scholar. The h5-index as of July 2023 reported here, therefore, is the largest number h such that h articles published in 2018–2022 have at least h citations each in the Google Scholar library, for any given publication.
Figure 1 lists the top 20 publications ranked by h5-index in the category “Computer Networks & Wireless Communication” (source: Google Scholar, July 2023). Besides the fact that among those 20 publications nearly two thirds are by our IEEE Communications Society, these results confirm the excellent technical quality of our best conferences, in particular INFOCOM, ICC, and GLOBECOM. Their h5-index is even higher than that of well-reputed journals.
On the other hand, the Google h5-index analysis may also highlight “areas for improvement” for our Society. For example, ComSoc has 0 publications, either conferences or journals, in the category “Computer Security & Cryptography,” only 2 (co-sponsored) in the category “Computing Systems,” only 1 (co-sponsored) in the category “Optics & Photonics.” Those categories are very important and closely related to Communications.
The bar diagram in Fig. 2 compares the h5-index of ComSoc Portfolio conferences, when available, with different colors to highlight categories. The three top conferences INFOCOM, ICC and GLOBECOM are significantly higher than all others. It is worth remarking that the different h5-index reported for ICC and GLOBECOM is probably due to their different calendar timing (May/June and December, respectively), since they are basically the same conference held twice a year.
An Ongoing Work
What is reported here is just an excerpt of a much bigger work done. Even more, this is just the beginning of a huge work still to be done! All performance indicators have been evaluated and compared, in order to assess the “health status” of all our conferences. Because of the limited space available, in this President’s Page it has been not possible to report more than these little tidbits. However, having complete data for all conferences held in the last year is not enough. To compare the performance of conference series one against the others, it is necessary to collect complete data at least for the last three, if not four or five years, then combine their results over those years to get average indicators.
The performance analysis over time is needed, conference by conference, because isolated incidents may always happen, but repeated problems must be investigated and fixed. We are committed to do that. Our priority is to improve to better serve ComSoc Members. Thank you.