IA Summit 2007 Review Process

Overview
General guidelines
Review Criteria
Reviewing system information

Thanks again for volunteering to help out reviewing proposals for the 2007 Information Architecture Summit.

First Steps:

Please login to your START Account (that's the reviewing system). You will have a set of reviews assigned to you (under the link 'Your assignments'). You will have around 15 assignments. Focus on these and if you finish, review others (see the reviewing system information for more details about how to find others).
Read through the guidelines below. They should give you an overview of how to approach scoring reviews. The online review form on START contains a shortened version of these guidelines to make reviewing easier, but the longer version below may provide some additional context.
Dive in and get started! Some people find that doing a few at a time is a useful strategy. Warning - waiting till the end of the review period is commonly known to cause angst, stress, and bad karma. Luckily, we have a large team this year to spread out the effort.
November 30th is the target deadline for completing reviews. If you find that this isn't feasible for you, please let Donna Maurer (ia07 AT maadmob DOT net) or Chris Farnum (crfarnum AT yahoo DOT com) know as soon as possible so your to-do list may be reassigned.

General Review Guidelines

Here are a few guidelines for the review process.

"Blind" Review Phase

This round of the review process is "blind" - meaning that you should not have access to the names and bios for the presenters. We've asked all those who submit proposals to remove names and identifying info (such as employer names) from their session descriptions. The purpose of this phase of the review is to collect scores and comments from a diverse set of people who represent a cross-section of the IA community. Note: this is the first time in the history of the IA Summit that the first round of reviewing has been "blind". Many other conferences are organized this way. We're trying it this year to remove any doubts about bias and/or favoritism in the process.

Of course, it IS important to consider WHO is presenting and what their qualifications are when putting together a conference. It just won't be part of the first round of reviews this time. In a subsequent round, the steering committee will take the collected scores and comments from the first phase, along with the additional bio information, to design a conference schedule. This second decision-making step is a blend of art and science (much like IA), and it absolutely depends on the quality of the first review phase.

Regarding "Conflict of Interest"

Since presenters names will be kept anonymous for this round of review, you'll base your score on the abstracts and detailed session descriptions they provide. Ideally these will have all personal names and company names removed. However, in some cases it may be easy enough to guess the identity of certain people. This is unavoidable, but in such cases please avoid reviewing when you feel that there is a conflict of interest. In essence, this means you should not review a submission if you know that you have a personal friendship/relationship with the presenter(s) OR if you have a current employment relationship with the presenter(s). A conflict of interest can either predispose you to give high scores or low scores to a proposal. Not sure if something constitutes a conflict of interest? Feel free to ask Donna Maurer (ia07 AT maadmob DOT net) or Chris Farnum (crfarnum AT yahoo DOT com). If you know that you need to pass on a doing a review because of a conflict of interest, just leave the review blank and it can later be reassigned.

Approach

Each proposal will have an abstract and an attached proposal document. The proposal document should contain additional information about the presentation, including why it is suitable for the IA Summit. Please read this document as well as the abstract.

Please take a balanced approach when reading these papers. One objective is to have a solid technical program; hence, it is important to be thorough. On the other hand, we also want a broad and interesting program - so please do not be picky. You were selected to serve on the committee because you are a leader in your field. You will read papers which are not up to your personal standards. Some may still have technical merit, and could be interesting to others. Please try to keep an open mind. Don't get hung up about recommending too many papers - if you think a paper will be interesting to attendees, then recommend it.

Numerical Scoring

The first four questions on the review form ask you to supply a number 1-5, where "1" is lowest/weakest/negative and "5" is highest/strongest/most positive.

Low 1 2 3 4 5 High

When filling in your review, try to avoid scoring ALL of your proposals in the middle of the range. It makes it difficult for the steering committee to make final decisions if tons of proposals are "3s". So, please don't equivocate. Take a stand, and don't be shy about using the extreme ends of the scale. Either you like a proposal or you don't.

Revising Your Reviews

You can make changes at any time until the end of the review period.

Review Criteria

Here is a detailed explanation of each criteria and what to look for:

1. Contribution to IA practice, theory and community (Low 1 2 3 4 5 High)

A presentation can make a contribution to the field if it is about:

Information Architecture examples and case studies from the real world - this most often means IA on the Web, but could also mean the IA of a grocery store, a cell phone application, a social network
Information Architecture theory, concepts, and definitions
DOING IA - including techniques, processes, and deliverables
Being a professional who has a career practicing IA or who manages people who practice IA

Note - First and foremost the IA Summit is about Information Architecture. Admittedly IA is eclectic and draws on many different disciplines. Not everyone who does IA or wants to learn about IA calls himself/herself an IA. That's great, this isn't an exclusive club and we hope the conference is an opportunity cross pollinate with other specialties and disciplines related to UX (user experience). Presentations that present concepts from outside the typical boundaries of IA are welcome, but the proposal should make a solid case for WHY a topic is relevant to information architecture.

2. Value and appeal to conference attendees (Low 1 2 3 4 5 High)

This score is based on how well you feel the proposal meets the needs of its intended audience, whether it's targeted toward beginners or veterans. As a reviewer, it's natural to think about what you want to see at the conference yourself. But remember that selecting presentations for the conference is itself an exercise in creating a user experience, and you should keep in mind that not everyone who attends the conference will have the same interests, opinions, and level of experience as you. Many people who attend the conference want to hear the latest thinking on cutting-edge advanced topics. Obviously the conference should deliver on this expectation. But we should also include presentations for those who are new to the conference and want an introduction to the "core" topics of IA. (Typically, a high proportion of attendees are first-timers.) So don't necessarily discount a presentation that covers ground similar to something you saw two or three years ago if it is meant to appeal to new IAs. Presentations provide value in different ways. Some of the best presentations follow very traditional models, but some of the best also stray from convention to include performance, humor, collaboration, and even competition. Keep an open mind about the type of value a presentation might provide to attendees.

3. Original, inspirational or thought-provoking content (Low 1 2 3 4 5 High)

We want people who attend the IA Summit to come away energized with new ideas about IA and the inspiration to apply them. A presentation has the makings of success if it is at least one of the following:

Original - introduces fresh ideas and techniques to the field
Inspirational - generates excitement about a topic (even if the material is more basic than cutting edge)
Thought provoking - provides a new perspective to consider or advances an existing debate or theory

4. Quality of proposal (Low 1 2 3 4 5 High)

The quality of a proposal is a partial indicator of the quality of the presentation. Please consider:

Is there sufficient information in the proposal to gain a good idea what the presentation will cover?
Has the author considered the style of conference and needs of the audience?
Is the proposal well structured, easy to read and understand?

5. Would you recommend this presentation be included in the program? (No, Maybe, Yes)

This question is your chance to make a recommendation. It also helps to account for the fact that some reviewers tend to give higher scores on average and others lower scores.

No - a weak proposal that doesn't meet the submission criteria.
Maybe - a proposal that has merit, but some flaws; it should be compared against presentations on similar topics before you'd feel comfortable endorsing.
Yes - a strong solid proposal that you feel confident deserves a place on the program.

6. Comments/feedback for the person who submitted the proposal

Include any suggestions, praise, and constructive criticisms for the presenter(s) - this information will be send with the accept or reject notice.

Please remember that people who submit proposals almost always mean well and almost always are interested in attending the conference. We want to encourage people who submit a proposal, but don't make the cut, to have a positive experience with the process regardless. In a strictly business sense, most people who submit proposals are potential "customers" for the IA Summit - not the kind we'd want to alienate.

7. Comments/feedback for the review committee

Include any comments you think would be useful for other members of the selection committee about how/why you scored the proposal as you did or questions you would like the authors to be asked.

Review system information

Each reviewer has been assigned a number of proposals - this makes sure all proposals have a minimum number of reviewers. You can find these in your START Account under the link 'Your assignments'.

If you would like to review other submissions, you can find them with the 'Submission List' link or the 'Detailed Submission List' link.