College football playoff: Lessons from evaluating teams as a mock selection committee member

GRAPEVINE, Texas — I was a student reporter at Baylor in 2014 when the final graphic on ESPN hit No. 4 Ohio State. The Bears, infamously co-champions of the Big 12 with TCU, found themselves sitting on the outside of the first College Football Playoff. Eight years later, much to my dismay, the College Football Playoff mock selection committee did the exact same thing.

But we’ll get to that.

Twelve members of the media, including myself, were invited to participate in the annual mock pick held by the College Football Playoff. The exercise takes place behind the doors of the same elegant boardroom at the Gaylord Texan as the actual committee that holds the fate of countless sports departments in its hands.

For the day, I played the role of Navy athletic director and current CFP committee member Chet Gladchuk. “Be a bit of a prick to play the part of Chet,” joked a CFP employee of the 72-year-old whose place I lived in. But if they wanted to be elegant, a deep rehash of the controversial 2014 playoffs was the perfect place to start.

The challenge of ranking the four “best” teams

The process for ranking the teams has been covered at length, but here’s a quick breakdown: The standings are divided into seven groups — each three positions 1-9 and each four 10-25. We vote to consider six teams at a time for each trio. The first team: Alabama, Ohio State, Oregon, Florida State, Baylor and TCU. As it should be.

CFP President and NC State Athletic Director Boo Corrigan tasked us with finding the 25 “best” teams in the country. From the beginning, however, it became clear that each of us had different definitions of this metric. Some valued the success of the season, while others wanted the best team from Selection Sunday. Regardless, resume ended up being a far more important consideration than which team would win each Saturday.

Discussing the criteria with some of the most famous names in the sport was probably the part I was most unprepared for. Former players in the room were adamant that Florida State’s undefeated season was an asset. I disagreed. Try to be a friend like me who tells Pro Bowl running back Deuce McAllister and former first round quarterback EJ Manuel that winning isn’t everything because of metrics.

The CFP features an analytical system created by SportSource Analytics that can compare up to four teams at a time with a wide range of metrics on giant screens in front of the team. When comparing Baylor vs. TCU vs. Ohio State vs. Florida State appeared on the screen to delineate the No. 3 and No. 4 spots, everything that happened in 2014 made sense.

Jeyarajah (left) and former NFL linebacker Kirk Morrison

Kevin Jairaj/College Football Playoffs

The teams’ schedules are displayed on the page with a results list and common opponents or opponents highlighted. However, the most striking part of the page is a color gradient with teams highlighted from green (good) to red (bad). When the schedules came for TCU and Baylor, the amount of red relative to the rest of the field was glaring.

Never underestimate the power of data visualization!

Additionally, one of the only true criteria listed by the committee is that similar teams with head-to-head wins or conference championships are given priority. CFP director Bill Hancock confirmed to the group that since Baylor and TCU were both submitted as co-championships by the Big 12, the teams should effectively be treated to 0.5 titles each.

Compared to Ohio State and Florida State, the decision wasn’t difficult. We can argue about the specific strength of the timeline and evaluation metrics the committee uses, but it gave us a complete picture of the committee’s decision.

Perhaps guided by Manuel Florida State’s hand in the room, the ‘Noles actually moved up to No. 2 in our mock rankings, behind only Alabama. I voted Florida State No. 5. The 12 metrics CFP identified as most correlated with victory hated FSU and their narrow margins of victory weren’t enough to win me over other (better) conference champions.

Full disclosure, I tended to lean more on performance numbers and quality of play than other voters. SportSource Analytics organized and color-coded the stats most associated with winning after extensive historical research, with relative score and games per point leading the way. (A special teams metric was added to the committee’s top 12 stat factors at the behest of former Nebraska coach Tom Osborne.) Other members of the mock committee talked about the quality of their program. Some leaned on quarterback play or tried to highlight NFL prospects.

Ultimately, this diversity of opinion is the point. Seven of the 13 CFP committee members are athletic directors. Seven are former college football players, including former NFL player and MIT mathematician Ph. D John Urschel. Two are coaches. One, former USA Today columnist Kelly Whiteside, is a journalist. Everyone sees the game differently.

The mix of former players, writers and TV personalities gave our team a similar interesting context. San Diego State legend and NFL vet Kirk Morrison wanted to make sure the Top Five prospects got the attention they needed. The Athletic’s Ari Wasserman asked us to look at recruiting gaps. AL.com editor John Talty begged us to remember that Katy Perry was at The Grove for Ole Miss’ game against Alabama. McAllister and former Michigan quarterback Devin Gardner pushed us to think about winning above all else. I mentioned that Ole Miss was outscored 30-0 by a Bret Bielema Arkansas team that pulled off a so-called “marginal love” win over Texas. Everyone has their role.

NC State athletic director and CFP chairman Boo Corrigan

Kevin Jairaj/College Football Playoffs

How to rank the best from the rest

If you’re wondering why there might be moments of inconsistency in a ranking, that’s why. Each secret ballot involves different voters weighing their own criteria while trying to convince other voters that their view is correct. And, really, that made the conversation with No. 7-25 infinitely more interesting and competitive.

“When I was the athletic director at Army, we would have been ecstatic to be in the top 25,” Corrigan told the team. “We have to pay attention to the bottom of the rankings as much as the top.”

That’s what we did. The procedure required it.

Some participants tried to bring the power of the conference into the picture or reject certain programs because they were from the Group of Five. CFP Executive Director Bill Hancock and Corrigan quickly shot them down. After digging into the mappings and seeing the programming gradient from green to red again, it became frankly clear that it wasn’t necessary.

The data visualization had a team record above .500 against the committee’s previous top 25 and previous top 10, but in reality, there were very few of these high-level data points to choose from. Instead, the programs that were full of green caught our attention. Ranking games — or borderline ranking games — were discussed on a case-by-case basis.

We drew Boise State and some were shocked by the number of quality opponents in a “Group of Five” conference. It went up. By contrast, Wisconsin’s “Power Five” program ran red. SEC contenders Ole Miss and Georgia were littered with green wins and red losses. Michigan State and Kansas State didn’t have green wins, but their only blemish came against the nation’s top teams.

After each round, we would vote for new teams. At the end, we had the opportunity to discuss any major disparities. UCLA was initially ranked behind an Arizona State team that the Bruins routed in Tempe, Arizona. This has been fixed. Georgia moved lower, while Arizona moved up several spots. Marshall entered the field after comparing well to teams like Minnesota and Louisville in the 12 factors.

By the way, this conversation lasted five hours. The real thing involves six weeks of talking, many hours of discussion, and countless weekends of watching bits of every football-related game imaginable. Hancock noted that there was one committee member who decided to grade each player individually in each game he watched. Every year is different.

What will change (and what won’t)

Of course, that takes us to the 12-team playoffs. In many ways, the decisions the committee makes are about to change. With six automatic bids, there will be less emphasis on finding out which teams are fielding and more emphasis on team rankings and seeding.

When the playoffs are extended, the committee will remain at 13 members. They still plan to field 25 teams and feel comfortable that at least six conference champions will fall within that number. Instead of filling the bowl pairings with teams below the division, they will now fill at-large playoff spots. Basically, the process won’t change much each time the playoffs are extended, which those in the room hope will happen sooner rather than later.

Eventually, I gained a lot of respect for the process. It’s not perfect, it’s not overwhelming, but it’s thorough. With the amount of information and videos available to committee members, I feel more confident that they have all the necessary tools to make the best decisions possible. More importantly, the secret ballot process makes it a bit more difficult to pull the ranking for branding or matching purposes. Transparency won’t quell the conspiracy theories, but it makes me more confident in the process.

And to Hancock, whenever you need a new name to add to your list of committee members … I’m available.

Leave a Comment

Your email address will not be published.