The methodology behind BusinessWeek's three-part test that uncovers and ranks the best employers for college undergraduates
For young men and women graduating from college, choosing a post-graduation employer is both exciting and perilous—never more so than when economic turmoil raises the stakes considerably. In such an environment, information matters, and this year's Best Places to Launch a Career ranking provides information in spades.
A few things have changed since our 2007 ranking, including the addition of 24 new employers. One thing that hasn't changed, though, is our basic methodology. While we made a few tweaks this year, it's still a three-part test designed to ferret out the best employers for new college graduates.
To narrow the field, we turn first to the nation's career-services directors. In February we conducted a nationwide survey of directors at top colleges and universities, public and private, in every academic discipline. The survey asked the directors to identify the top five employers for entry-level workers in 17 industries, and the top 20 top employers overall. Of the 263 directors surveyed, 60 replied, for a response rate of 23%.
Using the survey data on individual industries, BusinessWeek compiled a list of nearly 174 companies, nonprofits, and government agencies that were eligible for the final ranking. This was a two-step process. The first step involved identifying the five high scorers in each industry. For every career-services director who ranked a company No.?1 in its industry, the company received five points. Every No.?2 ranking was worth four points, every No.?3 ranking was worth three points, and so on. After tallying the points, including ties, we had a total of 83 companies, due to an inadequate number of employers in two industries, Internet and hospitality.
Adding to the List
To flesh out the list, we reviewed the remaining companies suggested by the career-services directors and identified those with the highest industry point totals among all the industries. That added an additional 91 employers to the list, for a total of 174.
Next, we invited each employer to answer an extensive survey seeking information on hiring, pay, benefits, training programs, and retention. Of the 174 organizations, 119 completed the survey, for a response rate of 68%. We then compared the employers' responses to each question with the responses of others in its industry.
For each ranking question on the survey, the best response, such as the highest pay or retention rate, was awarded 10 points. The worst response, or no response at all, received no points. Responses that fell in the middle of the pack were worth five points each.
To complete the ranking, we combined the results of the employer survey with two additional pieces of data. From the survey of career-services directors, we tallied the points for each respondent's overall ranking of employers—20 points for each No.?1 ranking, 19 points for each No.?2 ranking, and so on. From Universum Communications, a Philadelphia research firm that surveyed more than 40,000 U.S. undergrads this year, we received the percentage of students who listed each employer among the five for whom they would most like to work.
The employer survey contributes 50% of the final ranking, while the career-services survey and the student survey contribute 25% each.
Volatility in the Rankings
If no career-services director included an employer among the top 20 overall, or if no students included it among their top five, such employers forfeited the 25% associated with that survey. If data from both surveys were unavailable, the employer was placed at the bottom of the ranking, with its place determined solely by the employer survey. This year, 57 employers had data from all three surveys, 50 had data from two, and 12 were ranked based on their employer survey alone.
Several factors, including the addition of 24 new employers to the ranking, helped create a great deal of volatility this year, with about half of the employers who returned from last year's ranking making big swings either up or down in the ranking this year.
In many cases this was the result of employers doing better or worse on the student or career-services surveys. Marriott, for example, shot from No.?45 to No.?7 in part because it fared far better with students than it did last year. Since new competitors in an industry sometimes fare better on one or more surveys than existing employers, their presence is often enough to shake up the ranking. The retailing segment, for example, doubled in size with the addition of Sears, Abercrombie & Fitch, and Target, which almost single-handedly knocked Walgreens, Macy's, and Kohl's down 17, 31, and 47 spots, respectively.
This year, BusinessWeek made several minor modifications to its methodology. First, we added several new questions to the employer survey and eliminated others. We added a new industry category—media and entertainment—to accommodate new companies that participated for the first time this year, and shifted one company, Walt Disney, into the new category from hospitality. We also changed how we scored the employer surveys in the nonprofit/government category. Instead of eliminating many questions because they did not apply to both nonprofits like the Peace Corps and government agencies such as the IRS, we counted every question. This is why the Peace Corps fared far worse this year than it did last year.
Less Chance for Errors
Finally, we made a slight change in the way we scored the employer surveys from each industry. In the first two years of the ranking, we used a complex set of rules to determine how many points to award to each company for each question based on their responses. Last year, we awarded 10 points for the best answer—the highest pay, for example—and no points for the worst, with all others receiving 5 points. We considered no answer at all to be the worst answer for scoring purposes, so if a company did not answer a question, it received no points, the best answer continued to receive 10 points, and all others received 5. There were several exceptions to this general rule—mainly concerning follow-up questions—that made the scoring process needlessly complex.
This year, to reduce the complexity and the possibility for error, we awarded no points to employers who declined to answer the question and to employers whose answer was the worst among those that did answer the question. So, for example, if four companies responded to our question about the maximum 401(k) match—with answers of 8%, 8%, 2%, and NA—both the 2% answer and the NA answer received no points for that question. While some companies lost points on individual questions as a result of the change, our back-testing showed it had very little effect on their overall performance in the employer survey, or on the ranking as a whole. And since all employers in all industries were treated the same way, the change did not confer a benefit to anyone.
Business Exchange related topics: