Search engine evaluation: 2006-2007 comparison

Personal Evaluations of Search Engines:

Google, Yahoo! and Live (MSN)

 

Bing Liu

Department of Computer Science
University of Illinois at Chicago

Comparison of evaluation results from fall 2006 and fall 2007

A new evaluation was conducted in Spring 2011: Google vs. Bing vs. Blekko
If you interested in the results, please drop me an email

Fall 2006 evaluation setup: See Section 2 in this paper, which also gives the background of the evaluation.

Fall 2007 evaluation setup: This evaluation was conducted from October 1, 2007 to November 26, 2007 with 27 graduate students in my data mining and text mining class (CS583) at the Department of Computer Science, University of Illinois at Chicago. Since they were all graduate students, the results given below thus only reflect this segment of the general population.

An important difference from the 2006 evaluation is that for 2007 an evaluation system was implemented to hide the identities of the search engines. That is, the student does not know which search engine he/she is using when he/she performs each search. The search engine is randomly selected. The detailed selection process is a bit complicated as we also wanted to keep some task information. For each search task (which may consists of multiple queries on the same task), the system randomly select a search engine. However, some students do not always follow rules. They do not identify tasks and thus keep using the same search engines randomly selected when they log in. This causes some uneven numbers of queries for different search engines, especially for navigational queries as the number of such queries is small. Note that a login was required for each student.

Comparison results: Table 1 shows the comparative results for navigational queries and Table 2 shows the comparative results for informational queries. See the definitions of navigational queries and informational queries, and other relevant information here. Two main observations:

1.      Both Live search and Yahoo! search made significant progresses from 2006 to 2007. Google’s results stay roughly the same. However, Google still has a big lead particularly in navigational queries.

2.      There are significantly fewer navigational queries in 2007 than in 2006. I asked the students why. They said that they did not have a lot of such queries and they also used bookmarks quite often, which I believe are only part of the reason. The login requirement is perhaps a bigger factor because navigational queries are usually more instantaneous and students felt too troublesome to log in to find something quickly, which they agreed.  

Table 1: Results for navigational queries

Navigational query

Google

Yahoo!

MSN (Live)

Fall 2006

Fall 2007

Fall 2006

Fall 2007

Fall 2006

Fall 2007

Not satisfied

15(4%)

7(7.6%)

37(14%)

22(17.9%)

49(19%)

13(13.4%)

Partially satisfied

39(11%)

6(6.5%)

60(23%)

18(14.6%)

70(27%)

16(16.5%)

Completely satisfied

303(85%)

79(86%)

166(63%)

83(67%)

141(54%)

68(70%)

Total (queries)

357

92

263

123

260

97

Table 2. Results for informational queries

Informational query

Google

Yahoo!

MSN (Live)

Fall 2006

Fall 2007

Fall 2006

Fall 2007

Fall 2006

Fall 2007

Not satisfied

137(21%)

82(15%)

103(21%)

124(22%)

110(21%)

142(24%)

Partially satisfied

93(14%)

109(20%)

149(30%)

104(18%)

162(31%)

107(18%)

Completely satisfied

416(65%)

341(64%)

247(49%)

339(60%)

257(48%)

335(57%)

Total (queries)

646

532

499

567

529

584

 

Acknowledgements

I would like to thank the 27 graduate students in my fall 2007 CS583 class for their participation in the evaluation. Liangjie Zhang implemented the evaluation system, which hides the identities of the search engines. Yi Zhang helped set up the system and analyze the evaluation results. Many discussions with Zijian Zheng and Ruihua Song also helped in several ways.