Personal
Evaluations of Search Engines:
Google,
Yahoo! and Live (MSN)
Department of Computer
Science
University of Illinois at Chicago
Comparison of evaluation results from fall 2006 and fall 2007
Fall
2006 evaluation setup: See Section 2 in this
paper, which also gives the background of the evaluation.
Fall
2007 evaluation setup: This evaluation was conducted from October 1, 2007
to November 26, 2007 with 27 graduate students in my data mining and text
mining class (CS583) at the Department of Computer Science,
An important difference from the 2006
evaluation is that for 2007 an evaluation system was implemented to hide the
identities of the search engines. That is, the student does not know
which search engine he/she is using when he/she performs each search.
The search engine is randomly selected. The detailed selection
process is a bit
complicated as we also wanted to keep some task information. For
each search task (which may consists of multiple queries on the same task), the system randomly select a search engine. However,
some students do not always follow rules. They do not identify tasks and
thus keep using the same search engines randomly selected when they
log in. This causes some uneven numbers of queries for different search engines, especially for navigational queries as the number of such queries is
small. Note that a login was required for each student.
Comparison
results: Table 1
shows the comparative results for navigational queries and Table 2 shows the comparative
results for informational queries. See the definitions of navigational queries and informational queries, and other
relevant information here.
Two main observations:
1. Both Live search and Yahoo! search
made significant progresses from 2006 to 2007. Google’s results stay
roughly the same. However, Google still has a big lead particularly in navigational
queries.
2. There are significantly fewer navigational
queries in 2007 than in 2006. I asked the students why. They said that they did
not have a lot of such queries and they also used bookmarks quite often, which
I believe are only part of the reason. The login requirement is perhaps a
bigger factor because navigational queries are usually more instantaneous and students
felt too troublesome to log in to find something quickly, which they agreed.
Table
1: Results for navigational queries
Navigational query |
Google |
Yahoo! |
MSN (Live) |
|||
Fall 2006 |
Fall 2007 |
Fall 2006 |
Fall 2007 |
Fall 2006 |
Fall 2007 |
|
Not satisfied |
15(4%) |
7(7.6%) |
37(14%) |
22(17.9%) |
49(19%) |
13(13.4%) |
Partially satisfied |
39(11%) |
6(6.5%) |
60(23%) |
18(14.6%) |
70(27%) |
16(16.5%) |
Completely satisfied |
303(85%) |
79(86%) |
166(63%) |
83(67%) |
141(54%) |
68(70%) |
Total (queries) |
357 |
92 |
263 |
123 |
260 |
97 |
Table 2. Results for informational queries
Informational query |
Google |
Yahoo! |
MSN (Live) |
|||
Fall 2006 |
Fall 2007 |
Fall 2006 |
Fall 2007 |
Fall 2006 |
Fall 2007 |
|
Not satisfied |
137(21%) |
82(15%) |
103(21%) |
124(22%) |
110(21%) |
142(24%) |
Partially satisfied |
93(14%) |
109(20%) |
149(30%) |
104(18%) |
162(31%) |
107(18%) |
Completely satisfied |
416(65%) |
341(64%) |
247(49%) |
339(60%) |
257(48%) |
335(57%) |
Total (queries) |
646 |
532 |
499 |
567 |
529 |
584 |
Acknowledgements
I would like to thank the 27
graduate students in my fall 2007 CS583 class for their participation in the
evaluation. Liangjie Zhang implemented the evaluation
system, which hides the identities of the search engines. Yi Zhang helped set
up the system and analyze the evaluation results. Many discussions with Zijian Zheng
and Ruihua Song also helped in several ways.