Davis

Moving to Unicode 5.1

Filed under: Official Google Blog — Wrote by Lees on Tuesday, May 13th, 2008 @ 10:04 pm

Posted by Mark Davis, Senior
International Software Architect

Google has just begun supporting href="http://www.unicode.org/press/pr-5.1.html" id="yh5d" >Unicode
5.1, less than one month after it was released. It's now
available in search, so people speaking languages such as Malayalam
can now search for words containing the new characters in Unicode
5.1.

Web pages can use a variety of different character encodings, like
ASCII, Latin-1, or Windows 1252, or href="http://www.unicode.org/book/aboutbook.html#Foreword"
id="c42w" >Unicode. Most encodings can only represent a few
languages, but Unicode will handle anything from Chinese to French
to Arabic. We have long used Unicode as the internal format for all
the text we search: any other encoding is first converted to
Unicode for processing. So we regularly update to each new version
of Unicode (and relevant related standards like href="http://unicode.org/cldr/" id="az_w" >CLDR and href="http://www.rfc-editor.org/rfc/bcp/bcp47.txt"
id="da92" >BCP 47) to make sure we are current. Thus Unicode
plays a key role in our href="http://www.google.com/corporate/" >mission.

Uptick in native Unicode webpages

Just last December there was an interesting
milestone on the web. For the first time, we found that Unicode was
the most frequent encoding found on web pages, overtaking both
ASCII and Western European encodings—and by coincidence, within 10
days of one another. What's more impressive than simply
overtaking them is the speed with which this happened; take a look
at the blue line in this graph.

onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"
href="http://bp1.blogger.com/_Ap14FtNN91w/SBzrtHJfLnI/AAAAAAAAA5U/TV7_g2_sWq0/s1600-h/Unicode2.gif" >
alt="" id="BLOGGER_PHOTO_ID_5196287230324190834"
border="0" />

You can see a long-term decline in pages encoded in
ASCII (unaccented letters A through Z). More recently, there's
been a significant drop in the use of encodings covering only
Western European letters (ASCII and a few accented letters like Ä,
Ç, and Ø). We're seeing similar declines in other
language-specific encodings. Unicode, on the other hand, is showing
a sharp increase in usage.

This is based on our indexing of web pages, and thus may vary
somewhat from what other search engines find. However, the trends
are pretty clear, and the continued rise in use of Unicode makes it
even easier to do the processing for the many languages that we
cover.

href="http://feeds.feedburner.com/~f/blogspot/MKuf?a=QxeygH" > border="0" />