http://lucene.grantingersoll.com/2008/11/08/tao-and-the-art-of-search-yin-yang-and-tf-idf/ Thoughts on Apache Lucene, Mahout, Solr, Tika and Nutch Thu, 20 May 2010 14:31:37 +0000 hourly 1 http://wordpress.org/?v=3.0 http://lucene.grantingersoll.com/2008/11/08/tao-and-the-art-of-search-yin-yang-and-tf-idf/comment-page-1/#comment-7350 Google-gate and search engines « Techapilla Mon, 07 Dec 2009 10:29:07 +0000 http://lucene.grantingersoll.com/2008/11/08/tao-and-the-art-of-search-yin-yang-and-tf-idf/#comment-7350 [...] This balancing and counter-balancing involved in search algorithms has been called the “yin and yang of search” [...] [...] This balancing and counter-balancing involved in search algorithms has been called the “yin and yang of search” [...]

]]> http://lucene.grantingersoll.com/2008/11/08/tao-and-the-art-of-search-yin-yang-and-tf-idf/comment-page-1/#comment-6169 Enterprise Search in 2009 Wed, 12 Nov 2008 09:36:30 +0000 http://lucene.grantingersoll.com/2008/11/08/tao-and-the-art-of-search-yin-yang-and-tf-idf/#comment-6169 [...] * Tao and the Art of Search: Yin Yang and TF-IDF [...] [...] * Tao and the Art of Search: Yin Yang and TF-IDF [...]

]]> http://lucene.grantingersoll.com/2008/11/08/tao-and-the-art-of-search-yin-yang-and-tf-idf/comment-page-1/#comment-6166 Daniel Tunkelang Sat, 08 Nov 2008 15:51:09 +0000 http://lucene.grantingersoll.com/2008/11/08/tao-and-the-art-of-search-yin-yang-and-tf-idf/#comment-6166 Interesting, I might put that out there the next time I explain tf*idf. But, in my experience, people quickly grasp the basic information theory idea that the significance of a signal is a combination of its strength and its distinctiveness. The nice thing about explaining tf*idf this way is that it helps you further explain where it breaks down, e.g., that term frequency is a very crude proxy for signal strength and that inverse document frequency conflates true distinctiveness with noise. Interesting, I might put that out there the next time I explain tf*idf. But, in my experience, people quickly grasp the basic information theory idea that the significance of a signal is a combination of its strength and its distinctiveness. The nice thing about explaining tf*idf this way is that it helps you further explain where it breaks down, e.g., that term frequency is a very crude proxy for signal strength and that inverse document frequency conflates true distinctiveness with noise.

]]>