Advanced Solr Filters with Phonetics

Solr is a very popular full text search engine these days. One of the big benefits over text searching in mysql or some other database that is gained with Solr beside performance is the quality of it's fuzzy searching. Most databases support basic substring comparison; in mysql this is achieved with the 'column LIKE %query%' construct. There are also some databses that will handle regular expressions as well, but these tend to be slow and don't necessarily give rise to good search results with out a lot of work. Solr, more specifically Lucene, is built for this kind of thing and is optimized for string based comparisons and indexing. Solr also comes with a number of algorithms for good fuzzy searches out of the box.

If you are using acts_as_solr in your rails app the Solr configuration that is run when you type 'rake solr:start' supports some very basic query filtering. The fragment below is found in the default schema.xml

   1  <fieldtype name="text" class="solr.TextField" positionincrementgap="100">
   2      <analyzer type="index">
   3        <tokenizer class="solr.WhitespaceTokenizerFactory">
   4        </tokenizer><filter words="stopwords.txt" class="solr.StopFilterFactory" ignorecase="true">
   5        </filter><filter class="solr.WordDelimiterFilterFactory" generatenumberparts="1" catenatewords="1" catenatenumbers="1" generatewordparts="1" catenateall="0">
   6        </filter><filter class="solr.LowerCaseFilterFactory">
   7        </filter><filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt">
   8        </filter><filter class="solr.RemoveDuplicatesTokenFilterFactory">
   9      </filter></analyzer>
  10      <analyzer type="query">
  11        <tokenizer class="solr.WhitespaceTokenizerFactory">
  12        </tokenizer><filter class="solr.SynonymFilterFactory" ignorecase="true" expand="true" synonyms="synonyms.txt">
  13        </filter><filter words="stopwords.txt" class="solr.StopFilterFactory" ignorecase="true">
  14        </filter><filter class="solr.WordDelimiterFilterFactory" generatenumberparts="1" catenatewords="0" catenatenumbers="0" generatewordparts="1" catenateall="0">
  15        </filter><filter class="solr.LowerCaseFilterFactory">
  16        </filter><filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt">
  17        </filter><filter class="solr.RemoveDuplicatesTokenFilterFactory">
  18      </filter></analyzer>
  19    </fieldtype>

What this says is that for all text fields that are indexed, and all queries that look at text fields, the text will be split in to words based on white space, add in any synonyms it can find, drop certain "stop words", breaks up the words based on a few other triggers (hyphens, camel case, apostrophes), drops the case of all of the words, trims down some basic conjugations (Porter filter), and then removes any duplicates. The Solr wiki has a full description of these tokenizers and filters.

The real gain from fuzzy searching comes with the tilde (~) operator. This operator tells Solr to base it's relevancy score on a Levenshtein distance algorithm which looks at the number of changes to a string required to arrive at another string.

Levenshtein is great, but we can improve the quality of these search results. One reason for inaccurate search queries is misspellings. Levenshtein attempts to find words that are close, but this ignores how people actually work with words. A better solution would be to look at what kinds of substitutions are required and give certain ones higher scores. This would be based on their use in language; their effective pronunciation.

There are a number of algorithms that support comparing two words based on how similar they are in pronunciation. One of the more common ones is called Soundex. This algorithm produces a hash from a given string. Other strings that have a similar pronunciation are supposed to hash to the same value. There are a few limitations with the method. One major one is that it can't handle wrong first letters. So 'psychology' (P242), 'sychology' (S240), and 'cychology' (C420) will not match at all. There are a number of variations on Soundex as well as a few alternatives.

Solr supports a number of these phonetic filters. We'll be adding support for indexing and querying on one of the Soundex variations: Double Metaphone. Looking at the same schema.xml file as above we can add a single line to both the index and the query analyzer tag content.

   1  <fieldtype name="text" class="solr.TextField" positionincrementgap="100">
   2      <analyzer type="index">
   3        <tokenizer class="solr.WhitespaceTokenizerFactory">
   4        </tokenizer><filter words="stopwords.txt" class="solr.StopFilterFactory" ignorecase="true">
   5        </filter><filter class="solr.WordDelimiterFilterFactory" generatenumberparts="1" catenatewords="1" catenatenumbers="1" generatewordparts="1" catenateall="0">
   6        </filter><filter class="solr.LowerCaseFilterFactory">
   7        </filter><filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt">
   8        <!-- Add new phonetic filter -->
   9        </filter><filter class="solr.PhoneticFilterFactory" inject="true" encoder="DoubleMetaphone">
  10        </filter><filter class="solr.RemoveDuplicatesTokenFilterFactory">
  11      </filter></analyzer>
  12      <analyzer type="query">
  13        <tokenizer class="solr.WhitespaceTokenizerFactory">
  14        </tokenizer><filter class="solr.SynonymFilterFactory" ignorecase="true" expand="true" synonyms="synonyms.txt">
  15        </filter><filter words="stopwords.txt" class="solr.StopFilterFactory" ignorecase="true">
  16        </filter><filter class="solr.WordDelimiterFilterFactory" generatenumberparts="1" catenatewords="0" catenatenumbers="0" generatewordparts="1" catenateall="0">
  17        </filter><filter class="solr.LowerCaseFilterFactory">
  18        </filter><filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt">
  19        <!-- Add new phonetic filter -->
  20        </filter><filter class="solr.PhoneticFilterFactory" inject="true" encoder="DoubleMetaphone">
  21        </filter><filter class="solr.RemoveDuplicatesTokenFilterFactory">
  22      </filter></analyzer>
  23    </fieldtype>

We put this at the bottom of the list to make sure that the filter is applied last. Before this will affect our search queries we'll need to re-index our content. This is required so that the Metaphone hashes can be generated and indexed. Once the index is updated our queries will be processed as normal but also be hashed. When the comparisons are made the hashes of the record and the query will be compared to determine how similar they are from a phonetic standpoint.

The results you will see are usually pretty good, but as with any fuzzy algorithm it will require some tweaking before it will "feel" right. There are also a number of alternative algorithms that my index and query faster, or give better results. See the bottom of the AnalyzersTokenizersTokenFilters Solr wiki page for a full list.

Advanced db_free_solr

If you've implemented the db_free_solr plugin in your application, it's probably safe to assume that performance is pretty important to you. For those of you that are looking to squeeze just a little bit more efficiency out of your Solr implementation, here's another feature you might want to know about.

The out-of-box functionality that db_free_solr provides, much the same as acts_as_solr, is designed to make things as easy as possible for you. To that end, the default behavior is to index and store every field you call out in your models. The truth, though, is that you won't need every field you index to be returned with the results. Likewise, some of the fields you want to have returned will, no doubt, not be necessary to index for searching purposes.

So, how do we go about avoiding unnecessary indexing and storing?

The answer is actually simpler than you might think. Acts_as_solr allows you to specify the datatypes of the fields you tell it to index (if not specified, the index assumes a datatype of text). Like so:

acts_as_solr :fields => [{:field1 => :string}, {:field2 => :integer}, {:field3 => :date}]

acts_as_solr :fields => [{:field1 => :string_io}, {:field2 => :integer_so}, {:field3 => :date}]

acts_as_solr :fields => [:field1, {:field2 => :integer_so}, :field3]

"Fixing" acts_as_solr

For what is probably an overwhelming majority of projects, the search functionality that is offered by the acts_as_solr plugin is more than adequate. It's really a neat little piece of work, powerful and easily implemented. Part of the reason it's so easy to implement, though, is that it makes some concessions for the sake of simplicity. Concessions that, for a small percentage of projects, may not be in the best interests of the application and its performance.

What are these concessions, you ask? Well, let's begin by quickly summarizing the basic functionality.

After you install the plugin in your application, pretty much all that's left is to define the indexed fields in each model. Once you've done that and built the indexes, you can go into script/console and run a command like this:

Model.find_by_solr(“text”)

./script/plugin install http://dbfreesolr.rubyforge.org/svn/tags/1.0/db_free_solr/

Model.rebuild_solr_index

def user_display_name
  return self.user.display_name
end

image_tag(results.docs[0].profile_photo.public_filename(:thumb))

def profile_thumb
  return self.profile_photo.public_filename(:thumb).to_s
end

image_tag(results.docs[0].profile_thumb)

results = Post.find_by_solr(“text”).docs
results.each do |doc|
  doc.inspect
end

USPS Package Tracking

About

USPS Track is a plugin which allows you to track US Postal Service packages easily from your Rails app.

Installation

To get started, install it with the following command:

script/plugin install http://svn.elctech.com/svn/public/plugins/usps_track

Second, you'll need a USPS developer account:. You can register here

Usage

Initialize the class with your USPS username:

usps = ELC::USPS.new(username)

Track a package:

result = usps.track(tracking_number)

It returns an array of hashes with the following structure:

{ :eventstate   => "DE",
  :event        => "NOTICE LEFT",
  :eventzipcode => "19801",
  :eventtime    => "11:07 am",
  :eventdate    => "May 30, 2001",
  :eventcity    => "WILMINGTON" }

It's that simple! Please respond with any suggestions, comments or corrections.

Setting up rmagick on Ubuntu

I wanted to update rmagick to the latest version of the gem, so I ran

sudo gem install rmagick

The version that was going to be installed was 2.1.0, and it needed ImageMagick lib 6.3.x.

I went ahead and downloaded 6.3.8 from ImageMagick site (couldn’t use apt-get this time, latest version on repositories I found was 6.2.4.5). I had a previous version of ImageMagick laying on my system. So I ran:

sudo apt-get remove imagemagick 

tar xvvzf ImageMagick.tar.gz 

cd ImageMagick 

./configure –prefix=/usr 

make 

sudo make install 

sudo gem install rmagick 

sudo gem install rmagick -v 1.15.12 // if you want rmagick v1 installed. 

That’s all!

Source Thread

Spec refactoring

While trolling through old code, I came across the following specification:

describe Build do
  describe "passed?" do
     it "returns true for 1" do
       build = Build.new(:status => 1)
       build.passed?.should be_true
     end
  end
end

The problem with this test is it only tests for success. Realizing that it's equally--if not more--important to test for failure than it is success, I decided to ask the team for opinions on what this test should look like.

After bouncing this around with the team, we decided to refactor it as such:

describe Build
 describe "passed?" do
     before(:each) do
       @build= Build.new
     end

    it "returns false when status has not been set" do
      @build.passed?.should be_false
    end

     it "returns false when status is 0"
       @build.status= 0
       @build.passed?.should be_false
    end

     it "returns true when status is 1"
       @build.status= 1
       @build.passed?.should be_true
    end
 end
end

As you can see, we were able to find 2 failure conditions and only one success.

This is a very simple example but we think it has a broad application to testing in general.

replaceaface.com beta ultra version 0.113 build 18.2

replaceaface.com - putting faces on bodies and stuff

With replaceaface, you can mashup images by mixing and matching faces and bodies in a sweet online flash app. Totally photoshopped, check it out!

Executing Shell Commands in Ruby

When in need of executing shell commands, it's easy enough in ruby to use exec(), system(), the ubiquitous backtick or %x{...}, but what's the difference between these?

irb(main):001:0> `ls`
=> "one\nthree\ntwo\n"
irb(main):001:0> %x{ls}
=> "one\nthree\ntwo\n"

Ok, so the backticks and %x{} are exactly the same. What about system()?

irb(main):003:0> system('ls')
one   three two
=> true

Nice, no '\n'. And now exec():

irb(main):004:0> exec('ls')
one   three two
bash$

Ah! it kicked me out of irb, meaning it kills the current Thread. Good to know!

Return Values

Process::Status Objects

irb(main):001:0> system('ls')
one   three two
=> true
irb(main):002:0> $?
=> #<Process::Status: pid=79599,exited(0)>
irb(main):004:0> 

$? accesses the status of the last system executed command, whether you use the backticks, system() or %{}.

Use '.exitstatus' to get the status:

irb(main):001:0>irb(main):001:0> system('ls')
one   three two
=> true
irb(main):002:0> $? # a magic ruby method
=> #<Process::Status: pid=79606,exited(0)>
irb(main):003:0> $?.exitstatus
=> 0
irb(main):004:0> 

Contrary to what you might think, 0 means success, 1 means failure. If you want to make the $? more readable, require English:

irb(main):001:0> system('ls')
one   three two
=> true
irb(main):002:0> $?
=> #<Process::Status: pid=79615,exited(0)>
irb(main):003:0> require 'English'
=> true
irb(main):004:0> $CHILD_STATUS
=> #<Process::Status: pid=79615,exited(0)>
irb(main):005:0> $CHILD_STATUS.exitstatus
=> 0
irb(main):006:0>

Parsing STDERR and STDOUT can only be done by logging and parsing logs. More to come...

Also posted to fr.ivolo.us

Tutorial - Red5, AS3, FC4, and Shared Fridge Magnets!

Jargon...zZzZz

Red5 is an Open Source Flash Server written in Java that supports: Streaming Audio/Video (FLV and MP3), Recording Client Streams (FLV only), Shared Objects, Live Stream Publishing, Remoting”

Red5 Boostrap

Download Red5

Once installed, navigate the Red5 root directory to get started.

Red5 Structure Rundown

Just the important stuff...

In the root Red5 directory you’ll find red5.jar which holds a pre-compiled jar holding the red5 java classes. You’ll use this later to compile your main application class.

The directory webapps is where you will be creating new Red5 applications. All new applications have a consistent structure which is laid out as a template for you in doc/templates

Red5 New Application Structure Rundown

Creating a new Red5 application is simple. We just create a new folder in webapps/.

We will create an app called elcMagnets so we create a directory for it and now have webapps/elcMagnets. Red5 expects us to also have a directory inside our new application named WEB-INF/ so we create that as well, and now have webapps/elcMagnets/WEB-INF.

There are 4 default configuration files in doc/templates/myapp/WEB-INF that we will modify and use in our app, so we can copy those over to our elcMagnets directory. (log4j.properties, red5-web.properties, red5-web.xml, web.xml)

You can read up about all the properties and handlers specified in these files here.

For now, we only want to customize these files for our current app. We need to change the webapp.contextPath specified in red5-web.properties to reflect our newly created app. Take note of the virtualHosts parameter and remember to deal with it on deployment to a remote host.

webapp.contextPath=/elcMagnets
webapp.virtualHosts=localhost, 127.0.0.1

<bean id=”web.handler” 
	    class=”org.red5.server.webapp.elcMagnets.Application” 
		singleton=”true” />

<!-- <bean id=”myhandler.service” 
	    class=”the.path.to.my.ServiceHandler” 
		singleton=”true” /> -->

Application.java

To keep things organized, we will create the directories lib/, classes/, src/ inside webapps/. Next we will create Application.java inside of src/.

As you recall, Application.java will serve as our web.handler, and therefore needs to extend ApplicationAdapter. (For more info on handlers, see the ‘HOWTO-NewApplications.txt’) In short, it allows you to stack functionality on top of existing events (connect, disconnect, join, leave, start, stop) or create your own.

Our example handler will be trivial, but illustrate calls to the Red5 server.

package org.red5.server.webapp.elcMagnets;

import org.red5.server.adapter.ApplicationAdapter;

public class Application extends ApplicationAdapter {
	public String high5Red5(int how_many)
	{
		String highFives = “”;
		if(how_many==0){ return “I dont like you anyways, bro.”; }
		for(int i=0;i<how_many;i++){ highFives += “*Smack*\n”; }
		return highFives;
	}
}

nc.call("high5Red5",nr,5);
// Or more generally...
netconnection.call("function",responder,arguments.....)

Compiling Application.java

What we want to end up with is a nicely packaged up elcMagnets.jar located in lib/

So from src/ we compile Application.java to the classes/ directory.

javac -classpath ../../../../red5.jar -d ../classes Application.java

MANIFEST.MF

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7
Created-By: 1.5.0_06-b05 (Sun Microsystems Inc.)
Class-Path: ../../../../red5.jar

Compiling elcMagnets.jar

jar -cmf MANIFEST.MF ../lib/elcMagnets.jar ../classes/org/red5/server/webapp/elcMagnets/Application.class

OK! Done with Red5.

You should now be able to start up your Red5 server and have it load up our new elcMagnets application.

Flash App

Next we will create an AS3 flash app to interact with our Red5 application and utilize SharedObjects.

Fridge.as (Document Class)

(I'm just going to post code, comment if you need any explanation.)

//	=======================================
// 	Cary Dunn <cdunn@elctech.com>
//	ELC Technologies http://www.elctech.com
//

package
{
	import flash.events.*;
	import flash.display.MovieClip;
	import flash.text.TextField;
	import flash.errors.IOError;
	import flash.system.Security;
	import caurina.transitions.Tweener;
	import flash.net.*;
	import com.elctech.Magnet;
	
	public class Fridge extends MovieClip
	{
		NetConnection.defaultObjectEncoding = ObjectEncoding.AMF0 ;
		private var nc:NetConnection = null;
		private var alphabet:Array = new Array;
		private const LETTERS:Array	= ["A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z",",","!"]
		
		public function Fridge():void
		{
			nc = new NetConnection();
			nc.addEventListener(NetStatusEvent.NET_STATUS,netStatusHandler);
			var nr:Responder = new Responder(onHighFives,null);;
			nc.connect("rtmp://localhost/elcMagnets");
			nc.call("high5Red5",nr,5);
		}
		
		private function onHighFives(resp:String):void
		{
			trace(resp);
		}
		
		private function netStatusHandler(event:NetStatusEvent):void
		{
			trace(event.info.code);
			if(event.info.code == "NetConnection.Connect.Success")
			{
				createAlphabet();
			}
		}
		
		private function createAlphabet():void
		{
			for(var i:uint = 0; i < LETTERS.length; i++)
			{
				var new_magnet:Magnet = new com.elctech.Magnet(LETTERS[i],nc,i);
				addChild(new_magnet);
				alphabet.push(new_magnet);
			}
		}
	}
}

com.elctech.Magnet.as

//	=======================================
// 	Cary Dunn <cdunn@elctech.com>
//	ELC Technologies http://www.elctech.com
//

package com.elctech
{
	import flash.display.MovieClip;
	import flash.display.DisplayObject;
	import flash.display.DisplayObjectContainer;
	import flash.geom.Rectangle;
  import flash.events.*;
	import flash.display.Stage;
	import flash.text.TextField;
	import flash.net.*;
	import flash.utils.Timer;
	import caurina.transitions.Tweener;
	
	public class Magnet extends MovieClip
	{
		private var _id:uint = 0;
		private var _so = null;
		private var _nc:NetConnection = null;
		private var _l:String = null;
		static internal var dragging:Boolean = false;
		static const MAGNET_COLORS:Array = [0xFF3333, 0x3399FF, 0x339966, 0x993399, 0xFF9933]
		
		static internal const ELASTICITY:Number = .5;
		static internal const FRICTION:Number = .5;
		private var ax:Number = 0;
		private var ay:Number = 0;
    private var vx:Number = 0;
		private var vy:Number = 0;
		
		public function Magnet(l:String, nc:NetConnection, i:uint):void
		{
			this.letter.text = l;
			this.letter.textColor = MAGNET_COLORS[Math.round(Math.random()*(MAGNET_COLORS.length-1))];
			this.x = Math.round(Math.random()*390)+67;
			this.y = Math.round(Math.random()*260)+20;
			trace("New Magnet " + l);
			
			_l = l;
			_nc = nc;
			_id = i;
			
			_so = SharedObject.getRemote("letter_"+_id, _nc.uri, true);
			_so.addEventListener(SyncEvent.SYNC,syncEventHandler);
			_so.connect(_nc);
			
			this.buttonMode = true;
			
			this.addEventListener(MouseEvent.MOUSE_DOWN, function(evt:Event):void {
				dragging = true;
				stage.addEventListener(MouseEvent.MOUSE_UP, stopDragging);
				evt.target.parent.addEventListener(Event.ENTER_FRAME, updatePosition);
			});
			
			this.addEventListener(MouseEvent.MOUSE_UP, stopDragging);
		}
		
		private function stopDragging(evt:Event):void
		{
			this.removeEventListener(Event.ENTER_FRAME, updatePosition);
			stage.removeEventListener(MouseEvent.MOUSE_UP, stopDragging);
			dragging = false;
		}
		
		private function updatePosition(evt:Event):void
		{
      ax = (stage.mouseX-this.x+Math.random()*5)*ELASTICITY;
      ay = (stage.mouseY-this.y)*ELASTICITY;
      vx += ax;
      vy += ay;
      vx *= FRICTION;
      vy *= FRICTION;
      this.x += vx;
      this.y += vy;
      this.scaleX = (100-Math.abs(ax)*2)/100;
      this.scaleY = (100-Math.abs(ay)*2)/100;
      this.rotation = ax*3;

			_so.setProperty("currentPosition", {x: evt.target.x, y: evt.target.y});
		}
		
		private function syncEventHandler(evt:SyncEvent):void
		{
			trace("Sync for letter "+_l);
			if(dragging){ return; }
			
			Tweener.addTween(this, {x:_so.data.currentPosition.x, y:_so.data.currentPosition.y, time:0.5, transition:"easeOutBack"});
		}
	}
}

Installing Red5 on FC4

This section documents my trials and tribulations with installing Red5 on an Amazon EC2 instance running a standard FC4 image. This is a nasty (but working) implementation of this.

Some of the rpms are unnecessary, but some of the yum packages were failing on me. Anyways, this might help someone...

$ yum install gcc
$ rpm -ivh ftp://rpmfind.net/linux/fedora/core/updates/4/i386/libstdc++-devel-4.0.2-8.fc4.i386.rpm
$ rpm -ivh ftp://rpmfind.net/linux/fedora/core/updates/4/i386/gcc-c++-4.0.2-8.fc4.i386.rpm
$ yum install fedora-rpmdevtools
$ fedora-buildrpmtree
$ cd /etc/yum.repos.d/
$ wget http://www.jpackage.org/jpackage.repo
---->>> download jdk 5.0 update .bin from <a href="http://java.sun.com/javase/downloads/index.jsp" target="_blank">sun</a>, scp it up to server (requires accepting of TOS)
---->>> cp jdk-....bin rpmbuild/SOURCES
$ wget http://mirrors.dotsrc.org/jpackage/1.7/generic/non-free/SRPMS/java-1.5.0-sun-1.5.0.14-1jpp.nosrc.rpm
$ rpmbuild --rebuild java-1.5.0-sun-1.5.0.14-1jpp.nosrc.rpm
$ cd ~/rpmbuild/RPMS/i586/
$ (echo config gpgcheck 0; echo localinstall java-1.5.0-sun*.rpm; echo run) > yum-cmd
$ yum shell yum-cmd
$ rm yum-cmd

$ cd ~/
wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.7.0-bin.tar.gz
$ cd /usr/local/
$ tar -zxf ~/apache-ant-1.7.0-bin.tar.gz
$ mv apache-ant-1.7.0 ant

$ vim /etc/profile
PATH=$PATH:$HOME/bin:/usr/local/ant/bin
export PATH

$ source /etc/profile

$ tar -zxvf red5-0.6.3.tar.gz
$ cd red5-0.6.3
$ make
$ make install
$ /usr/lib/red5/red5.sh

JRuby&Red5

Setting up a Flash Media Server on EC2 (CentOS)

Windows Server? Yeah right. Redhat? No thanks. It may not be “supported” by the Flash Media Server, but by installing a few libraries you can get one up and running on CentOS in a few minutes…

yum install compat-libstdc++-296.i386
yum install compat-libstdc++-33.i386
cd /lib
ln -s libcrypto.so.*** libcrypto.so.4
ln -s libssl.so.*** libssl.so.4