The lexer is the bit that breaks up the string into tokens (words in this case).

		olliesaunders on Aug 1, 2010 \| parent \| context \| favorite \| on: Natural language parsing for the web The lexer is the bit that breaks up the string into tokens (words in this case).