Token package:parsec

A helper module to parse lexical elements (tokens). See makeTokenParser for a description of how to use it.

Parsec compatibility module

The parser token showTok posFromTok testTok accepts a token t with result x when the function testTok t returns Just x. The source position of the t should be returned by posFromTok t and the token can be shown using showTok t. This combinator is expressed in terms of tokenPrim. It is used to accept user defined token streams. For example, suppose that we have a stream of basic tokens tupled with source positions. We can then define a parser that accepts single tokens as:

mytoken x
= token showTok posFromTok testTok
where
showTok (pos,t)     = show t
posFromTok (pos,t)  = pos
testTok (pos,t)     = if x == t then Just t else Nothing

The parser tokenPrim showTok nextPos testTok accepts a token t with result x when the function testTok t returns Just x. The token can be shown using showTok t. The position of the next token should be returned when nextPos is called with the current source position pos, the current token t and the rest of the tokens toks, nextPos pos t toks. This is the most primitive combinator for accepting tokens. For example, the char parser could be implemented as:

char c
= tokenPrim showChar nextPos testChar
where
showChar x        = "'" ++ x ++ "'"
testChar x        = if x == c then Just x else Nothing
nextPos pos x xs  = updatePosChar pos x

Like tokens, but doesn't consume matching prefix.

The parser anyToken accepts any kind of token. It is for example used to implement eof. Returns the accepted token.

The type of the record that holds lexical parsers that work on s streams with state u over a monad m.

The expression makeTokenParser language creates a GenTokenParser record that contains lexical parsers that are defined using the definitions in the language record. The use of this function is quite stylized - one imports the appropriate language definition and selects the lexical parsers that are needed from the resulting GenTokenParser.

module Main where

import Text.Parsec
import qualified Text.Parsec.Token as P
import Text.Parsec.Language (haskellDef)

-- The parser
...

expr  =   parens expr
<|> identifier
<|> ...


-- The lexer
lexer       = P.makeTokenParser haskellDef

parens      = P.parens lexer
braces      = P.braces lexer
identifier  = P.identifier lexer
reserved    = P.reserved lexer
...