www.digitalmars.com [Home] [Search] [D]
Last update Jan 24, 2004

Lexical

In D, the lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text up into tokens. The lexical grammar describes what those tokens are. The D lexical grammar is designed to be suitable for high speed scanning, it has a minimum of special case rules, there is only one phase of translation, and to make it easy to write a correct scanner for. The tokens are readily recognizable by those familiar with C and C++.

Phases of Compilation

The process of compiling is divided into multiple phases. Each phase has no dependence on subsequent phases. For example, the scanner is not perturbed by the semantic analyzer. This separation of the passes makes language tools like syntax directed editors relatively easy to produce.
  1. source character set
    The source file is checked to see what character set it is, and the appropriate scanner is loaded. ASCII and UTF formats are accepted.
  2. lexical analysis
    The source file is divided up into a sequence of tokens. Special tokens are processed and removed.
  3. syntax analysis
    The sequence of tokens is parsed to form syntax trees.
  4. semantic analysis
    The syntax trees are traversed to declare variables, load symbol tables, assign types, and in general determine the meaning of the program.
  5. optimization
  6. code generation

Source Text

D source text can be in one of the following formats: Note that UTF-8 is a superset of traditional 7-bit ASCII. The source text is assumed to be in UTF-8, unless one of the following BOMs (Byte Order Marks) is present at the beginning of the source text:

Format BOM
UTF-8 EF BB BF
UTF-16BE FE FF
UTF-16LE FF FE
UTF-32BE 00 00 FE FF
UTF-32LE FF FE 00 00
UTF-8 none of the above

There are no digraphs or trigraphs in D. The source text is split into tokens using the maximal munch technique, i.e., the lexical analyzer tries to make the longest token it can. For example >> is a right shift token, not two greater than tokens.

End of File

	EndOfFile:
		physical end of the file
		\u0000
		\u001A
	
The source text is terminated by whichever comes first.

End of Line

	EndOfLine:
		\u000D
		\u000A
		\u000D \u000A
		EndOfFile
	
There is no backslash line splicing, nor are there any limits on the length of a line.

White Space

	WhiteSpace:
		Space
		Space WhiteSpace

	Space:
		\u0020
		\u0009
		\u000B
		\u000C
		EndOfLine
		Comment
	
White space is defined as a sequence of one or more of spaces, tabs, vertical tabs, form feeds, end of lines, or comments.

Comments

	Comment:
		/* Characters */
		// Characters EndOfLine
		/+ Characters +/
	
D has three kinds of comments:
  1. Block comments can span multiple lines, but do not nest.
  2. Line comments terminate at the end of the line.
  3. Nesting comments can span multiple lines and can nest.
Comment processing conceptually happens before tokenization. This means that embedded strings and comments do not prevent recognition of comment openings and closings:
	a = /+ // +/ 1;		// parses as if 'a = 1;'
	a = /+ "+/" +/ 1";	// parses as if 'a = " +/1";'
	a = /+ /* +/ */ 3;	// parses as if 'a = */ 3;'
	
Comments cannot be used as token concatenators, for example, abc/**/def is two tokens, abc and def, not one abcdef token.

Identifiers

	Identifier:
		IdentiferStart
		IdentiferStart IdentifierChars

	IdentifierChars:
		IdentiferChar
		IdentiferChar IdentifierChars

	IdentifierStart:
		_
		Letter
		UniversalAlpha

	IdentifierChar:
		IdentiferStart
		Digit
	
Identifiers start with a letter, _, or unicode alpha, and are followed by any number of letters, _, digits, or universal alphas. Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D. (This is the C99 Standard.) Identifiers can be arbitrarily long, and are case sensitive. Identifiers starting with __ (two underscores) are reserved.

String Literals

	StringLiteral:
		WysiwygString
		AlternateWysiwygString
		DoubleQuotedString
		EscapeSequence
		HexString

	WysiwygString:
		r" WysiwygCharacters "

	AlternateWysiwygString:
		` WysiwygCharacters `

	WysiwygCharacter:
		Character
		EndOfLine

	DoubleQuotedString:
		" DoubleQuotedCharacters "

	DoubleQuotedCharacter:
		Character
		EscapeSequence
		EndOfLine

	EscapeSequence:
		\'
		\"
		\?
		\\
		\a
		\b
		\f
		\n
		\r
		\t
		\v
		\ EndOfFile
		\x HexDigit HexDigit
		\ OctalDigit
		\ OctalDigit OctalDigit
		\ OctalDigit OctalDigit OctalDigit
		\u HexDigit HexDigit HexDigit HexDigit
		\U HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit

	HexString:
		x" HexStringChars "

	HexStringChar
		HexDigit
		WhiteSpace
		EndOfLine
	
A string literal is either a double quoted string, a wysiwyg quoted string, an escape sequence, or a hex string.

Wysiwyg quoted strings are enclosed by r" and ". All characters between the r" and " are part of the string except for EndOfLine which is regarded as a single \n character. There are no escape sequences inside r" ":

	r"hello"
	r"c:\root\foo.exe"
	r"ab\n"			string is 4 characters, 'a', 'b', '\', 'n'
	
An alternate form of wysiwyg strings are enclosed by backquotes, the ` character. The ` character is not available on some keyboards and the font rendering of it is sometimes indistinguishable from the regular ' character. Since, however, the ` is rarely used, it is useful to delineate strings with " in them.
	`hello`
	`c:\root\foo.exe`
	`ab\n`			string is 4 characters, 'a', 'b', '\', 'n'
	
Double quoted strings are enclosed by "". Escape sequences can be embedded into them with the typical \ notation. EndOfLine is regarded as a single \n character.
	"hello"
	"c:\\root\\foo.exe"
	"ab\n"			string is 3 characters, 'a', 'b', and a linefeed
	"ab
	"			string is 3 characters, 'a', 'b', and a linefeed
	
Escape strings start with a \ and form an escape character sequence. Adjacent escape strings are concatenated:
	\n			the linefeed character
	\t			the tab character
	\"			the double quote character
	\012			octal
	\x1A			hex
	\u1234			wchar character
	\U00101234		dchar character
	\r\n			carriage return, line feed
	
Escape sequences not listed above are errors.

Hex strings allow string literals to be created using hex data:

	x"0A"			same as "\x0A
	x"00 FBCD 32FD 0A"	same as "\x00\xFB\xCD\x32\xFD\x0A"
	
Whitespace and newlines are ignored, so the hex data can be easily formatted. The number of hex characters must be a multiple of 2.

Adjacent strings are concatenated with the ~ operator, or by simple juxtaposition:

	"hello " ~ "world" ~ \n	// forms the string 'h','e','l','l','o',' ','w','o','r','l','d',linefeed
	
The following are all equivalent:
	"ab" "c"
	r"ab" r"c"
	r"a" "bc"
	"a" ~ "b" ~ "c"
	\x61"bc"
	

Character Literals

	CharacterLiteral:
		' SingleQuotedCharacter '

	SingleQuotedCharacter
		Character
		EscapeSequence
	
Character literals are a single character or escape sequence enclosed by single quotes, ' '.

Integer Literals

	IntegerLiteral:
		Integer
		Integer IntegerSuffix

	Integer:
		Decimal
		Binary
		Octal
		Hexadecimal
		Integer _

	IntegerSuffix:
		l
		L
		u
		U
		lu
		Lu
		lU
		LU
		ul
		uL
		Ul
		UL

	Decimal:
		0
		NonZeroDigit
		NonZeroDigit Decimal
		NonZeroDigit _ Decimal

	Binary:
		0b BinaryDigits
		0B BinaryDigits

	Octal:
		0 OctalDigits

	Hexadecimal:
		0x HexDigits
		0X HexDigits
	
Integers can be specified in decimal, binary, octal, or hexadecimal.

Decimal integers are a sequence of decimal digits.

Binary integers are a sequence of binary digits preceded by a '0b'.

Octal integers are a sequence of octal digits preceded by a '0'.

Hexadecimal integers are a sequence of hexadecimal digits preceded by a '0x' or followed by an 'h'.

Integers can have embedded '_' characters, which are ignored. The embedded '_' are useful for formatting long literals, such as using them as a thousands separator:

	123_456		// 123456
	1_2_3_4_5_6_	// 123456
	
Integers can be immediately followed by one 'l' or one 'u' or both.

The type of the integer is resolved as follows:

  1. If it is decimal it is the last representable of ulong, long, or int.
  2. If it is not decimal, it is the last representable of ulong, long, uint, or int.
  3. If it has the 'u' suffix, it is the last representable of ulong or uint.
  4. If it has the 'l' suffix, it is the last representable of ulong or long.
  5. If it has the 'u' and 'l' suffixes, it is ulong.

Floating Literals

	FloatLiteral:
		Float
		Float FloatSuffix
		Float ImaginarySuffix
		Float FloatSuffix ImaginarySuffix

	Float:
		DecimalFloat
		HexFloat
		Float _

	FloatSuffix:
		f
		F
		l
		L

	ImaginarySuffix:
		i
		I
	
Floats can be in decimal or hexadecimal format, as in standard C.

Hexadecimal floats are preceded with a 0x and the exponent is a p or P followed by a power of 2.

Floating literalss can have embedded '_' characters, which are ignored. The embedded '_' are useful for formatting long literals to make them more readable, such as using them as a thousands separator:

	123_456.567_8		// 123456.5678
	1_2_3_4_5_6_._5_6_7_8	// 123456.5678
	1_2_3_4_5_6_._5e-6_	// 123456.5e-6
	
Floats can be followed by one f, F, l or L suffix. The f or F suffix means it is a float, and l or L means it is an extended.

If a floating literal is followed by i or I, then it is an ireal (imaginary) type.

Examples:

	0x1.FFFFFFFFFFFFFp1023		// double.max
	0x1p-52				// double.epsilon
	1.175494351e-38F		// float.min
	6.3i				// idouble 6.3
	6.3fi				// ifloat 6.3
	6.3LI				// ireal 6.3
	
It is an error if the literal exceeds the range of the type. It is not an error if the literal is rounded to fit into the significant digits of the type.

Complex literals are not tokens, but are assembled from real and imaginary expressions in the semantic analysis:

	4.5 + 6.2i		// complex number
	

Keywords

Keywords are reserved identifiers.
	Keyword:
		abstract
		alias
		align
		asm
		assert
		auto

		bit
		body
		break
		byte

		case
		cast
		catch
		cent
		char
		class
		cfloat
		cdouble
		creal
		const
		continue

		dchar
		debug
		default
		delegate
		delete
		deprecated
		do
		double

		else
		enum
		export
		extern

		false
		final
		finally
		float
		for
		foreach
		function

		goto

		idouble
		if
		ifloat
		import
		in
		inout
		int
		interface
		invariant
		ireal
		is

		long

		module

		new
		null

		out
		override

		pragma
		private
		protected
		public

		real
		return

		short
		static
		struct
		super
		switch
		synchronized

		template
		this
		throw
		true
		try
		typedef
		typeof

		ubyte
		ucent
		uint
		ulong
		union
		unittest
		ushort

		version
		void
		volatile

		wchar
		while
		with
	

Tokens

	Token:
		Identifier
		StringLiteral
		IntegerLiteral
		FloatLiteral
		Keyword
		/
		/=
		.
		..
		...
		&
		&=
		&&
		|
		|=
		||
		-
		-=
		--
		+
		+=
		++
		<
		<=
		<<
		<<=
		<>
		<>=
		>
		>=
		>>=
		>>>=
		>>
		>>>
		!
		!=
		!==
		!<>
		!<>=
		!<
		!<=
		!>
		!>=
		(
		)
		[
		]
		{
		}
		?
		,
		;
		:
		$
		=
		==
		===
		*
		*=
		%
		%=
		^
		^=
		~
		~=
	

Special Token Sequences

Special token sequences are processed by the lexical analyzer, may appear between any other tokens, and do not affect the syntax parsing.

There is currently only one special token sequence, #line.

	SpecialTokenSequence
		# line Integer EndOfLine
		# line Integer Filespec EndOfLine

	Filespec
		" Characters "
	
This sets the source line number to Integer, and optionally the source file name to Filespec, beginning with the next line of source text. The source file and line number is used for printing error messages and for mapping generated code back to the source for the symbolic debugging output.

For example:

	int #line 6 "foo\bar"
	x;			// this is now line 6 of file foo\bar
	
Note that the backslash character is not treated specially inside Filespec strings.
Copyright (c) 1999-2004 by Digital Mars, All Rights Reserved