Creating a Syntax Highlighting, Outlining editor with Eclipse and XText

Error message

  • Notice: Use of undefined constant filter - assumed 'filter' in preg_replace() (line 1 of /home/davemr/mo-seph.com/sites/all/modules/drutex/drutex.module(363) : regexp code).
  • Notice: Use of undefined constant filter - assumed 'filter' in preg_replace() (line 1 of /home/davemr/mo-seph.com/sites/all/modules/drutex/drutex.module(363) : regexp code).
  • Notice: Use of undefined constant filter - assumed 'filter' in preg_replace() (line 1 of /home/davemr/mo-seph.com/sites/all/modules/drutex/drutex.module(363) : regexp code).
  • Notice: Use of undefined constant filter - assumed 'filter' in preg_replace() (line 1 of /home/davemr/mo-seph.com/sites/all/modules/drutex/drutex.module(363) : regexp code).
  • Notice: Use of undefined constant filter - assumed 'filter' in preg_replace() (line 1 of /home/davemr/mo-seph.com/sites/all/modules/drutex/drutex.module(363) : regexp code).
  • Notice: Use of undefined constant filter - assumed 'filter' in preg_replace() (line 1 of /home/davemr/mo-seph.com/sites/all/modules/drutex/drutex.module(363) : regexp code).
  • Notice: Use of undefined constant filter - assumed 'filter' in preg_replace() (line 1 of /home/davemr/mo-seph.com/sites/all/modules/drutex/drutex.module(363) : regexp code).
  • Notice: Undefined index: en in drutex_node_view() (line 81 of /home/davemr/mo-seph.com/sites/all/modules/drutex/drutex.module).

I've been working with LPJ-Guess recently. It's a digital vegetation model, but that's not important to this discussion. What's important is that it has it's own, non-standard config syntax. Editing this can be confusing, without the tools that we take for granted, like syntax highlighting, and outline editors etc.

The latest version Eclipse comes with Xtext, which is designed for easy generation of parsers and editors for DSLs, so this looks like a perfect test case. I set up eclipse Indigo, with all my usual editors etc, and added the Xtext components from the Indigo repository.

Creating a Project[edit | edit source]

IF you follow the Tutorial, then you will create a complete XText setup, ready to go. It creates three projects: yourproject, yourproject.ui and yourproject.tests.

In this case, as I'm working on the LPJ-Guess model, and I'm working at CECS, my project is called org.cecs.gessconfig.

The tutorial sets up a basic grammar for parsing a simple DSL. From this point, I'm adapting it to my particular needs.

Creating the DSL Grammar[edit | edit source]

The first step is to create a grammar for the DSL. In this case, it is slightly tricky, as the language spec is not really known. I won't post a full config file, but it looks a bit like this:

param "param" (str "param_str") ! comment
variable 1  ! comment
var2 0  ! variable with 2 values
var3 "value" ! variable with a string value
 
group "group name" (
 gvar 0
 gvar2 0.8
)
 
pft "pft name" (
 include1
 include2
 pvar1 0
 pvar2 0.3
)

So, we have:

  • comments, starting with an exclamation mark
  • variables, with a name and either a string value or one or more numeric values
  • parameters, with a param name, and either a numeric or string value
  • groups and pfts which define a block, and
    • may include another group or pft
    • have some number of variable definitions.

Based on this, and the XText documentation, I made a DSL grammar like this:

grammar org.cecs.GuessConfig with org.eclipse.xtext.common.Terminals hidden (GUESS_COMMENT,WS)
 
generate guessConfig "<a rel="nofollow" class="external free" href="http://www.cecs.org/GuessConfig%22%0A%0AConfig:&#10">http://www.cecs.org/GuessConfig%22%0A%0AConfig:&#10</a>; p = Preamble
 b = Body
;
 
Preamble: params += ConfigItem+;
Body: groups += EntityItem+;
ConfigItem: Param | Variable;
EntityItem: Group | PFT;
Param: 'param' name=GUESS_STRING val=ParamVal;
ParamVal: ('(str' val=GUESS_STRING ')' ) | ( '(num' val=FLOAT ')' );
Variable: NumVar | StringVar | MultiNumVar;
StringVar: varname=ID val=GUESS_STRING;
NumVar: varname=ID val=FLOAT;
MultiNumVar: varname=ID val=FLOAT val2+=FLOAT+;
Group:'group' name=GUESS_STRING '(' lines+=GroupLine+ ')';
PFT:'pft' name=GUESS_STRING '(' lines+=GroupLine+ ')'; 
GroupLine:Variable | Include;
Include:include=ID;
 
terminal FLOAT: '-'?('0'..'9')+('.'('0'..'9')+)?;
terminal GUESS_COMMENT : '!' !('\n'|'\r')* ('\r'? '\n');
terminal GUESS_STRING : '"' (!('"'))* '"';

This is a slightly tricky process - to check if it works, you have to run the "Generate*Config" task, and then launch a new version of eclipse which uses it. Then you can look for parse errors to see if it is correctly parsing your file.

NOTE: there is an extra level of indirection at the start, to gather all the parameters and groups into separate places - this is just to make the outline easier to read.

At this point, you get a black-and-white editor, with an outline view, which will tell you if anything in a file doesn't match spec (with a red underline, and a semi-comprehensible error message).

Syntax Highlighting[edit | edit source]

So far so easy. Now, I'd like some syntax highlighting. At this point we need to write some Java code, and I had two problems:

  • all the docs on the web are for XText 1.0 or 0.7, and the version with Eclipse is 2.0
  • The code examples don't say how to actually use the code (or at least, it's not obvious).

itemis was a good help, but it was somewhere on the Eclipse forums that I found the answers I needed

Firstly, there are two different highlighting methods: lexical and semantic. Semantic is more complex and slower, but I can't find any examples of Lexical highlighting actually working, so we'll go with the Semantic version. Here, you need two components:

  • a IHighlightingConfiguration, which sets up and registers the styles which are to be used
  • a ISemanticHighlightingCalculator, which calculates which regions of the document to apply styles to

you also need to configure these to be used.

This all happens in the <yourpackage>.ui project, i.e. *not* the project where you write the DSL.

Setup[edit | edit source]

You can start by making skeleton Configurator and Calculator classes. For me, these go in src/org.cecs.ui, in the org.cecs.guessconfig.ui project. They look like this:

public class GuessHighlightingCalculator implements ISemanticHighlightingCalculator
{
 public void provideHighlightingFor( XtextResource resource, IHighlightedPositionAcceptor acceptor ) {}
}
 
public class GuessHighlightingConfiguration implements IHighlightingConfiguration
{
 public void configure(IHighlightingConfigurationAcceptor acceptor) {}
} 

Now these need to be added to the UI framework. To do this, you should have a class called yourprojectUiModule (again in the UI project). For me, this was org.cecs.ui.GuessConfigUiModule: XTExt UI Project

This class should already exist, and I added two methods to it, so it looks as follows:

public class GuessConfigUiModule extends org.cecs.ui.AbstractGuessConfigUiModule {
 public GuessConfigUiModule(AbstractUIPlugin plugin) {
 super(plugin);
 }
 
 public Class<? extends IHighlightingConfiguration> bindIHighlightingConfiguration () {
 return GuessHighlightingConfiguration.class;
 }
 public Class<? extends ISemanticHighlightingCalculator> bindISemanticHighlightingCalculator(){
 return GuessHighlightingCalculator.class;
 }
}

These bind the highlighting Configuration and Calculator respectively.

At this point, you should be able to run your editor, with minimal obvious difference, except that there are probably no Styles to configure in the Syntax highlighting box.

Configuration[edit | edit source]

Now, you need to set up the configurator - in my case GuessHighlingtingConfiguration - to add all the necessary styles. This is just a case of adding them to an Acceptor, with a unique id, a human readable name, and a text style. My version looks like this:

package org.cecs.ui;
 
import org.eclipse.swt.SWT;
import org.eclipse.swt.graphics.RGB;
import org.eclipse.xtext.ui.editor.syntaxcoloring.*;
import org.eclipse.xtext.ui.editor.utils.TextStyle;
import static org.eclipse.swt.SWT.*;
 
public class GuessHighlightingConfiguration implements IHighlightingConfiguration
{
 
 // provide an id string for the highlighting calculator
 public static final String PARAM = "Parameter";
 public static final String PARAM_VAL = "Parameter Value";
 public static final String VARIABLE = "Variable";
 public static final String VARIABLE_VAL = "Variable Val";
 public static final String COMMENT = "Comment";
 public static final String STRING = "String";
 public static final String NUMBER = "Number";
 public static final String GROUP = "Group";
 public static final String PFT = "PFT";
 public static final String GROUP_NAME = "Group Name";
 public static final String[] ALL_STRINGS =
 { PARAM , PARAM_VAL , VARIABLE , VARIABLE_VAL , STRING , NUMBER , GROUP , PFT
 };
 
 // configure the acceptor providing the id, the description string
 // that will appear in the preference page and the initial text style
 public void configure(IHighlightingConfigurationAcceptor acceptor) 
 {
 addType( acceptor, PARAM, 50, 0, 0, NORMAL );
 addType( acceptor, PARAM_VAL, 50, 0, 0, NORMAL );
 addType( acceptor, VARIABLE, 50, 0, 0, NORMAL );
 addType( acceptor, VARIABLE_VAL, 50, 0, 0, NORMAL );
 addType( acceptor, STRING, 50, 0, 0, NORMAL );
 addType( acceptor, NUMBER, 50, 0, 0, NORMAL );
 addType( acceptor, GROUP, 50, 0, 0, NORMAL );
 addType( acceptor, PFT, 50, 0, 0, NORMAL );
 addType( acceptor, GROUP_NAME, 50, 0, 0, NORMAL );
 addType( acceptor, COMMENT, 150, 200, 200, NORMAL );
 }
 
 public void addType( IHighlightingConfigurationAcceptor acceptor, String s, int r, int g, int b, int style )
 {
 TextStyle textStyle = new TextStyle();
 textStyle.setBackgroundColor(new RGB(255, 255, 255));
 textStyle.setColor(new RGB(r, g, b));
 textStyle.setStyle(style);
 acceptor.acceptDefaultHighlighting(s, s, textStyle);
 }
 
}

I'm just returning reasonable defaults, with the ability to choose default text colour and style.

If you now run the editor, you should see all of these show up in the Syntax Coloring Dialogue: Syntax Coloring Dialog

It won't, however, colour your text.

Calculation[edit | edit source]

The final step is to calculate the regions where the text styles should be applied. I found this took a bit of trial and error, mostly because I was too lazy to read the docs, and the examples on the web are for earlier versions.

I started from the premise that you go through the nodes, and decide whether to colour them based on the node content, so it looks a bit like this:

 public void provideHighlightingFor( XtextResource resource, IHighlightedPositionAcceptor acceptor )
 {
 if( resource == null || resource.getParseResult() == null ) return;
 INode root = resource.getParseResult().getRootNode();
 BidiTreeIterator<INode> it = root.getAsTreeIterable().iterator();
 while( it.hasNext() )
 {
 INode node = it.next();
 if( condition ) acceptor.addPosition( node.getOffset(), node.getLength(), STYLE_ID )
 else ...

This is pretty much what I'm still doing, but with a few helper functions to skip over whitespace and to set styles for several nodes at once (i.e. the ones making up the start of a declaration). Some common problems I ran into:

  • differentiating between node and node.getSemanticElement(). The second doesn't necessarily change with the node, as leaves seem to have the parent as their semantic element. I'm sure there's more to this!
  • not wanting to set the style on the first Node I found (which would be the whole block of text) but on the next node, which is the first token within that block.

package org.cecs.ui;
 
import org.cecs.guessConfig.*;
import org.eclipse.xtext.impl.TerminalRuleImpl;
import org.eclipse.xtext.nodemodel.*;
import org.eclipse.xtext.nodemodel.impl.*;
import org.eclipse.xtext.resource.XtextResource;
import org.eclipse.xtext.ui.editor.syntaxcoloring.*;
import static org.cecs.ui.GuessHighlightingConfiguration.*;
 
public class GuessHighlightingCalculator implements ISemanticHighlightingCalculator
{
 
 public void provideHighlightingFor( XtextResource resource, IHighlightedPositionAcceptor acceptor )
 {
 if( resource == null || resource.getParseResult() == null ) return;
 
 INode root = resource.getParseResult().getRootNode();
 BidiTreeIterator<INode> it = root.getAsTreeIterable().iterator();
 while( it.hasNext() )
 {
 INode node = it.next();
 if( node instanceof CompositeNodeWithSemanticElement && node.getSemanticElement() instanceof Group )
 {
 setStyles( acceptor, it, GROUP, GROUP_NAME, GROUP );
 setStyles( acceptor, node.getAsTreeIterable().reverse().iterator(), null, GROUP );
 }
 else if( node instanceof CompositeNodeWithSemanticElement && node.getSemanticElement() instanceof PFT )
 {
 setStyles( acceptor, it, PFT, GROUP_NAME, PFT );
 setStyles( acceptor, node.getAsTreeIterable().reverse().iterator(), null, PFT );
 }
 else if( node.getSemanticElement() instanceof Include )
 {
 setStyles( acceptor, it, GROUP_NAME );
 }
 else if( node.getSemanticElement() instanceof Param && node instanceof CompositeNode )
 {
 setStyles( acceptor, it, PARAM, STRING, null, PARAM_VAL, STRING, PARAM_VAL );
 }
 else if( node.getSemanticElement() instanceof Variable && node instanceof CompositeNode )
 {
 setStyles( acceptor, it, VARIABLE, VARIABLE_VAL );
 } 
 else if( node instanceof HiddenLeafNode && node.getGrammarElement() instanceof TerminalRuleImpl )
 {
 TerminalRuleImpl ge = (TerminalRuleImpl) node.getGrammarElement();
 if( ge.getName().equalsIgnoreCase( "GUESS_COMMENT" ) ) acceptor.addPosition( node.getOffset(), node.getLength(), COMMENT );
 }
 //else
 //System.err.println( "Node: " + node.getClass().getSimpleName() + " " + node.getGrammarElement().getClass().getSimpleName() );
 }
 }
 
 void setStyles( IHighlightedPositionAcceptor acceptor, BidiIterator<INode> it, String...styles )
 {
 for( String s : styles )
 {
 if( ! it.hasNext() ) return;
 INode n = skipWhiteSpace( acceptor, it );
 if( n != null && s != null ) acceptor.addPosition( n.getOffset(), n.getLength(), s );
 }
 }
 
 INode skipWhiteSpace( IHighlightedPositionAcceptor acceptor, BidiIterator<INode> it )
 {
 INode n = null;
 while ( it.hasNext() && ( n = it.next() ).getClass() == HiddenLeafNode.class )
 processHiddenNode( acceptor, (HiddenLeafNode)n );
 return n;
 }
 
 INode skipWhiteSpaceBackwards( IHighlightedPositionAcceptor acceptor, BidiIterator<INode> it )
 {
 INode n = null;
 while ( it.hasPrevious() && ( n = it.previous() ).getClass() == HiddenLeafNode.class )
 processHiddenNode( acceptor, (HiddenLeafNode)n );
 return n;
 }
 
 
 
 void processHiddenNode( IHighlightedPositionAcceptor acceptor, HiddenLeafNode node )
 {
 if( node.getGrammarElement() instanceof TerminalRuleImpl )
 {
 TerminalRuleImpl ge = (TerminalRuleImpl) node.getGrammarElement();
 if( ge.getName().equalsIgnoreCase( "GUESS_COMMENT" ) ) acceptor.addPosition( node.getOffset(), node.getLength(), COMMENT );
 }
 
 }
}

Result[edit | edit source]

After a half day hacking, what I have is a little bit ugly, but it works, and produces nice (garish) text like this:

Finished Syntax Highlighter

And, pretty much for free, we have:

  • outline
  • code folding
  • error highlighting
  • basic code completion

Next, I might look at the variable names, and see if there is a limited set, and they have specific types, so that they can be completed, and a bit more semantics can be checked.

Project Type: