Class Token


  • public class Token
    extends java.lang.Object
    A linked list of tokens. Each token has three fields - a token identifier, which is a byte value that can be looked up in the array returned by SyntaxDocument.getColors() to get a color value, a length value which is the length of the token in the text, and a pointer to the next token in the list.
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static byte COMMENT1
      Comment 1 token id.
      static byte COMMENT2
      Comment 2 token id.
      static byte END
      The token type, that along with a length of 0 marks the end of the token list.
      byte id
      The id of this token.
      static byte ID_COUNT
      The total number of defined token ids.
      static byte INTERNAL_FIRST
      The first id that can be used for internal state in a token marker.
      static byte INTERNAL_LAST
      The last id that can be used for internal state in a token marker.
      static byte INVALID
      Invalid token id.
      static byte KEYWORD1
      Keyword 1 token id.
      static byte KEYWORD2
      Keyword 2 token id.
      static byte KEYWORD3
      Keyword 3 token id.
      static byte LABEL
      Label token id.
      int length
      The length of this token.
      static byte LITERAL1
      Literal 1 token id.
      static byte LITERAL2
      Literal 2 token id.
      Token next
      The next token in the linked list.
      static byte NULL
      Normal text token id.
      static byte OPERATOR
      Operator token id.
    • Constructor Summary

      Constructors 
      Constructor Description
      Token​(int length, byte id)
      Creates a new token.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      java.lang.String toString()
      Returns a string representation of this token.
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
    • Field Detail

      • NULL

        public static final byte NULL
        Normal text token id. This should be used to mark normal text.
        See Also:
        Constant Field Values
      • COMMENT1

        public static final byte COMMENT1
        Comment 1 token id. This can be used to mark a comment.
        See Also:
        Constant Field Values
      • COMMENT2

        public static final byte COMMENT2
        Comment 2 token id. This can be used to mark a comment.
        See Also:
        Constant Field Values
      • LITERAL1

        public static final byte LITERAL1
        Literal 1 token id. This can be used to mark a string literal (eg, C mode uses this to mark "..." literals)
        See Also:
        Constant Field Values
      • LITERAL2

        public static final byte LITERAL2
        Literal 2 token id. This can be used to mark an object literal (eg, Java mode uses this to mark true, false, etc)
        See Also:
        Constant Field Values
      • LABEL

        public static final byte LABEL
        Label token id. This can be used to mark labels (eg, C mode uses this to mark ...: sequences)
        See Also:
        Constant Field Values
      • KEYWORD1

        public static final byte KEYWORD1
        Keyword 1 token id. This can be used to mark a keyword. This should be used for general language constructs.
        See Also:
        Constant Field Values
      • KEYWORD2

        public static final byte KEYWORD2
        Keyword 2 token id. This can be used to mark a keyword. This should be used for preprocessor commands, or variables.
        See Also:
        Constant Field Values
      • KEYWORD3

        public static final byte KEYWORD3
        Keyword 3 token id. This can be used to mark a keyword. This should be used for data types.
        See Also:
        Constant Field Values
      • OPERATOR

        public static final byte OPERATOR
        Operator token id. This can be used to mark an operator. (eg, SQL mode marks +, -, etc with this token type)
        See Also:
        Constant Field Values
      • INVALID

        public static final byte INVALID
        Invalid token id. This can be used to mark invalid or incomplete tokens, so the user can easily spot syntax errors.
        See Also:
        Constant Field Values
      • ID_COUNT

        public static final byte ID_COUNT
        The total number of defined token ids.
        See Also:
        Constant Field Values
      • INTERNAL_FIRST

        public static final byte INTERNAL_FIRST
        The first id that can be used for internal state in a token marker.
        See Also:
        Constant Field Values
      • INTERNAL_LAST

        public static final byte INTERNAL_LAST
        The last id that can be used for internal state in a token marker.
        See Also:
        Constant Field Values
      • END

        public static final byte END
        The token type, that along with a length of 0 marks the end of the token list.
        See Also:
        Constant Field Values
      • length

        public int length
        The length of this token.
      • id

        public byte id
        The id of this token.
      • next

        public Token next
        The next token in the linked list.
    • Constructor Detail

      • Token

        public Token​(int length,
                     byte id)
        Creates a new token.
        Parameters:
        length - The length of the token
        id - The id of the token
    • Method Detail

      • toString

        public java.lang.String toString()
        Returns a string representation of this token.
        Overrides:
        toString in class java.lang.Object