Programming Tutorials

StreamTokenizer sample program in Java

By: Emiley J in Java Tutorials on 2022-09-15  

StreamTokenizer defines several methods. In this example, we will use only a few. To reset the default set of delimiters, we will employ the resetSyntax() method. The default set of delimiters is finely tuned for tokenizing Java programs and is thus too specialized for this example. We declare that our tokens, or "words," are any consecutive string of visible characters delimited on both sides by whitespace.

import java.io.*;

public class StreamTokenizerDemo {
    public static void main(String[] args) throws IOException {
        String input = "42 true 3.14 \"hello\" end";
        Reader reader = new StringReader(input);
        StreamTokenizer tokenizer = new StreamTokenizer(reader);

        // Set up tokenizer properties
        tokenizer.quoteChar('"');
        tokenizer.ordinaryChar(' ');

        // Read and print tokens
        while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
            switch (tokenizer.ttype) {
                case StreamTokenizer.TT_NUMBER:
                    System.out.println("Number: " + tokenizer.nval);
                    break;
                case StreamTokenizer.TT_WORD:
                    System.out.println("Word: " + tokenizer.sval);
                    break;
                case '"':
                    System.out.println("Quoted string: " + tokenizer.sval);
                    break;
                default:
                    System.out.println("Other character: " + (char) tokenizer.ttype);
                    break;
            }
        }
    }
}

In this example, we create a StreamTokenizer instance to parse the string "42 true 3.14 "hello" end". We then set some properties on the tokenizer, such as using double quotes as the quote character and treating space as an ordinary character.

We then read tokens from the tokenizer using the nextToken() method, which returns a token type constant indicating the type of the next token, such as TT_NUMBER for a numeric value or TT_WORD for a string value. We use a switch statement to handle each token type, printing out the token value or character as appropriate.

Here are some of the important methods provided by StreamTokenizer:

• nextToken(): Returns the type of the next token in the input stream.
• sval: Returns the string value of the current token if it is a word or quoted string.
• nval: Returns the numeric value of the current token if it is a number.
• ttype: Returns the type of the current token as an integer constant.
• quoteChar(char c): Sets the quote character to use for parsing quoted strings.
• ordinaryChar(char c): Treats the specified character as an ordinary character rather than a delimiter.
• eolIsSignificant(boolean flag): Indicates whether end-of-line characters should be treated as separate tokens.