Using StringTokenizer in Java

By: Emiley J Emailed: 1743 times Printed: 2431 times    

Latest comments
By: rohit kumar - how this program is work
By: Kirti - Hi..thx for the hadoop in
By: Spijker - I have altered the code a
By: ali mohammed - why we use the java in ne
By: ali mohammed - why we use the java in ne
By: mizhelle - when I exported the data
By: raul - no output as well, i'm ge
By: Rajesh - thanx very much...
By: Suindu De - Suppose we are executing

The processing of text often consists of parsing a formatted input string. Parsing is the division of text into a set of discrete parts, or tokens, which in a certain sequence can convey a semantic meaning. The StringTokenizer class provides the first step in this parsing process, often called the lexer (lexical analyzer) or scanner. StringTokenizer implements the Enumeration interface. Therefore, given an input string, you can enumerate the individual tokens contained in it using StringTokenizer.

To use StringTokenizer, you specify an input string and a string that contains delimiters. Delimiters are characters that separate tokens. Each character in the delimiters string is considered a valid delimiter—for example, ",;:" sets the delimiters to a comma, semicolon, and colon. The default set of delimiters consists of the whitespace characters: space, tab, newline, and carriage return.

The StringTokenizer constructors are shown here:

StringTokenizer(String str)
StringTokenizer(String str, String delimiters)
StringTokenizer(String str, String delimiters, boolean delimAsToken)

In all versions, str is the string that will be tokenized. In the first version, the default delimiters are used. In the second and third versions, delimiters is a string that specifies the delimiters. In the third version, if delimAsToken is true, then the delimiters are also returned as tokens when the string is parsed. Otherwise, the delimiters are not returned. 

Delimiters are not returned as tokens by the first two forms. Once you have created a StringTokenizer object, the nextToken( ) method is used to extract consecutive tokens. The hasMoreTokens( ) method returns true while there are more tokens to be extracted. Since StringTokenizer implements Enumeration, the hasMoreElements( ) and nextElement( ) methods are also implemented, and they act the same as hasMoreTokens( ) and nextToken( ), respectively.

Here is an example that creates a StringTokenizer to parse "key=value" pairs. Consecutive sets of "key=value" pairs are separated by a semicolon.

// Demonstrate StringTokenizer.
import java.util.StringTokenizer;
class STDemo {
static String in = "title=Java-Samples;" +
"author=Emiley J;" +
";" +
public static void main(String args[]) {
StringTokenizer st = new StringTokenizer(in, "=;");
while(st.hasMoreTokens()) {
String key = st.nextToken();
String val = st.nextToken();
System.out.println(key + "\t" + val);

The output from this program is shown here:

title Java-samples
author Emiley J
copyright 2007

Java Home | All Java Tutorials | Latest Java Tutorials

Sponsored Links

If this tutorial doesn't answer your question, or you have a specific question, just ask an expert here. Post your question to get a direct answer.

Bookmark and Share


1. View Comment

very good explanation... thnx...

View Tutorial          By: gobu at 2009-04-05 23:43:19
2. View Comment

thnx, 4 ur explanation, that was usefl
but this input wont give that output

View Tutorial          By: some1 at 2009-04-28 03:26:28
3. View Comment

prog was understandable wat u gave there,but it not gave the particular o/p sir.

View Tutorial          By: sailaja at 2009-07-03 02:53:20
4. View Comment

thank you sir

View Tutorial          By: ramya at 2010-01-11 00:42:05
5. View Comment

awesome example

View Tutorial          By: jagjyot singh at 2010-01-12 20:39:04
6. View Comment

there will be a semi-colon(;) after Emiley J and
to produce the desire output..........thank you........example is good

View Tutorial          By: Manojit at 2010-02-21 01:37:47
7. View Comment

We need an example to tokenize a xml file

View Tutorial          By: Todo at 2010-03-03 03:27:01
8. View Comment

thanks emily, this was fantastic... just saved me a huge headache.

also, my apologies for people calling you "sir"... sometimes dudes are dumb... ;')

View Tutorial          By: philip at 2010-04-10 20:39:08
9. View Comment

It gives strange result when we use == as delimiter.
String myStr = "This==is==correct=but=assignment";
it will consider "=" as delimeter and this is wrong.

Please let me know how to parse with exact delimiters.

View Tutorial          By: Amit at 2010-05-07 05:03:41
10. View Comment

Thank you very much for explaning the things in such a very good manner and with good example.

View Tutorial          By: Sunil Yadav at 2010-05-27 04:04:31
11. View Comment

Simplified explanation with perfect Examples

View Tutorial          By: Sushant Chaudhary at 2010-07-12 00:32:18
12. View Comment

nice example!!

View Tutorial          By: gayathri subramanian at 2010-08-18 09:27:59
13. View Comment

thanks for your explanation, but for desired output following changes is required
"\\t" should be "\t"
; is required in 2 and 3 line of string n

View Tutorial          By: bismillah at 2010-09-02 10:16:09
14. View Comment

the best explaination

View Tutorial          By: kingh at 2010-09-17 04:06:46
15. View Comment

how to use tokenizer "No.1 hello,No.2 world,No.3 NO"
to get
1 hello,
2 world,
3 No

View Tutorial          By: saanu at 2011-03-04 00:41:21
16. View Comment

Create a lexical analyzer of the C- programming language using Java program. The following statements available in C- are as follows:

Variable Declarations:
1. Variable must start with a letter and can be followed by a number.
2. Special characters are invalid variable name.
3. Variable must be declared before the corresponding data type.
Data types:
int char
float double
Sample variable declarations:
int A, B, C1.
Mathematical Operators
1. Addition (+)
2. Subtraction (-)
3. Multiplication (X)
4. Division (/)
Logical Operators
1. Not (!)
2. AND (&)
3. OR (| )
Relational Operators
1. Greater than (>)
2. Less than (<)
3. Equal (==)
4. Less than equal (<=)
5. Greater than equal (>=)
6. Not equal (!>)
Assignment operator (:)
NOTE: All statements are terminated by period!

please help me how to start this program, i can't start this kind of program because i don't have a background in java ..

View Tutorial          By: lucky at 2011-05-06 23:05:22
17. View Comment

Hi.....u have put an extra backslash in the last for \t......
u have put \\'s only \t......
otherwise u'll get \t in d o/p as well!!!!!!!

View Tutorial          By: Samantha at 2011-05-12 12:54:39
18. View Comment

Dear Samantha, Thx for pointing the extra \. I have removed the extra \ now.

View Tutorial          By: Emiley at 2011-06-06 04:50:40
19. View Comment

its a very very good artical

View Tutorial          By: xyz at 2011-06-17 08:12:26
20. View Comment

Nice example to understand what does the string Tokennizer do.
Great job.

View Tutorial          By: vikash from india at 2011-06-23 06:39:03
21. View Comment

Nice example to understand what does the string Tokennizer do.

View Tutorial          By: Spha at 2011-07-14 08:00:03
22. View Comment

You just save ma arse, had a SQLite database, and returns a Cursor which I printed as a string using the StringBuilder, then wanted to get columns from the String as key/value pair, couldn't come up with the logic, but you nailed it here girl. Solved my problem, well done!!!

View Tutorial          By: codeRealm at 2011-11-13 12:58:05
23. View Comment

great solution and brief explanation, love it.

View Tutorial          By: habesha at 2011-12-02 16:14:45
24. View Comment

can u please show me the code for finding the frequency of the tokens ie tokens that are repeating in a file......u need to find the frequency of those tokens and then sort the tokens according to highest frequency.......
awaiting urgently for your reply

View Tutorial          By: pooja at 2012-07-08 16:08:08
25. View Comment

thanx nice explanation...

View Tutorial          By: vignesh at 2012-07-15 20:12:22
26. View Comment

@pooja: this method should work, although i'm sure it isn't the most efficient one. str is your string.

public void doIt(String str) {
Hashtable<String, Integer> hTable = new Hashtable<String, Integer>();
String next = null;
StringTokenizer st = new StringTokenizer(str);
while (st.hasMoreTokens()) {
next = st.nextToken();
if (hTable.containsKey(next)) {
hTable.put(next, hTable.get(next) + 1);
} else {
hTable.put(next, 1);
Collection<Integer> coll = hTable.values();
Object[] integerArray = coll.toArray();
Arrays.sort(integerArray, Collections.reverseOrder());
Set<Object> mySet = new LinkedHashSet<Object>(Arrays
for (Object i : mySet) {
for (Map.Entry<String, Integer> entry : hTable.entrySet()) {
if (i.equals(entry.getValue())) {

View Tutorial          By: Anush at 2012-07-16 08:30:58
27. View Comment

it worked n this case but could not find some more cases...however good explanation ..

View Tutorial          By: abhilash koleti at 2012-09-14 16:21:58
28. View Comment

thanks buddy is's very nice and useful example

View Tutorial          By: mayur at 2012-11-05 05:00:11
29. View Comment

how can we split 12+12 into three tokens like 12,+and 12.or there is any alternate way to split these..plz reply me to my mail

View Tutorial          By: arjun at 2012-11-26 11:07:24
30. View Comment

Thanks. What datatype can we give to a variable Student number lets say for example student # 199300052. reply to my email.

View Tutorial          By: Bafokeng Lebesa at 2012-11-27 12:35:40
31. View Comment

Thanks for Your valuable Explanation.

View Tutorial          By: Srikanth at 2012-12-24 10:49:02
32. View Comment

good explanation with sample example makes me to understand the concept easily ., thanks for your work ..,

View Tutorial          By: at 2013-01-04 09:10:21
33. View Comment

date span progarm sorce code in jsp by using string tokenizer concept for ex:if u have entered one date 26-01-2013 then give the span date :2years 6 months 2 days then print the output from expire date plz send the code its mail id is

View Tutorial          By: prudhvi at 2013-01-27 13:55:09
34. View Comment

i have a program in which we have to find the summation of 2 whole number entered in following format
suppose we have to find the sum of 82 and 235 so the input format should be {8,2},{2,3,5} and the sun of 82 + 235 is 317 so output should be in format {3,1,7}

View Tutorial          By: Navjyot at 2013-01-28 10:38:55
35. View Comment

it's emergency
plz explain delimiter in tokenizer

View Tutorial          By: sajad at 2013-02-03 05:56:44
36. View Comment

what if the delimiter is "(quotation mark)?

View Tutorial          By: hiakoto at 2013-02-18 16:15:20
37. View Comment

Thanks for this explanation. I'm not sure whether this is completely explained in Java API !! The developer should be aware when to use StringTokenizer and String.split() !!

View Tutorial          By: Venkatesh Challa at 2013-03-29 04:35:21
38. View Comment

give some good examples very bad i dont lke dis example worst behaviour

View Tutorial          By: abinaya banu at 2014-07-14 06:04:28
39. View Comment

pls help how to fetch vowels as a first letter of d name using jdbc

View Tutorial          By: anjali at 2014-09-18 20:45:49
40. View Comment

Very good example .!

View Tutorial          By: Mprogrammer at 2014-11-25 12:10:21
41. View Comment

Thanks a lot for that excellent explanation. It has helped me a great deal. @ Emiley

View Tutorial          By: Gregory at 2014-12-30 05:49:21

Your name (required):

Your email(required, will not be shown to the public):

Your sites URL (optional):

Your comments:

More Tutorials by Emiley J
Password must include both numeric and alphabetic characters - Magento
What is Hadoop?
Returning multiple values from a web service
Tomcat and httpd configured in port 8080 and 80
Java Webservices using Netbeans and Tomcat
Java WebService connected to Database
How to Deploy a Java Web Service
Call a webservice in Java
Java WebService - Create your first web service in Java
package javax.jws does not exist
Getting Started with Android
HTML5 Location - getCurrentPosition() in HTML5
HTML5 Canvas - Using Canvas in HTML5
HTML5 - Introduction
HTML5 Video - Handling video in HTML5

More Tutorials in Java
Update contents of a file within a jar file
Tomcat and httpd configured in port 8080 and 80
Java File
Java String
Count number of vowels, consonants and digits in a String in Java
Reverse a number in Java
Student marks calculation program in Java
Handling Fractions in Java
Calculate gross salary in Java
Calculate average sale of the week in Java
Vector in Java - Sample Program
MultiLevel Inheritance sample in Java
Multiple Inheritance sample in Java
Java program using Method Overriding
Java program to check if user input is an even number

More Latest News
Most Viewed Articles (in Java )
How to Send SMS using Java Program (full code sample included)
Stack example in Java - push(), pop(), empty(), search()
What is Java?
Student marks calculation program in Java
XML and Java - Parsing XML using Java Tutorial
Using StringTokenizer in Java
wait(), notify() and notifyAll() in Java - A tutorial
indexOf( ) and lastIndexOf( ) in Java
FileReader and FileWriter example program in Java
Integer: byte, short, int, and long data types in Java
The Basic Structure of a Simple Java program
Method Overriding in Java
The switch Statement example in Java
Count number of vowels, consonants and digits in a String in Java
compareTo( ) in Java
Most Emailed Articles (in Java)
Update contents of a file within a jar file
Java Chat Client Sample Applet
Execute system commands in a Java Program
Get user input in Java
Reading from a file and writing to a file using Java program
ThreadGroup Sample in Java
String Concatenation using Java
Insert, Delete, Search, Print an int Array in Java
Text to Speech conversion program in Java
Major components of FxObjects in JavaFX
How to use and access the inner class in java
History of Java
Importing a Certificate for the CA example using keytool in Java
Using continue Statement - example in Java
instanceof operator example in Java