Using StringTokenizer in Java

By: Emiley J Viewed: 157371 times    

The processing of text often consists of parsing a formatted input string. Parsing is the division of text into a set of discrete parts, or tokens, which in a certain sequence can convey a semantic meaning. The StringTokenizer class provides the first step in this parsing process, often called the lexer (lexical analyzer) or scanner. StringTokenizer implements the Enumeration interface. Therefore, given an input string, you can enumerate the individual tokens contained in it using StringTokenizer.

To use StringTokenizer, you specify an input string and a string that contains delimiters. Delimiters are characters that separate tokens. Each character in the delimiters string is considered a valid delimiter—for example, ",;:" sets the delimiters to a comma, semicolon, and colon. The default set of delimiters consists of the whitespace characters: space, tab, newline, and carriage return.

The StringTokenizer constructors are shown here:

StringTokenizer(String str)
StringTokenizer(String str, String delimiters)
StringTokenizer(String str, String delimiters, boolean delimAsToken)

In all versions, str is the string that will be tokenized. In the first version, the default delimiters are used. In the second and third versions, delimiters is a string that specifies the delimiters. In the third version, if delimAsToken is true, then the delimiters are also returned as tokens when the string is parsed. Otherwise, the delimiters are not returned. 

Delimiters are not returned as tokens by the first two forms. Once you have created a StringTokenizer object, the nextToken( ) method is used to extract consecutive tokens. The hasMoreTokens( ) method returns true while there are more tokens to be extracted. Since StringTokenizer implements Enumeration, the hasMoreElements( ) and nextElement( ) methods are also implemented, and they act the same as hasMoreTokens( ) and nextToken( ), respectively.

Here is an example that creates a StringTokenizer to parse "key=value" pairs. Consecutive sets of "key=value" pairs are separated by a semicolon.

// Demonstrate StringTokenizer.
import java.util.StringTokenizer;
class STDemo {
static String in = "title=Java-Samples;" +
"author=Emiley J;" +
"publisher=java-samples.com;" +
"copyright=2007;";
public static void main(String args[]) {
StringTokenizer st = new StringTokenizer(in, "=;");
while(st.hasMoreTokens()) {
String key = st.nextToken();
String val = st.nextToken();
System.out.println(key + "\t" + val);
}
}
}

The output from this program is shown here:

title Java-samples
author Emiley J
publisher java-samples.com
copyright 2007

Most Viewed Articles (in Java )

Latest Articles (in Java)

Comment on this tutorial

Subscribe to Tutorials

Related Tutorials

Archived Comments

1. f**k
View Tutorial          By: stupid head at 2009-03-11 04:37:07

2. very good explanation... thnx...
View Tutorial          By: gobu at 2009-04-05 23:43:19

3. thnx, 4 ur explanation, that was usefl
but

View Tutorial          By: some1 at 2009-04-28 03:26:28

4. prog was understandable wat u gave there,but it no
View Tutorial          By: sailaja at 2009-07-03 02:53:20

5. thank you sir
View Tutorial          By: ramya at 2010-01-11 00:42:05

6. awesome example
View Tutorial          By: jagjyot singh at 2010-01-12 20:39:04

7. cse exalters
View Tutorial          By: jayasimha at 2010-02-20 02:12:52

8. there will be a semi-colon(;) after Emiley J and j
View Tutorial          By: Manojit at 2010-02-21 01:37:47

9. We need an example to tokenize a xml file
View Tutorial          By: Todo at 2010-03-03 03:27:01

10. thanks emily, this was fantastic... just saved me
View Tutorial          By: philip at 2010-04-10 20:39:08

11. It gives strange result when we use == as delimite
View Tutorial          By: Amit at 2010-05-07 05:03:41

12. Thank you very much for explaning the things in su
View Tutorial          By: Sunil Yadav at 2010-05-27 04:04:31

13. @:some1

The input should be

View Tutorial          By: Sam at 2010-06-16 07:18:22

14. it was very help ful
i got it when i really

View Tutorial          By: anand harshan at 2010-07-07 12:18:22

15. Simplified explanation with perfect Examples
View Tutorial          By: Sushant Chaudhary at 2010-07-12 00:32:18

16. nice example!!
View Tutorial          By: gayathri subramanian at 2010-08-18 09:27:59

17. nice example!!
View Tutorial          By: gayathri subramanian at 2010-08-18 09:29:35

18. thanks for your explanation, but for desired outpu
View Tutorial          By: bismillah at 2010-09-02 10:15:35

19. thanks for your explanation, but for desired outpu
View Tutorial          By: bismillah at 2010-09-02 10:16:09

20. the best explaination
View Tutorial          By: kingh at 2010-09-17 04:06:46

21. Wonderful...
It was Really Useful...

View Tutorial          By: Gift Lee at 2010-12-02 23:27:15

22. huahahaha, ya the ";" is forgotten at th
View Tutorial          By: g at 2011-01-07 22:08:26

23. thanx 4 these info......
View Tutorial          By: rajkamal at 2011-02-18 23:39:35

24. how to use tokenizer "No.1 hello,No.2 world,N
View Tutorial          By: saanu at 2011-03-04 00:41:21

25. Create a lexical analyzer of the C- programming la
View Tutorial          By: lucky at 2011-05-06 23:01:43

26. Create a lexical analyzer of the C- programming la
View Tutorial          By: lucky at 2011-05-06 23:05:22

27. Hi.....u have put an extra backslash in the last f
View Tutorial          By: Samantha at 2011-05-12 12:54:39

28. Dear Samantha, Thx for pointing the extra \. I hav
View Tutorial          By: Emiley at 2011-06-06 04:50:40

29. its a very very good artical
View Tutorial          By: xyz at 2011-06-17 08:12:26

30. Nice example to understand what does the string T
View Tutorial          By: vikash from india at 2011-06-23 06:39:03

31. Nice example to understand what does the string To
View Tutorial          By: Spha at 2011-07-14 08:00:03

32. You just save ma arse, had a SQLite database, and
View Tutorial          By: codeRealm at 2011-11-13 12:58:05

33. Very good example.
It's very useful article

View Tutorial          By: elangovan at 2011-11-16 12:33:10

34. great solution and brief explanation, love it.
View Tutorial          By: habesha at 2011-12-02 16:14:45

35. shes a hot chick programmer!
View Tutorial          By: pola at 2012-01-16 02:33:21

36. Awesome explanation....
superb!

View Tutorial          By: Abhishek Patel at 2012-03-27 05:29:32

37. hi....
can u please show me the code for fi

View Tutorial          By: pooja at 2012-07-08 16:08:08

38. thanx nice explanation...
View Tutorial          By: vignesh at 2012-07-15 20:12:22

39. @pooja: this method should work, although i'm sure
View Tutorial          By: Anush at 2012-07-16 08:30:58

40. Nice explanation and a valuable example. Awesome!
View Tutorial          By: kesariJena at 2012-07-20 13:05:35

41. it worked n this case but could not find some more
View Tutorial          By: abhilash koleti at 2012-09-14 16:21:58

42. thanks buddy is's very nice and useful example
View Tutorial          By: mayur at 2012-11-05 05:00:11

43. how can we split 12+12 into three tokens like 12,+
View Tutorial          By: arjun at 2012-11-26 11:07:24

44. Thanks. What datatype can we give to a variable St
View Tutorial          By: Bafokeng Lebesa at 2012-11-27 12:35:40

45. Thanks for Your valuable Explanation.
View Tutorial          By: Srikanth at 2012-12-24 10:49:02

46. good explanation with sample example makes me to u
View Tutorial          By: jagan.java at 2013-01-04 09:10:21

47. date span progarm sorce code in jsp by using strin
View Tutorial          By: prudhvi at 2013-01-27 13:55:09

48. i have a program in which we have to find the summ
View Tutorial          By: Navjyot at 2013-01-28 10:38:55

49. it's emergency
plz explain delimiter in tok

View Tutorial          By: sajad at 2013-02-03 05:56:44

50. what if the delimiter is "(quotation mark)?
View Tutorial          By: hiakoto at 2013-02-18 16:15:20

51. Thanks for this explanation. I'm not sure whether
View Tutorial          By: Venkatesh Challa at 2013-03-29 04:35:21

52. give some good examples very bad i dont lke dis ex
View Tutorial          By: abinaya banu at 2014-07-14 06:04:28

53. pls help how to fetch vowels as a first letter of
View Tutorial          By: anjali at 2014-09-18 20:45:49

54. Very good example .!
Ty

View Tutorial          By: Mprogrammer at 2014-11-25 12:10:21

55. Thanks a lot for that excellent explanation. It ha
View Tutorial          By: Gregory at 2014-12-30 05:49:21

56. Thanks i am looking for that how to use stringToke
View Tutorial          By: waqas at 2015-06-01 18:34:48

57. I see you don't monetize your website, don't waste
View Tutorial          By: 86Julio at 2017-07-26 23:06:03