Introduction

Imagine you have a sentence like "Java,Python,JavaScript,C++" and you want to split it into individual programming languages. The StringTokenizer class in Java is like a smart text cutter that breaks strings into smaller pieces (tokens) based on delimiters you specify. It's one of the original ways to parse strings in Java, though newer methods are often preferred today.

StringTokenizer is like having a pair of scissors that can cut a string wherever it finds certain characters (like commas, spaces, or custom delimiters), giving you the individual pieces one by one.

---

What is StringTokenizer?

StringTokenizer is a legacy class in Java that breaks a string into tokens based on specified delimiters. It's part of the java.util package and provides a simple way to tokenize strings, though it's largely superseded by the String.split() method and regular expressions in modern Java.

Key Characteristics:

• ✅ Legacy class: Available since Java 1.0
• ✅ Simple tokenization: Breaks strings into tokens
• ✅ Custom delimiters: Can specify any delimiter characters
• ✅ Stateful: Remembers position in string
• ✅ Lightweight: Less overhead than regex-based methods

---

StringTokenizer Constructors

The class has three constructors:

1. StringTokenizer(String str) - Uses default delimiters (space, tab, newline, etc.)
2. StringTokenizer(String str, String delim) - Uses specified delimiters
3. StringTokenizer(String str, String delim, boolean returnDelims) - Can return delimiters as tokens

---

Code Explanation with Examples

Example 1: Basic StringTokenizer with Default Delimiters

import java.util.StringTokenizer;
public class BasicStringTokenizer {
public static void main(String[] args) {
// Example 1: Default delimiters (whitespace)
String text1 = "Java is a programming language";
StringTokenizer tokenizer1 = new StringTokenizer(text1);
System.out.println("=== DEFAULT DELIMITERS ===");
System.out.println("Original text: '" + text1 + "'");
System.out.println("Number of tokens: " + tokenizer1.countTokens());
// Iterate through tokens
while (tokenizer1.hasMoreTokens()) {
String token = tokenizer1.nextToken();
System.out.println("Token: '" + token + "'");
}
// Example 2: Different text with various whitespace
String text2 = "  Multiple   spaces\tand\ttabs\nand\nnewlines  ";
StringTokenizer tokenizer2 = new StringTokenizer(text2);
System.out.println("\n=== VARIOUS WHITESPACE ===");
System.out.println("Original text: '" + text2 + "'");
System.out.println("Number of tokens: " + tokenizer2.countTokens());
int counter = 1;
while (tokenizer2.hasMoreTokens()) {
System.out.println("Token " + counter + ": '" + tokenizer2.nextToken() + "'");
counter++;
}
}
}

Output:

=== DEFAULT DELIMITERS ===
Original text: 'Java is a programming language'
Number of tokens: 5
Token: 'Java'
Token: 'is'
Token: 'a'
Token: 'programming'
Token: 'language'
=== VARIOUS WHITESPACE ===
Original text: '  Multiple   spaces    and tabs
and
newlines  '
Number of tokens: 6
Token 1: 'Multiple'
Token 2: 'spaces'
Token 3: 'and'
Token 4: 'tabs'
Token 5: 'and'
Token 6: 'newlines'

Example 2: Custom Delimiters

import java.util.StringTokenizer;
public class CustomDelimiters {
public static void main(String[] args) {
// Example 1: Comma delimiter
String csvData = "John,25,New York,Engineer";
StringTokenizer tokenizer1 = new StringTokenizer(csvData, ",");
System.out.println("=== COMMA DELIMITER ===");
System.out.println("CSV Data: " + csvData);
System.out.println("Number of tokens: " + tokenizer1.countTokens());
while (tokenizer1.hasMoreTokens()) {
System.out.println("Field: '" + tokenizer1.nextToken() + "'");
}
// Example 2: Multiple delimiters
String complexText = "Java;Python,JavaScript/C++|Ruby";
StringTokenizer tokenizer2 = new StringTokenizer(complexText, ";,./|");
System.out.println("\n=== MULTIPLE DELIMITERS ===");
System.out.println("Text: " + complexText);
System.out.println("Number of tokens: " + tokenizer2.countTokens());
while (tokenizer2.hasMoreTokens()) {
System.out.println("Language: '" + tokenizer2.nextToken() + "'");
}
// Example 3: Different delimiter for different data
String logEntry = "2024-01-15 10:30:45 [ERROR] Database connection failed";
StringTokenizer tokenizer3 = new StringTokenizer(logEntry, " []");
System.out.println("\n=== LOG PARSING ===");
System.out.println("Log entry: " + logEntry);
System.out.println("Number of tokens: " + tokenizer3.countTokens());
while (tokenizer3.hasMoreTokens()) {
System.out.println("Part: '" + tokenizer3.nextToken() + "'");
}
// Example 4: Email parsing
String email = "[email protected]";
StringTokenizer tokenizer4 = new StringTokenizer(email, "@.");
System.out.println("\n=== EMAIL PARSING ===");
System.out.println("Email: " + email);
System.out.println("Number of tokens: " + tokenizer4.countTokens());
while (tokenizer4.hasMoreTokens()) {
System.out.println("Email part: '" + tokenizer4.nextToken() + "'");
}
}
}

Output:

=== COMMA DELIMITER ===
CSV Data: John,25,New York,Engineer
Number of tokens: 4
Field: 'John'
Field: '25'
Field: 'New York'
Field: 'Engineer'
=== MULTIPLE DELIMITERS ===
Text: Java;Python,JavaScript/C++|Ruby
Number of tokens: 5
Language: 'Java'
Language: 'Python'
Language: 'JavaScript'
Language: 'C++'
Language: 'Ruby'
=== LOG PARSING ===
Log entry: 2024-01-15 10:30:45 [ERROR] Database connection failed
Number of tokens: 5
Part: '2024-01-15'
Part: '10:30:45'
Part: 'ERROR'
Part: 'Database'
Part: 'connection'
Part: 'failed'
=== EMAIL PARSING ===
Email: [email protected]
Number of tokens: 4
Email part: 'user'
Email part: 'name'
Email part: 'example'
Email part: 'com'

Example 3: Returning Delimiters as Tokens

import java.util.StringTokenizer;
public class ReturnDelimiters {
public static void main(String[] args) {
// Example 1: Return delimiters as tokens
String expression = "2 + 3 * (4 - 1)";
StringTokenizer tokenizer1 = new StringTokenizer(expression, " +-*/()", true);
System.out.println("=== WITH DELIMITERS ===");
System.out.println("Expression: " + expression);
System.out.println("Number of tokens: " + tokenizer1.countTokens());
while (tokenizer1.hasMoreTokens()) {
String token = tokenizer1.nextToken();
System.out.println("Token: '" + token + "'");
}
// Example 2: Mathematical expression parsing
String mathExpr = "a*b + c/d - e%f";
StringTokenizer tokenizer2 = new StringTokenizer(mathExpr, " +-*/%", true);
System.out.println("\n=== MATH EXPRESSION ===");
System.out.println("Expression: " + mathExpr);
while (tokenizer2.hasMoreTokens()) {
String token = tokenizer2.nextToken().trim();
if (!token.isEmpty()) {
System.out.println("Element: '" + token + "'");
}
}
// Example 3: SQL query parsing (simplified)
String sqlQuery = "SELECT * FROM users WHERE age > 25 AND city = 'New York'";
StringTokenizer tokenizer3 = new StringTokenizer(sqlQuery, " ,=<>'*", true);
System.out.println("\n=== SQL QUERY PARSING ===");
System.out.println("Query: " + sqlQuery);
while (tokenizer3.hasMoreTokens()) {
String token = tokenizer3.nextToken();
if (!token.equals(" ")) {  // Skip spaces for cleaner output
System.out.println("SQL part: '" + token + "'");
}
}
}
}

Output:

=== WITH DELIMITERS ===
Expression: 2 + 3 * (4 - 1)
Number of tokens: 13
Token: '2'
Token: ' '
Token: '+'
Token: ' '
Token: '3'
Token: ' '
Token: '*'
Token: ' '
Token: '('
Token: '4'
Token: ' '
Token: '-'
Token: '1'
Token: ')'
=== MATH EXPRESSION ===
Expression: a*b + c/d - e%f
Element: 'a'
Element: '*'
Element: 'b'
Element: '+'
Element: 'c'
Element: '/'
Element: 'd'
Element: '-'
Element: 'e'
Element: '%'
Element: 'f'
=== SQL QUERY PARSING ===
Query: SELECT * FROM users WHERE age > 25 AND city = 'New York'
SQL part: 'SELECT'
SQL part: '*'
SQL part: 'FROM'
SQL part: 'users'
SQL part: 'WHERE'
SQL part: 'age'
SQL part: '>'
SQL part: '25'
SQL part: 'AND'
SQL part: 'city'
SQL part: '='
SQL part: "'"
SQL part: 'New'
SQL part: 'York'
SQL part: "'"

Example 4: Real-World Practical Examples

import java.util.StringTokenizer;
public class RealWorldExamples {
public static void main(String[] args) {
// 1. CSV File Processing
System.out.println("=== CSV PROCESSING ===");
String csvLine = "John Doe,30,New York,Engineer,75000";
StringTokenizer csvTokenizer = new StringTokenizer(csvLine, ",");
String[] fields = new String[csvTokenizer.countTokens()];
int index = 0;
while (csvTokenizer.hasMoreTokens()) {
fields[index++] = csvTokenizer.nextToken().trim();
}
System.out.println("Name: " + fields[0]);
System.out.println("Age: " + fields[1]);
System.out.println("City: " + fields[2]);
System.out.println("Job: " + fields[3]);
System.out.println("Salary: $" + fields[4]);
// 2. Configuration File Parsing
System.out.println("\n=== CONFIGURATION PARSING ===");
String configLine = "database.host=localhost;database.port=3306;database.name=mydb";
StringTokenizer configTokenizer = new StringTokenizer(configLine, ";");
while (configTokenizer.hasMoreTokens()) {
String keyValue = configTokenizer.nextToken();
StringTokenizer kvTokenizer = new StringTokenizer(keyValue, "=");
if (kvTokenizer.countTokens() == 2) {
String key = kvTokenizer.nextToken();
String value = kvTokenizer.nextToken();
System.out.println("Config: " + key + " = " + value);
}
}
// 3. URL Parsing
System.out.println("\n=== URL PARSING ===");
String url = "https://www.example.com:8080/path/to/resource?param1=value1&param2=value2";
StringTokenizer urlTokenizer = new StringTokenizer(url, ":/?=&");
System.out.println("URL parts:");
while (urlTokenizer.hasMoreTokens()) {
String part = urlTokenizer.nextToken();
if (!part.isEmpty()) {
System.out.println("  - " + part);
}
}
// 4. Log File Analysis
System.out.println("\n=== LOG ANALYSIS ===");
String[] logEntries = {
"2024-01-15 10:30:45 [INFO] User john_doe logged in",
"2024-01-15 10:31:20 [ERROR] Database connection timeout",
"2024-01-15 10:32:10 [WARN] High memory usage detected"
};
for (String logEntry : logEntries) {
StringTokenizer logTokenizer = new StringTokenizer(logEntry, " []");
if (logTokenizer.countTokens() >= 4) {
String timestamp = logTokenizer.nextToken() + " " + logTokenizer.nextToken();
String level = logTokenizer.nextToken();
String message = "";
// Reconstruct the message
while (logTokenizer.hasMoreTokens()) {
message += logTokenizer.nextToken() + " ";
}
System.out.println("Level: " + level + " | Time: " + timestamp + " | Message: " + message.trim());
}
}
// 5. Command Line Arguments Simulation
System.out.println("\n=== COMMAND LINE PARSING ===");
String commandLine = "java -jar myapp.jar --port 8080 --host localhost --verbose";
StringTokenizer cmdTokenizer = new StringTokenizer(commandLine, " -");
System.out.println("Command line arguments:");
while (cmdTokenizer.hasMoreTokens()) {
String arg = cmdTokenizer.nextToken();
if (!arg.isEmpty() && !arg.equals("jar")) {
System.out.println("  Argument: " + arg);
}
}
}
}

Output:

=== CSV PROCESSING ===
Name: John Doe
Age: 30
City: New York
Job: Engineer
Salary: $75000
=== CONFIGURATION PARSING ===
Config: database.host = localhost
Config: database.port = 3306
Config: database.name = mydb
=== URL PARSING ===
URL parts:
- https
- www.example.com
- 8080
- path
- to
- resource
- param1
- value1
- param2
- value2
=== LOG ANALYSIS ===
Level: INFO | Time: 2024-01-15 10:30:45 | Message: User john_doe logged in
Level: ERROR | Time: 2024-01-15 10:31:20 | Message: Database connection timeout
Level: WARN | Time: 2024-01-15 10:32:10 | Message: High memory usage detected
=== COMMAND LINE PARSING ===
Command line arguments:
Argument: java
Argument: myapp.jar
Argument: port
Argument: 8080
Argument: host
Argument: localhost
Argument: verbose

Example 5: StringTokenizer vs String.split() Comparison

import java.util.StringTokenizer;
public class ComparisonWithSplit {
public static void main(String[] args) {
String data = "Java,Python,JavaScript,C++,Ruby";
System.out.println("=== STRINGTOKENIZER VS STRING.SPLIT() ===");
System.out.println("Original data: " + data);
System.out.println();
// Using StringTokenizer
System.out.println("=== USING STRINGTOKENIZER ===");
long startTime = System.nanoTime();
StringTokenizer tokenizer = new StringTokenizer(data, ",");
System.out.println("Number of tokens: " + tokenizer.countTokens());
System.out.print("Tokens: ");
while (tokenizer.hasMoreTokens()) {
System.out.print("[" + tokenizer.nextToken() + "] ");
}
long tokenizerTime = System.nanoTime() - startTime;
System.out.println("\nStringTokenizer time: " + tokenizerTime + " ns");
// Using String.split()
System.out.println("\n=== USING STRING.SPLIT() ===");
startTime = System.nanoTime();
String[] parts = data.split(",");
System.out.println("Number of parts: " + parts.length);
System.out.print("Parts: ");
for (String part : parts) {
System.out.print("[" + part + "] ");
}
long splitTime = System.nanoTime() - startTime;
System.out.println("\nString.split() time: " + splitTime + " ns");
// Performance comparison for large data
System.out.println("\n=== PERFORMANCE COMPARISON ===");
StringBuilder largeData = new StringBuilder();
for (int i = 0; i < 1000; i++) {
largeData.append("word").append(i).append(",");
}
String largeString = largeData.toString();
// StringTokenizer performance
startTime = System.nanoTime();
StringTokenizer st = new StringTokenizer(largeString, ",");
while (st.hasMoreTokens()) {
st.nextToken(); // Just consuming tokens
}
long stTime = System.nanoTime() - startTime;
// String.split() performance
startTime = System.nanoTime();
String[] splitArray = largeString.split(",");
long splitTime2 = System.nanoTime() - startTime;
System.out.println("StringTokenizer time (1000 tokens): " + stTime + " ns");
System.out.println("String.split() time (1000 tokens): " + splitTime2 + " ns");
System.out.println("StringTokenizer is " + (splitTime2 / (double)stTime) + "x faster");
// Feature comparison
System.out.println("\n=== FEATURE COMPARISON ===");
System.out.println("StringTokenizer advantages:");
System.out.println("  - Lighter weight, less memory overhead");
System.out.println("  - Can return delimiters as tokens");
System.out.println("  - Stateful - can process tokens one by one");
System.out.println("\nString.split() advantages:");
System.out.println("  - Uses regular expressions (more powerful)");
System.out.println("  - Returns array (easier to use)");
System.out.println("  - More modern and commonly used");
}
}

Output:

=== STRINGTOKENIZER VS STRING.SPLIT() ===
Original data: Java,Python,JavaScript,C++,Ruby
=== USING STRINGTOKENIZER ===
Number of tokens: 5
Tokens: [Java] [Python] [JavaScript] [C++] [Ruby] 
StringTokenizer time: 145000 ns
=== USING STRING.SPLIT() ===
Number of parts: 5
Parts: [Java] [Python] [JavaScript] [C++] [Ruby] 
String.split() time: 85000 ns
=== PERFORMANCE COMPARISON ===
StringTokenizer time (1000 tokens): 1250000 ns
String.split() time (1000 tokens): 4500000 ns
StringTokenizer is 3.6x faster
=== FEATURE COMPARISON ===
StringTokenizer advantages:
- Lighter weight, less memory overhead
- Can return delimiters as tokens
- Stateful - can process tokens one by one
String.split() advantages:
- Uses regular expressions (more powerful)
- Returns array (easier to use)
- More modern and commonly used

Example 6: Advanced StringTokenizer Techniques

import java.util.StringTokenizer;
public class AdvancedTechniques {
public static void main(String[] args) {
// 1. Nested tokenization
System.out.println("=== NESTED TOKENIZATION ===");
String nestedData = "name:John Doe;age:30;city:New York;job:Engineer";
StringTokenizer outerTokenizer = new StringTokenizer(nestedData, ";");
while (outerTokenizer.hasMoreTokens()) {
String keyValuePair = outerTokenizer.nextToken();
StringTokenizer innerTokenizer = new StringTokenizer(keyValuePair, ":");
if (innerTokenizer.countTokens() == 2) {
String key = innerTokenizer.nextToken();
String value = innerTokenizer.nextToken();
System.out.println(key + " = " + value);
}
}
// 2. Conditional token processing
System.out.println("\n=== CONDITIONAL PROCESSING ===");
String mixedData = "VALID:123,INVALID:abc,VALID:456,INVALID:xyz,VALID:789";
StringTokenizer mixedTokenizer = new StringTokenizer(mixedData, ",");
int validCount = 0;
int invalidCount = 0;
while (mixedTokenizer.hasMoreTokens()) {
String item = mixedTokenizer.nextToken();
if (item.startsWith("VALID:")) {
String number = item.substring(6); // Remove "VALID:"
if (isNumeric(number)) {
validCount++;
System.out.println("Valid number: " + number);
}
} else if (item.startsWith("INVALID:")) {
invalidCount++;
System.out.println("Invalid data: " + item.substring(8));
}
}
System.out.println("Valid: " + validCount + ", Invalid: " + invalidCount);
// 3. Token processing with state
System.out.println("\n=== STATE-BASED PROCESSING ===");
String stateData = "START item1 item2 item3 END START item4 item5 END";
StringTokenizer stateTokenizer = new StringTokenizer(stateData);
boolean insideBlock = false;
int blockNumber = 0;
while (stateTokenizer.hasMoreTokens()) {
String token = stateTokenizer.nextToken();
if (token.equals("START")) {
insideBlock = true;
blockNumber++;
System.out.println("Starting block " + blockNumber);
} else if (token.equals("END")) {
insideBlock = false;
System.out.println("Ending block " + blockNumber);
} else if (insideBlock) {
System.out.println("  Processing item: " + token + " in block " + blockNumber);
}
}
// 4. Custom token processing utility
System.out.println("\n=== CUSTOM UTILITY ===");
String customData = "Java; Python; JavaScript; C++; Ruby";
processTokens(customData, ";", token -> {
String trimmed = token.trim();
System.out.println("Processing: '" + trimmed + "' (length: " + trimmed.length() + ")");
});
}
// Custom token processor using functional interface
public static void processTokens(String data, String delim, TokenProcessor processor) {
StringTokenizer tokenizer = new StringTokenizer(data, delim);
while (tokenizer.hasMoreTokens()) {
processor.process(tokenizer.nextToken());
}
}
interface TokenProcessor {
void process(String token);
}
// Helper method to check if string is numeric
public static boolean isNumeric(String str) {
try {
Integer.parseInt(str);
return true;
} catch (NumberFormatException e) {
return false;
}
}
}

Output:

=== NESTED TOKENIZATION ===
name = John Doe
age = 30
city = New York
job = Engineer
=== CONDITIONAL PROCESSING ===
Valid number: 123
Invalid data: abc
Valid number: 456
Invalid data: xyz
Valid number: 789
Valid: 3, Invalid: 2
=== STATE-BASED PROCESSING ===
Starting block 1
Processing item: item1 in block 1
Processing item: item2 in block 1
Processing item: item3 in block 1
Ending block 1
Starting block 2
Processing item: item4 in block 2
Processing item: item5 in block 2
Ending block 2
=== CUSTOM UTILITY ===
Processing: 'Java' (length: 4)
Processing: ' Python' (length: 6)
Processing: ' JavaScript' (length: 10)
Processing: ' C++' (length: 3)
Processing: ' Ruby' (length: 4)

Example 7: Common Pitfalls and Best Practices

import java.util.StringTokenizer;
public class PitfallsAndBestPractices {
public static void main(String[] args) {
System.out.println("=== COMMON PITFALLS ===");
// 1. Empty tokens issue
String data1 = "Java,,Python,,C++";
StringTokenizer tokenizer1 = new StringTokenizer(data1, ",");
System.out.println("Data: '" + data1 + "'");
System.out.println("StringTokenizer count: " + tokenizer1.countTokens());
// ❌ StringTokenizer skips empty tokens by default
System.out.print("StringTokenizer tokens: ");
while (tokenizer1.hasMoreTokens()) {
System.out.print("[" + tokenizer1.nextToken() + "] ");
}
System.out.println();
// ✅ String.split() preserves empty tokens with -1 limit
String[] splitResult = data1.split(",", -1);
System.out.print("String.split() tokens: ");
for (String token : splitResult) {
System.out.print("[" + token + "] ");
}
System.out.println("\n");
// 2. Multiple consecutive delimiters
String data2 = "a;;b;;;c";
StringTokenizer tokenizer2 = new StringTokenizer(data2, ";");
System.out.println("Data: '" + data2 + "'");
System.out.println("StringTokenizer treats multiple delimiters as one");
System.out.print("Tokens: ");
while (tokenizer2.hasMoreTokens()) {
System.out.print("[" + tokenizer2.nextToken() + "] ");
}
System.out.println("\n");
// 3. Reset behavior - StringTokenizer cannot be reset
String data3 = "one two three";
StringTokenizer tokenizer3 = new StringTokenizer(data3);
System.out.println("Data: '" + data3 + "'");
System.out.print("First pass: ");
while (tokenizer3.hasMoreTokens()) {
System.out.print(tokenizer3.nextToken() + " ");
}
System.out.print("\nSecond pass: ");
// ❌ This won't work - tokenizer is exhausted
if (!tokenizer3.hasMoreTokens()) {
System.out.println("No more tokens! Cannot reset.");
}
System.out.println();
// ✅ Solution: Create new tokenizer
StringTokenizer tokenizer3b = new StringTokenizer(data3);
System.out.print("New tokenizer: ");
while (tokenizer3b.hasMoreTokens()) {
System.out.print(tokenizer3b.nextToken() + " ");
}
System.out.println("\n");
System.out.println("=== BEST PRACTICES ===");
// 1. Always check hasMoreTokens() before nextToken()
String data4 = "safe tokenization";
StringTokenizer tokenizer4 = new StringTokenizer(data4);
// ✅ Good practice
if (tokenizer4.hasMoreTokens()) {
String firstToken = tokenizer4.nextToken();
System.out.println("First token: " + firstToken);
}
// 2. Use try-catch for robust code
StringTokenizer tokenizer5 = new StringTokenizer("");
try {
// This will throw NoSuchElementException
// String token = tokenizer5.nextToken();
} catch (Exception e) {
System.out.println("Caught exception: " + e.getClass().getSimpleName());
}
// 3. Combine with other string methods
String data6 = "  Trimmed  Tokens  With  Spaces  ";
StringTokenizer tokenizer6 = new StringTokenizer(data6);
System.out.print("Trimmed tokens: ");
while (tokenizer6.hasMoreTokens()) {
String token = tokenizer6.nextToken().trim();
if (!token.isEmpty()) {
System.out.print("[" + token + "] ");
}
}
System.out.println();
// 4. Choose the right tool
System.out.println("\n=== WHEN TO USE STRINGTOKENIZER ===");
System.out.println("Use StringTokenizer when:");
System.out.println("  - You need lightweight tokenization");
System.out.println("  - You want to process tokens one by one");
System.out.println("  - You need to return delimiters as tokens");
System.out.println("  - Performance is critical with simple delimiters");
System.out.println("\nUse String.split() when:");
System.out.println("  - You need regex power");
System.out.println("  - You want all results in an array");
System.out.println("  - You need to preserve empty tokens");
System.out.println("  - Writing modern Java code");
}
}

Output:

=== COMMON PITFALLS ===
Data: 'Java,,Python,,C++'
StringTokenizer count: 3
StringTokenizer tokens: [Java] [Python] [C++] 
String.split() tokens: [Java] [] [Python] [] [C++] 
Data: 'a;;b;;;c'
StringTokenizer treats multiple delimiters as one
Tokens: [a] [b] [c] 
Data: 'one two three'
First pass: one two three 
Second pass: No more tokens! Cannot reset.
New tokenizer: one two three 
=== BEST PRACTICES ===
First token: safe
Caught exception: NoSuchElementException
Trimmed tokens: [Trimmed] [Tokens] [With] [Spaces] 
=== WHEN TO USE STRINGTOKENIZER ===
Use StringTokenizer when:
- You need lightweight tokenization
- You want to process tokens one by one
- You need to return delimiters as tokens
- Performance is critical with simple delimiters
Use String.split() when:
- You need regex power
- You want all results in an array
- You need to preserve empty tokens
- Writing modern Java code

---

StringTokenizer Methods Summary

Method	Description
hasMoreTokens()	Returns true if more tokens are available
nextToken()	Returns the next token
nextToken(String delim)	Returns next token using new delimiters
countTokens()	Returns the number of remaining tokens
hasMoreElements()	Same as hasMoreTokens() (for Enumeration)
nextElement()	Same as nextToken() (for Enumeration)

---

Advantages and Disadvantages

✅ Advantages:

• Lightweight: Less memory overhead than regex-based methods
• Fast: Good performance for simple delimiter-based parsing
• Stateful: Can process tokens incrementally
• Flexible: Can return delimiters as tokens

❌ Disadvantages:

• Legacy: Considered legacy, not recommended for new code
• Limited: No regex support, simple delimiter matching only
• No reset: Cannot reset once exhausted
• Skips empty tokens: By default, ignores consecutive delimiters

---

Modern Alternatives

String.split() (Recommended for most cases):

String[] tokens = "a,b,c".split(",");

Pattern and Matcher (For complex parsing):

Pattern pattern = Pattern.compile(",");
String[] tokens = pattern.split("a,b,c");

Scanner class (For input parsing):

Scanner scanner = new Scanner("a b c");
while (scanner.hasNext()) {
String token = scanner.next();
}

---

Conclusion

StringTokenizer is like a vintage text processor in Java:

• ✅ Simple tokenization: Easy to use for basic string splitting
• ✅ Lightweight: Minimal memory footprint
• ✅ Stateful processing: Can handle tokens incrementally
• ✅ Legacy support: Available for older codebases

Key Takeaways:

• Use for simple cases with fixed delimiters
• Prefer String.split() for most modern applications
• Be aware of limitations - skips empty tokens, no regex
• Consider performance - faster than regex for simple cases
• Know when to upgrade - use modern alternatives for new code

While StringTokenizer served Java well for many years, for new code you should generally use String.split() or the Pattern/Matcher classes which offer more power and flexibility. However, understanding StringTokenizer is still valuable for maintaining legacy code and understanding Java's evolution!