strtok

francesco dot montorsi at gmail dot com (2013-04-04 10:33:19)

A version of strtok() that returns an array containing all tokens found could be written as:
function strtok_all($string, $delim)
{
//echo "string: " . $string . " delim: " . $delim . "<br/>";
$tok = strtok($string, $delim);
$arr = array();
//echo "tok: " . $tok;
while ($tok !== false)
{
$arr[] = $tok;
$tok = strtok($delim);
}

return $arr;
}
note that this works similarly to explode() but, unlike explode(), ANY of the character provided as $delim will be used to tokenize the string!

gilthans at NOSPAM dot gmail dot com (2012-07-15 18:47:47)

Note that strtok may receive different tokens each time. Therefore, if, for example, you wish to extract several words and then the rest of the sentence:

<?php
$text = "13 202 5 This is a long message explaining the error codes.";
$error1 = strtok($text, " "); //13
$error2 = strtok(" "); //202
$error3 = strtok(" "); //5
$error_message = strtok(""); //Notice the different token parameter
echo $error_message; //This is a long message explaining the error codes.
?>

emmanuel dot antico at gmail dot com (2012-03-15 13:18:08)

get the last part of a string after a given character in one line

<?php
$class_name = strrev(strtok(strrev('java.lang.Object'), '.'));
echo $class_name;
?>

will output:

Object

pradador at me dot com (2011-05-13 12:10:54)

Here's a simple class that allows you to iterate through string tokens using a foreach loop.

<?php
/**
 * The TokenIterator class allows you to iterate through string tokens using
 * the familiar foreach control structure.
 * 
 * Example:
 * <code>
 * <?php
 * $string = 'This is a test.';
 * $delimiters = ' ';
 * $ti = new TokenIterator($string, $delimiters);
 * 
 * foreach ($ti as $count => $token) {
 *     echo sprintf("%d, %s\n", $count, $token); 
 * }
 * 
 * // Prints the following output:
 * // 0. This
 * // 1. is
 * // 2. a
 * // 3. test.
 * </code>
 */
class TokenIterator implements Iterator
{
    /**
     * The string to tokenize.
     * @var string
     */
    protected $_string;
    
    /**
     * The token delimiters.
     * @var string
     */
    protected $_delims;
    
    /**
     * Stores the current token.
     * @var mixed
     */
    protected $_token;
    
    /**
     * Internal token counter.
     * @var int
     */
    protected $_counter = 0;
    
    /**
     * Constructor.
     * 
     * @param string $string The string to tokenize.
     * @param string $delims The token delimiters.
     */
    public function __construct($string, $delims)
    {
        $this->_string = $string;
        $this->_delims = $delims;
        $this->_token = strtok($string, $delims);
    }
    
    /**
     * @see Iterator::current()
     */
    public function current()
    {
        return $this->_token;
    }

    /**
     * @see Iterator::key()
     */
    public function key()
    {
        return $this->_counter;
    }

    /**
     * @see Iterator::next()
     */
    public function next()
    {
        $this->_token = strtok($this->_delims);
        
        if ($this->valid()) {
            ++$this->_counter;
        }
    }

    /**
     * @see Iterator::rewind()
     */
    public function rewind()
    {
        $this->_counter = 0;
        $this->_token   = strtok($this->_string, $this->_delims);
    }

    /**
     * @see Iterator::valid()
     */
    public function valid()
    {
        return $this->_token !== FALSE;
    }
}
?>

elarlang at gmail dot com (2011-03-22 10:45:01)

If you have memory-usage critical solution, you should keep in mind, that strtok function holds input string parameter (or reference to it?) in memory after usage.

<?php
function tokenize($str, $token_symbols) {
    $word = strtok($str, $token_symbols);
    while (false !== $word) {
        // do something here...
        $word = strtok($token_symbols);
    }
}
?>
Test-cases with handling ~10MB plain-text file:
Case #1 - unset $str variable
<?php
$token_symbols = " \t\n";
$str = file_get_contents('10MB.txt'); // mem usage 9.75383758545 MB (memory_get_usage() / 1024 / 1024)); 
tokenize($str, $token_symbols); // mem usage 9.75400161743 MB
unset($str); // 9.75395584106 MB
?>
Case #1 result: memory is still used

Case #2 - call strtok again
<?php
$token_symbols = " \t\n";
$str = file_get_contents('10MB.txt'); // 9.75401306152 MB
tokenize($str, $token_symbols); // 9.75417709351
strtok('', ''); // 9.75421524048
?>
Case #2 result: memory is still used

Case #3 - call strtok again AND unset $str variable
<?php
$token_symbols = " \t\n";
$str = file_get_contents('10MB.txt'); // 9.75410079956 MB
tokenize($str, $token_symbols); // 9.75426483154 MB
unset($str);
strtok('', ''); // 0.0543975830078 MB
?>
Case #3 result: memory is free

So, better solution for tokenize function:
<?php
function tokenize($str, $token_symbols, $token_reset = true) {
    $word = strtok($str, $token_symbols);
    while (false !== $word) {
        // do something here...
        $word = strtok($token_symbols);
    }

    if($token_reset)
        strtok('', '');
}
?>

fabiolimasouto at gmail dot com (2010-11-20 05:54:36)

this example will hopefully help you understand how this function works:

<?php
$selector = 'div.class#id';
$tagname = strtok($selector,'.#');
echo $tagname.'<br/>';

while($tok = strtok('.#'))
{
 echo $tok.'<br/>';
}

?>

Outputs:
div
class
id

php at ciutureanu dot ro (2010-09-08 08:08:24)

I wrote a function which adds links based on a list of keywords => hrefs, avoiding attributes inside HTML tags which matches those keywords.

<?php
    function addLinks($str_in, $replaces) {
        $str_out = '';
        $tok = strtok($str_in, '<>');
        $must_replace = (substr($str_in, 0, 1) !== '<');
        while ($tok !== false) {
            if ($must_replace) {
                foreach ($replaces as $tag => $href) {
                    if (preg_match('/\b' . $tag . '\b/', $tok)) {
                        $tok = preg_replace('/\b(' . $tag . ')\b/', '<a title="' . $href . '" href="' . $href . '">\1</a>', $tok, 1);
                        unset($replaces[$tag]);
                    }
                }
            } else {
                $tok = "<$tok>";
            }
            $str_out .= $tok;
            $tok = strtok('<>');
            $must_replace = !$must_replace;
        }
        return $str_out;
    }

    $replaces = array(
        'Obama' => 'http://en.wikipedia.org/wiki/Barack_Obama',
        'php' => 'http://www.php.net/',
    );
    $s = 'This php function will <em title="Obama">replace</em> only first occurance of Obama, but not the second occurance of Obama';
    echo "$s<hr />" . addLinks($s, $replaces);
?>

benighted at gmail dot com (2009-11-06 06:47:15)

Simple way to tokenize search parameters, including double or single quoted keys.  If only one quote is found, the rest of the string is assumed to be part of that token.

<?php
            $token = strtok($keywords,' ');
            while ($token) {
                // find double quoted tokens
                if ($token{0}=='"') { $token .= ' '.strtok('"').'"'; }
                // find single quoted tokens
                if ($token{0}=="'") { $token .= ' '.strtok("'")."'"; }

                $tokens[] = $token;
                $token = strtok(' ');
            }
?>

Use substr(1,strlen($token)) and remove the part that adds the trailing quotes if you want your output without quotes.

azeem (2009-09-02 12:14:05)

Here is a java like StringTokenizer class using strtok function:

<?php

/**
 * The string tokenizer class allows an application to break a string into tokens.
 *
 * @example The following is one example of the use of the tokenizer. The code:
 * <code>
 * <?php
 *    $str = 'this is:@\t\n a test!';
 *    $delim = ' !@:'\t\n; // remove these chars
 *    $st = new StringTokenizer($str, $delim);
 *    while ($st->hasMoreTokens()) {
 *        echo $st->nextToken() . "\n";
 *    }
 *    prints the following output:
 *      this
 *      is
 *      a
 *      test
 * ?>
 * </code>
 */
class StringTokenizer {

    /**
     * @var string
     */
    private $token;

    /**
     * @var string
     */
    private $delim;
    /**
     * Constructs a string tokenizer for the specified string
     * @param string $str String to tokenize
     * @param string $delim The set of delimiters (the characters that separate tokens)
     * specified at creation time, default to ' '
     */
    public function __construct(/*string*/ $str, /*string*/ $delim = ' ') {
        $this->token = strtok($str, $delim);
        $this->delim = $delim;
    }

    public function __destruct() {
        unset($this);
    }

    /**
     * Tests if there are more tokens available from this tokenizer's string. It
     * does not move the internal pointer in any way. To move the internal pointer
     * to the next element call nextToken()
     * @return boolean - true if has more tokens, false otherwise
     */
    public function hasMoreTokens() {
        return ($this->token !== false);
    }

    /**
     * Returns the next token from this string tokenizer and advances the internal
     * pointer by one.
     * @return string - next element in the tokenized string
     */
    public function nextToken() {
        $current = $this->token;
        $this->token = strtok($this->delim);
        return $current;
    }
}
?>

yanick dot rochon at gmail dot com (2009-06-15 09:37:27)

Here is a small function I wrote as I needed to extract some named tokens from a string (a la Google). For example, I needed to format a string like "extension:gif size:64M animated:true author:'John Bash'" into

array(
  'extension' => 'gif',
  'size' => '64M',
  'animated' => true,
  'author' => 'John Bash'
)

So, here's the code: 

<?php

header('Content-type: text/plain; charset=utf-8');

/**
 * NOTE : use mbstring.func_overload for multi-byte support with this function
 *
 * @param string $string             the string to tokenize
 * @param int $offset                the starting offset
 * @param string $defaultTokenName   the default token name if none specified
 * @param string $groupDelimiters    the characters to delimit token groups
 * @param string $groupNameDelimiter the character(s) to delimit token group names
 * @return array
 */
function getTokens(
        $string, 
        $offset = 0, 
        $defaultTokenName = null, 
        $groupDelimiters = '\'"',
        $groupNameDelimiter = ':') 
{

    if ($offset >= strlen($string)) {
        //echo "offset out of range";
        return false;
    }

    $spaces = " \t\n\r";   // space characters

    // add group delimiters to spaces...
    $groupSpaces = $spaces . $groupNameDelimiter;
    $delimiters = $groupSpaces . $groupDelimiters;

    //var_dump($groupSpaces);

    $string = ltrim(substr($string, $offset), $groupSpaces);
    $token_strings = array();

    //echo "String is : " . $string . "\n";

    // 1. split all tokens...
    while ($offset < strlen($string)) {
        $lastOffset = $offset;
        $escaped = false;

        if (false !== strpos($groupDelimiters, $char = $string[$offset])) {
            $groupChar = $char;
        } else {
            $groupChar = null;
        }

        if (null !== $groupChar) {
            while (($offset < strlen($string)) && (($groupChar !== ($char = $string[++$offset])) || $escaped)) {
                //$offset++;
                $escaped = ('\\' === $char);
            }
            $offset++;
            //echo "*** Grouped : " . substr($string, $lastOffset, $offset - $lastOffset) . "\n";
        } else {
            while (($offset < strlen($string)) && ((false === strpos($delimiters, $char = $string[$offset])) || $escaped)) {
                $offset++;
                $escaped = ('\\' === $char);
            }
            //echo "*** Non-group : " . substr($string, $lastOffset, $offset - $lastOffset) . "\n";
        }
        //skip spaces...
        while (($offset < strlen($string)) && ((false !== strpos($groupSpaces, $char = $string[$offset])) || $escaped)) {
            $offset++;
            $escaped = ('\\' === $char);
        }

        $token_strings[] = substr($string, $lastOffset, $offset - $lastOffset);
        //echo "Next token = '" . end($token_strings) . "'\n";
    }

    $tokens = array();
    $tokenName = null;
    foreach ($token_strings as $token_str) {
        // clean $token_str
        $token_str = trim(stripslashes($token_str), $spaces);
        $str_value = trim($token_str, $delimiters);
        switch (strtolower($str_value)) {
            case 'true': $str_value = true; break;
            case 'false': $str_value = false; break;
            default: break;
        }

        // is it a token name?
        if (':' === substr($token_str, -1, 1)) {
            if (!empty($tokenName)) {
                $tokens[$tokenName] = '';
            }
            $tokenName = trim($token_str, $delimiters);
        } else {
            if (!empty($tokenName)) {
                if (isset($tokens[$tokenName])) {
                    $tokens[$tokenName] = array(
                        $tokens[$tokenName],
                        $str_value
                    );
                } else {
                    $tokens[$tokenName] = $str_value;
                }
                $tokenName = null;
            } elseif (empty($defaultTokenName)) {
                $tokens[] = trim($token_str, $delimiters);;
            } else {
                if (isset($tokens[$defaultTokenName])) {
                    $tokens[$defaultTokenName] = array(
                        $tokens[$defaultTokenName],
                        $str_value
                    );
                } else {
                    $tokens[$defaultTokenName] = $str_value;
                }
            }
        }
    }
    if (!empty($tokenName)) {
        $tokens[$tokenName] = '';
    }

    return $tokens;
}

$str = "check1: test "
     . "check2:'hello world' "
     . 'check3: "foo" '
     . "check4: \\\"try this\\\""
     . '"buz" '
     . 'check1:true';

?>

KrazyBox (2009-03-06 18:11:06)

As of the change in strtok()'s handling of empty strings, it is now useless for scripts that rely on empty data to function.
Take for instance, a standard header. (with UNIX newlines)
http/1.0 200 OK\n
Content-Type: text/html\n
\n
--HTML BODY HERE---
When parsing this with strtok, one would wait until it found an empty string to signal the end of the header. However, because strtok now skips empty segments, it is impossible to know when the header has ended.
This should not be called `correct' behavior, it certainly is not. It has rendered strtok incapable of (properly) processing a very simple standard.
This new functionality, however, does not affect Windows style headers. You would search for a line that only contains "\r"
This, however, is not a justification for the change.

Logikos (2009-01-03 22:12:02)

This looks very simple, but it took me a long time to figure out so I thought I'd share it incase someone else was wanting the same thing:

this should work similar to substr() but with tokens instead!

<?php
/* subtok(string,chr,pos,len)
 *
 * chr = chr used to seperate tokens
 * pos = starting postion
 * len = length, if negative count back from right
 * 
 *  subtok('a.b.c.d.e','.',0)     = 'a.b.c.d.e'
 *  subtok('a.b.c.d.e','.',0,2)   = 'a.b'
 *  subtok('a.b.c.d.e','.',2,1)   = 'c'
 *  subtok('a.b.c.d.e','.',2,-1)  = 'c.d'
 *  subtok('a.b.c.d.e','.',-4)    = 'b.c.d.e'
 *  subtok('a.b.c.d.e','.',-4,2)  = 'b.c'
 *  subtok('a.b.c.d.e','.',-4,-1) = 'b.c.d'
 */
function subtok($string,$chr,$pos,$len = NULL) {
  return implode($chr,array_slice(explode($chr,$string),$pos,$len));
}
?>

explode breaks the tokens up into an array, array slice alows you to pick then tokens you want, and then implode converts it back to a string

although its far from a clone, this was inspired by mIRC's gettok() function

magne (2008-06-25 11:43:14)

Quick and dirty function to tokenize:
function stringTokenize($sBuffer, $sSplit) {
$iCount = 0;

if(strlen($sBuffer) == 0)
return;

$sToken = strtok($sBuffer, $sSplit);
$aTokens[$iCount] = $sToken;
while ($sToken !== false) {
$sToken = strtok($sSplit);
if(strlen($sToken) > 0) {
$iCount++;
$aTokens[$iCount] = $sToken;
}
} // end while

return $aTokens;
}

Aditya P Bhatt (adityabhai at gmail dot com) (2008-03-27 22:16:34)

<head>
<title>Tokenise the incoming string</title>
</head>
<body bgcolor=white>
The input was
<?php
$name = "SOME TEXT EXAMPLE"; // You can change string here
print ("$name.<P>In pieces, thats:<br>");
 
$toke = strtok($name," \t");
    while ($toke) {
        print ("$toke<BR>");
        $toke = strtok(" \t");
    }
$oneword = implode("_",explode(" ",$name));
print ("<br>or as one word: $oneword");
 
?>
</body>

Geert (2006-04-14 06:37:08)

Shauns function needs a little update because it produces an error message that the variables $text and $words were not defined. Written like this it won't produce an error:

<?php
function summarize($paragraph, $limit){
  $tok = strtok($paragraph, " ");
  $text="";
  $words='0';
  while($tok){
   $text .= " ".$tok;
   $words++;
   if(($words >= $limit) && ((substr($tok, -1) == "!")||(substr($tok, -1) == ".")))
     break;
   $tok = strtok(" ");
  }
return ltrim($text);
}
?>

mac.com@nemo (2006-02-18 13:49:39)

This function takes a string and returns an array with words (delimited by spaces), also taking into account quotes, doublequotes, backticks and backslashes (for escaping stuff).
So

$string = "cp   'my file' to `Judy's file`";
var_dump(parse_cli($string));

would yield:

array(4) {
  [0]=>
  string(2) "cp"
  [1]=>
  string(7) "my file"
  [2]=>
  string(5) "to"
  [3]=>
  string(11) "Judy's file"
}

Way it works, runs through the string character by character, for each character looking up the action to take, based on that character and its current $state.
Actions can be (one or more of) adding the character/string to the current word, adding the word to the output array, and changing or (re)storing the state.
For example a space will become part of the current 'word' (or 'token') if $state is 'doublequoted', but it will start a new token if $state was 'unquoted'.
I was later told it's a "tokeniser using a finite state automaton". Who knew :-)

<?php

#_____________________
# parse_cli($string) /
function parse_cli($string) {
    $state = 'space';
    $previous = '';     // stores current state when encountering a backslash (which changes $state to 'escaped', but has to fall back into the previous $state afterwards)
    $out = array();     // the return value
    $word = '';
    $type = '';         // type of character
    // array[states][chartypes] => actions
    $chart = array(
        'space'        => array('space'=>'',   'quote'=>'q',  'doublequote'=>'d',  'backtick'=>'b',  'backslash'=>'ue', 'other'=>'ua'),
        'unquoted'     => array('space'=>'w ', 'quote'=>'a',  'doublequote'=>'a',  'backtick'=>'a',  'backslash'=>'e',  'other'=>'a'),
        'quoted'       => array('space'=>'a',  'quote'=>'w ', 'doublequote'=>'a',  'backtick'=>'a',  'backslash'=>'e',  'other'=>'a'),
        'doublequoted' => array('space'=>'a',  'quote'=>'a',  'doublequote'=>'w ', 'backtick'=>'a',  'backslash'=>'e',  'other'=>'a'),
        'backticked'   => array('space'=>'a',  'quote'=>'a',  'doublequote'=>'a',  'backtick'=>'w ', 'backslash'=>'e',  'other'=>'a'),
        'escaped'      => array('space'=>'ap', 'quote'=>'ap', 'doublequote'=>'ap', 'backtick'=>'ap', 'backslash'=>'ap', 'other'=>'ap'));
    for ($i=0; $i<=strlen($string); $i++) {
        $char = substr($string, $i, 1);
        $type = array_search($char, array('space'=>' ', 'quote'=>'\'', 'doublequote'=>'"', 'backtick'=>'`', 'backslash'=>'\\'));
        if (! $type) $type = 'other';
        if ($type == 'other') {
            // grabs all characters that are also 'other' following the current one in one go
            preg_match("/[ \'\"\`\\\]/", $string, $matches, PREG_OFFSET_CAPTURE, $i);
            if ($matches) {
                $matches = $matches[0];
                $char = substr($string, $i, $matches[1]-$i); // yep, $char length can be > 1
                $i = $matches[1] - 1;
            }else{
                // no more match on special characters, that must mean this is the last word!
                // the .= hereunder is because we *might* be in the middle of a word that just contained special chars
                $word .= substr($string, $i);
                break; // jumps out of the for() loop
            }
        }
        $actions = $chart[$state][$type];
        for($j=0; $j<strlen($actions); $j++) {
            $act = substr($actions, $j, 1);
            if ($act == ' ') $state = 'space';
            if ($act == 'u') $state = 'unquoted';
            if ($act == 'q') $state = 'quoted';
            if ($act == 'd') $state = 'doublequoted';
            if ($act == 'b') $state = 'backticked';
            if ($act == 'e') { $previous = $state; $state = 'escaped'; }
            if ($act == 'a') $word .= $char;
            if ($act == 'w') { $out[] = $word; $word = ''; }
            if ($act == 'p') $state = $previous;
        }
    }
    if (strlen($word)) $out[] = $word;
    return $out;
}

?>

rawat dot arun at gmail dot com (2005-08-25 19:31:56)

I was trying to compare two strings of equal length using strtok. However using them at same time leads into erratic output. Therefore the output of each strok can first be stored in an array and then be used for comparison. Here is small code for it.
<?php
$string = "This is an XYZ example string";
$tok = strtok($string, ' ');
while ($tok !== false) { $toks[] = $tok; $tok = strtok(' '); }
$string_1= "This is an unknown example string";
$tok1= strtok($string_1, ' ');while ($tok1 !== false) {
$toks1[] = $tok1;$tok1 = strtok(' '); }
$ctr=0;
while (each ($toks))
if ($toks[$ctr]==$toks1[$ctr])
{echo "W=$toks[$ctr]<br />"; echo "W1=$toks1[$ctr]<br />";
  $ctr++;  } else $ctr++;     ?> 
Thanks,
Arun

brian dot cairns dot remove dot this at commerx dot com (2005-05-26 12:11:22)

I was looking for a function to tokenize a string, taking double-quoted inline strings into account (for breaking up search queries, for example), but none of the ones I found seemed particularly efficient or elegant, so I wrote my own.  Here it is:

<?
// split a string into an array of space-delimited tokens, taking double-quoted strings into account
function tokenizeQuoted($string)
{
    for($tokens=array(), $nextToken=strtok($string, ' '); $nextToken!==false; $nextToken=strtok(' '))
    {
        if($nextToken{0}=='"')
            $nextToken = $nextToken{strlen($nextToken)-1}=='"' ? 
                substr($nextToken, 1, -1) : substr($nextToken, 1) . ' ' . strtok('"');
        $tokens[] = $nextToken;
    }
    return $tokens;
}
?>

Example:

$tokens = tokenizeQuoted('this is "my test string" single "words" work too');

Results in $tokens containing:

Array
(
    [0] => this
    [1] => is
    [2] => my test string
    [3] => single
    [4] => words
    [5] => work
    [6] => too
)

Hope this helps someone.

soletan at toxa dot de (2005-04-28 04:47:31)

strtok's new behaviour isn't more correct than the old one.
Example: When parsing a string for a quoted-string (e.g. RFC822-header without wanting to install mailparse from PECL!) then I walk char by char and whenever I encounter a double-quote I take strtok to find the related closing double-quote in string quite easily ... this is done for improved performance.
But what if there's an empty quoted-string ...
Another example is then having lines like
name="quoted-value"; second="another one";
I get the name using strtok with '=', then I check value to be quoted, which is true and thus I take the method described before to get the quoted string. Then all what's left is
; second="another one";
now I advance and drop any whitespaces after current value assignment ... well users shouldn't obey to never ever in life have no whitespaces before that damn semicolon for sure, and that's why I drop that with strtok (using ';') again to get to the next optional assignment with another
$s = strtok( '' )
I KNOW, there are ways to work around this using trim and the alikes. But that doesn't explain why strtok is now working "correct" while it didn't do before ...

cs2xz at bath dot ac dot uk (2005-04-07 06:30:15)

There is a method to remove all the punctuations and only put the words into an array called "$token", where variable $invalid lists all the punctuations and "\xxx" are the octal numbers of punctuations. At the end, dispalys total number of words in $string and the 4th words in the string.
$string = "Hello, $%^\n\\\"jeff!!!!\"/. 'How are you!'";
$invalid = "\40\41\42\43\44\45\46\47\48\49\50\51\52\53 \54\55\56\57\72\73\74\75\76 \77\100\133\134\135\136\137\138\139\140 \173\174\175\176\n\r\t";
$tok = strtok($string, $invalid);
while ($tok) {
echo "Word=$tok<br>";
$token[]=$tok;
$tok = strtok($invalid);
}
// displays the number of words in the string and the 4th word
echo "Number of token: " . count($token) . "<br>";
echo $token[3];

James (2004-09-28 18:51:56)

Be very careful with using strtok if there's any chance that you may be calling other functions that may use strtok as well. If any other function that you call while parsing the string decides to call strtok as well, it will clobber the internal string pointer being used by strtok and you may get unexpected results. Here's some code to explain what I mean:
function parse_string2($string2) {
for($tok = strtok($string2, '.'); $tok !== false; $tok = strtok(".")) {
echo $tok;
}
}
$string1 = "1.2.3.4.!.8.9";
$string2 = "5.6.7";
for($word = strtok($string1, '.'); $word !== false; $word = strtok(".")) {
if ($word == '!') {
echo parse_string2($string2);
} else {
echo $word;
}
}
If I didn't know the internals of the function parse_string2 (say someone else develops that), but all I know is that parse_string2 should print out 567, then my expected output might be:
123456789
Instead, you only get: 1234567.
It would be interesting if they could implement a strtok_r where you could explicitly denote which string to tokenize.

manicdepressive at mindless dot com (2004-06-19 06:01:53)

<pre><?php
/** get leading, trailing, and embedded separator tokens that were 'skipped'
if for some ungodly reason you are using php to implement a simple parser that 
needs to detect nested clauses as it builds a parse tree */

$str = "(((alpha(beta))(gamma))";

$seps = '()';
$tok = strtok( $str,$seps ); // return false on empty string or null
$cur = 0;      
$dumbDone = FALSE;
$done = (FALSE===$tok);
while (!$done) {
   // process skipped tokens (if any at first iteration) (special for last)
   $posTok = $dumbDone ? strlen($str) : strpos($str, $tok, $cur );
   $skippedMany = substr( $str, $cur, $posTok-$cur ); // false when 0 width
   $lenSkipped = strlen($skippedMany); // 0 when false
   if (0!==$lenSkipped) {
      $last = strlen($skippedMany) -1;
      for($i=0; $i<=$last; $i++){
         $skipped = $skippedMany[$i];
         $cur += strlen($skipped);
         echo "skipped: $skipped\n";
      }
   }
   if ($dumbDone) break; // this is the only place the loop is terminated

   // process current tok
   echo "curr tok: ".$tok."\n";

   // update cursor
   $cur += strlen($tok);

   // get any next tok
   if (!$dumbDone){
      $tok = strtok($seps);
      $dumbDone = (FALSE===$tok); 
      // you're not really done till you check for trailing skipped
   }
};
?></pre>

DethMetal Jeff (2004-03-10 10:18:18)

If you need to parse through a very large delimited text file (such as a word list) combine strtok with file_get_contents. It is much faster than all of the other alternatives i have found (using file() to parse the file into an array, reading the file line by line using fgets())
$dictionary=file_get_contents('path/to/dictionary', 1);

//check that the file was read properly
if(!$dictionary){
return("read error");
}

//dictionary is \n delimited
$tok=strtok($dictionary, "\n");
//loop through until we reach the end of the string
while($tok){
//do whatever it is you need to do with the $tok string here
$tok=strtok("\n"); //get next string
}

torsten at KILL_ALL_SPAM dot dargers dot de (2003-12-20 20:00:07)

Beware! This function cannot be used to start a recursion during the loop. Sh..
You have to collect the results in an array and then cycle the recursion through that array.
Example:
$word=strtok($line,TOKENS);
while ($word) {
// DO NOT START RECURSION HERE USING $word PARAMETER
$words[] = $word;
}
foreach( $words as $word ) {
*RECURSE*($word);
}
// This seems very silly but as the function is not instantiated between recursions it cannot work directly.

jrust AT rustyparts DOT com (2003-05-23 19:09:18)

Had a website which was using way too many of the old functionality of strtok to convert to the new >PHP 4.1.0 way so I wrote this function to mimic the way strtok was done prior to 4.1.0
function strtok_old($string, $delim = null) {
static $origDelim, $origString, $origPos;
if (!isset($origDelim)) {
$origDelim = null;
}
if (!isset($origString)) {
$origString = null;
}
if (!isset($origPos)) {
$origPos = null;
}
// continuing an already started strtok
if ($string == $origDelim) {
$string = $origString;
$delim = $origDelim;
}
// else starting from scratch
else {
$origString = $string;
$origDelim = $delim;
$origPos = 0;
}
if ($origPos !== false && $origPos < strlen($string)) {
$newPos = strpos($string, $delim, $origPos);
}
else {
$newPos = false;
}
// the token wasn't found, go to end of string
if ($newPos === false) {
$newPos = strlen($string);
}
$return = substr($string, $origPos, ($newPos - $origPos));
$origPos = ++$newPos;
return $return;
}

shaun at phplabs dot com (2002-12-04 21:57:17)

Here's some code to extract the first part of a long paragraph, e.g. to use as a summary. Starting at the beginning of the paragraph it gets as many complete sentences as are necessary to contain $limit words. For example, with $limit at 20 it would return the first two sentences of the paragraph you're reading right now (the first 20 words plus the rest of the sentence in which the limit was hit).
function summarize($paragraph, $limit){
$tok = strtok($paragraph, " ");
while($tok){
$text .= " $tok";
$words++;
if(($words >= $limit) && ((substr($tok, -1) == "!")||(substr($tok, -1) == ".")))
break;
$tok = strtok(" ");
}
return ltrim($text);
}
Might be a better way to do this, but it worked for me. Hope you find it useful!

desolate19 at NOSPAM dot hotmail dot com (2002-05-02 15:28:52)

Here is yet another explanation of strtok for the explode/split comments.

You can do things with strtok that you can't do with explode/split. explode breaks a string using another string, split breaks a string using a regular expression.  strtok breaks a string using single _characters_ , but the best part is you can use multiple characters at the same time.

For example, if you are accepting user input and aren't sure how the user will decide to divide up their data you could choose to tokenize on spaces, hyphens, slashes and backslashes ALL AT THE SAME TIME:

<?PHP

$teststr = "blah1 blah2/blah3-blah4\\blah5";

$tok = strtok($teststr," /-\\");
while ($tok !== FALSE)
{
  $toks[] = $tok;
  $tok = strtok(" /-\\");
}

while (list($k,$v) = each($toks))
{
  print ("$k => $v&lt;BR&gt;\n");
}

?>

/* OUTPUT:
0 => blah1
1 => blah2
2 => blah3
3 => blah4
4 => blah5
*/

You can't do that with explode, and this should be faster than using split because split uses regular expressions.

And for the comments about explode/split putting your output into an array... as you can see, it's not hard to work with arrays in PHP.

(2001-12-11 12:57:43)

The example is unnecessarily confusing for beginners.
1) It is NOT strtok that fails when the returned string evaluates
to false in conditional expression, it is the loop test. A correct test is
while($tok !== false)
2) the same functionality (AS THE EXAMPLE) can be obtained with
explode. Note that if you only need the first few tokens you can
put a limit on explode!! read the manual :)
array explode (string separator, string string [, INT LIMIT])
What you can NOT do with explode (or split) is changing the
separator after a token is returned, as for example, when parsing a
string along a simple format :
$styleStr = "color:#FFFFFF;font-size:12";
$key = strtok($styleStr,":");
while ($key !== false){
$styleTab[$key]= strtok(";"); // do not change the target
$key = strtok(":"); // string, just the separator list
}
$styleTab is array("color"=>"#FFFFFF","font-size"=>"12")
If you need the remaining of the string do :
$remaining = strtok(''); //(empty separator)
Ivan

slt at municipium dot com dot pl (2001-02-14 11:12:47)

As 'mckay' wrote, strtok 2nd argument is a list of tokens, not a string delimiter. It's not so obvious as one may think and it may be confusing for beginners like me. So, in the docs, it should state sth. like that
strtok(string where2search, char token2cut).
And for the above split-lover =) 'tysonlt' -> it's better to use explode bcoz it's lighter than split (to quote original manual: "(...) use explode(), which doesn't incur the overhead of the regular expression engine")
regards,
StarLight

tysonlt at spamless dot webmedia dot com dot au (2000-10-03 15:34:59)

Why use strtok at all?
If it's so flaky, why not just use split?
eg.
$token_array = split("$delim", $string);
Then you can use all the nice array functions on it! :)

David dot Mazur at student dot kuleuven dot ac dot be (2000-08-16 04:40:40)

If you want to tokenize only part of the string, and store the "untokenized" part in some
variable, you have to call strtok one last time with separator "" (i.e. the empty string).