stackoverflow.

com /questions/4094699/how-does-the-windows-command-interpreter-cmd-exe-parse-scripts/4095133

This question shows research effort; it is useful and clear

187

This question does not show any research effort; it is unclear or not useful

Save this question.

Show activity on this post.

I ran into ss64.com which provides good help regarding how to write batch scripts that the Windows
Command Interpreter will run.

However, I have been unable to find a good explanation of the grammar of batch scripts, how things expand
or do not expand, and how to escape things.

Here are sample questions that I have not been able to solve:

How is the quote system managed? I made a TinyPerl script

( foreach $i (@ARGV) { print '*' . $i ; } ), compiled it and called it this way :
my_script.exe "a ""b"" c" → output is *a "b*c
my_script.exe """a b c""" → output it *"a*b*c"
How does the internal echo command work? What is expanded inside that command?
Why do I have to use for [...] %%I in file scripts, but for [...] %I in interactive sessions?
What are the escape characters, and in what context? How to escape a percent sign? For example,
how can I echo %PROCESSOR_ARCHITECTURE% literally? I found that echo.exe
%""PROCESSOR_ARCHITECTURE% works, is there a better solution?
How do pairs of % match? Example:
set b=a , echo %a %b% c% → %a a c%
set a =b, echo %a %b% c% → bb% c%
How do I ensure a variable passes to a command as a single argument if ever this variable contains
double quotes?
How are variables stored when using the set command? For example, if I do set a=a" b and then
echo.%a% I obtain a" b. If I however use echo.exe from the UnxUtils, I get a b. How comes %a%
expands in a different way?

1/31
 Rob van der Woude has an awesome Batch scripting and Windows Command prompt reference on
his site.
– JBRWilkinson
Nov 4, 2010 at 10:50

Add a comment |

8 Answers 8
Reset to default

This answer is useful

272

This answer is not useful

Save this answer.

Show activity on this post.

We performed experiments to investigate the grammar of batch scripts. We also investigated differences
between batch and command line mode.

Batch Line Parser:

Here is a brief overview of phases in the batch file line parser:

Phase 0) Read Line:

Phase 1) Percent Expansion:

Phase 2) Process special characters, tokenize, and build a cached command block: This is a complex
process that is affected by things such as quotes, special characters, token delimiters, and caret escapes.

Phase 3) Echo the parsed command(s) Only if the command block did not begin with @, and ECHO was
ON at the start of the preceding step.

Phase 4) FOR %X variable expansion: Only if a FOR command is active and the commands after DO are
being processed.

Phase 5) Delayed Expansion: Only if delayed expansion is enabled

Phase 5.3) Pipe processing: Only if commands are on either side of a pipe

2/31
Phase 5.5) Execute Redirection:

Phase 6) CALL processing/Caret doubling: Only if the command token is CALL

Phase 7) Execute: The command is executed

Here are details for each phase:

Note that the phases described below are only a model of how the batch parser works. The actual cmd.exe
internals may not reflect these phases. But this model is effective at predicting behavior of batch scripts.

Phase 0) Read Line: Read line of input through first <LF>.

When reading a line to be parsed as a command, <Ctrl-Z> (0x1A) is read as <LF> (LineFeed 0x0A)
When GOTO or CALL reads lines while scanning for a :label, <Ctrl-Z>, is treated as itself - it is not
converted to <LF>

Phase 1) Percent Expansion:

A double %% is replaced by a single %

Expansion of arguments (%*, %1, %2, etc.)
Expansion of %var%, if var does not exist replace it with nothing
Line is truncated at first <LF> not within %var% expansion
For a complete explanation read the first half of this from dbenham Same thread: Percent Phase

Phase 2) Process special characters, tokenize, and build a cached command block: This is a complex
process that is affected by things such as quotes, special characters, token delimiters, and caret escapes.
What follows is an approximation of this process.

There are concepts that are important throughout this phase.

A token is simply a string of characters that is treated as a unit.

Tokens are separated by token delimiters. The standard token delimiters are <space> <tab> ; , =
<0x0B> <0x0C> and <0xFF>
Consecutive token delimiters are treated as one - there are no empty tokens between token delimiters
There are no token delimiters within a quoted string. The entire quoted string is always treated as part
of a single token. A single token may consist of a combination of quoted strings and unquoted
characters.

The following characters may have special meaning in this phase, depending on context: <CR> ^ ( @ & | < >
<LF> <space> <tab> ; , = <0x0B> <0x0C> <0xFF>

Look at each character from left to right:

3/31
If <CR> then remove it, as if it were never there (except for weird redirection behavior)
If a caret (^), the next character is escaped, and the escaping caret is removed. Escaped characters
lose all special meaning (except for <LF>).
If a quote ("), toggle the quote flag. If the quote flag is active, then only " and <LF> are special. All
other characters lose their special meaning until the next quote toggles the quote flag off. It is not
possible to escape the closing quote. All quoted characters are always within the same token.
<LF> always turns off the quote flag. Other behaviors vary depending on context, but quotes never
alter the behavior of <LF>.
Escaped <LF>
<LF> is stripped
The next character is escaped. If at the end of line buffer, then the next line is read and
processed by phases 1 and 1.5 and appended to the current one before escaping the next
character. If the next character is <LF>, then it is treated as a literal, meaning this process
is not recursive.
Unescaped <LF> not within parentheses
<LF> is stripped and parsing of the current line is terminated.
Any remaining characters in the line buffer are simply ignored.
Unescaped <LF> within a FOR IN parenthesized block
<LF> is converted into a <space>
If at the end of the line buffer, then the next line is read and appended to the current one.
Unescaped <LF> within a parenthesized command block
<LF> is converted into <LF><space>, and the <space> is treated as part of the next line
of the command block.
If at the end of line buffer, then the next line is read and appended to the space.
If one of the special characters & | < or >, split the line at this point in order to handle pipes, command
concatenation, and redirection.
In the case of a pipe (|), each side is a separate command (or command block) that gets special
handling in phase 5.3
In the case of &, &&, or || command concatenation, each side of the concatenation is treated as
a separate command.
In the case of <, <<, >, or >> redirection, the redirection clause is parsed, temporarily removed,
and then appended to the end of the current command. A redirection clause consists of an
optional file handle digit, the redirection operator, and the redirection destination token.
If the token that precedes the redirection operator is a single unescaped digit, then the
digit specifies the file handle to be redirected. If the handle token is not found, then output
redirection defaults to 1 (stdout), and input redirection defaults to 0 (stdin).
If the very first token for this command (prior to moving redirection to the end) begins with @, then the @
has special meaning. (@ is not special in any other context)
The special @ is removed.

4/31
 If ECHO is ON, then this command, along with any following concatenated commands on this
line, are excluded from the phase 3 echo. If the @ is before an opening (, then the entire
parenthesized block is excluded from the phase 3 echo.
Process parenthesis (provides for compound statements across multiple lines):
If the parser is not looking for a command token, then ( is not special.
If the parser is looking for a command token and finds (, then start a new compound statement
and increment the parenthesis counter
If the parenthesis counter is > 0 then ) terminates the compound statement and decrements the
parenthesis counter.
If the line end is reached and the parenthesis counter is > 0 then the next line will be appended
to the compound statement (starts again with phase 0)
If the parenthesis counter is 0 and the parser is looking for a command, then ) functions similar
to a REM statement as long as it is immediately followed by a token delimiter, special character,
newline, or end-of-file
All special characters lose their meaning except ^ (line concatenation is possible)
Once the end of the logical line is reached, the entire "command" is discarded.
Each command is parsed into a series of tokens. The first token is always treated as a command token
(after special @ have been stripped and redirection moved to the end).
Leading token delimiters prior to the command token are stripped
When parsing the command token, ( functions as a command token delimiter, in addition to the
standard token delimiters
The handling of subsequent tokens depends on the command.
Most commands simply concatenate all arguments after the command token into a single argument
token. All argument token delimiters are preserved. Argument options are typically not parsed until
phase 7.
Three commands get special handling - IF, FOR, and REM
IF is split into two or three distinct parts that are processed independently. A syntax error in the
IF construction will result in a fatal syntax error.
The comparison operation is the actual command that flows all the way through to phase 7
All IF options are fully parsed in phase 2.
Consecutive token delimiters collapse into a single space.
Depending on the comparison operator, there will be one or two value tokens that
are identified.
The True command block is the set of commands after the condition, and is parsed like
any other command block. If ELSE is to be used, then the True block must be
parenthesized.
The optional False command block is the set of commands after ELSE. Again, this
command block is parsed normally.
The True and False command blocks do not automatically flow into the subsequent
phases. Their subsequent processing is controled by phase 7.

5/31
 FOR is split in two after the DO. A syntax error in the FOR construction will result in a fatal
syntax error.
The portion through DO is the actual FOR iteration command that flows all the way
through phase 7
All FOR options are fully parsed in phase 2.
The IN parenthesized clause treats <LF> as <space>. After the IN clause is parsed,
all tokens are concatenated together to form a single token.
Consecutive unescaped/unquoted token delimiters collapse into a single space
throughout the FOR command through DO.
The portion after DO is a command block that is parsed normally. Subsequent processing
of the DO command block is controled by the iteration in phase 7.
REM detected in phase 2 is treated dramatically different than all other commands.
Only one argument token is parsed - the parser ignores characters after the first argument
token.
The REM command may appear in phase 3 output, but the command is never executed,
and the original argument text is echoed - escaping carets are not removed, except...
If there is only one argument token that ends with an unescaped ^ that ends the line,
then the argument token is thrown away, and the subsequent line is parsed and
appended to the REM. This repeats until there is more than one token, or the last
character is not ^.
If the command token begins with :, and this is the first round of phase 2 (not a restart due to CALL in
phase 6) then
The token is normally treated as an Unexecuted Label.
The remainder of the line is parsed, however ), <, >, & and | no longer have special
meaning. The entire remainder of the line is considered to be part of the label "command".
The ^ continues to be special, meaning that line continuation can be used to append the
subsequent line to the label.
An Unexecuted Label within a parenthesized block will result in a fatal syntax error unless
it is immediately followed by a command or Executed Label on the next line.
( no longer has special meaning for the first command that follows the Unexecuted
Label.
The command is aborted after label parsing is complete. Subsequent phases do not take
place for the label
There are three exceptions that can cause a label found in phase 2 to be treated as an Executed
Label that continues parsing through phase 7.
There is redirection that precedes the label token, and there is a | pipe or &, &&, or ||
command concatenation on the line.
There is redirection that precedes the label token, and the command is within a
parenthesized block.
The label token is the very first command on a line within a parenthesized block, and the
line above ended with an Unexecuted Label.

6/31
 The following occurs when an Executed Label is discovered in phase 2
The label, its arguments, and its redirection are all excluded from any echo output in phase
3
Any subsequent concatenated commands on the line are fully parsed and executed.
For more information about Executed Labels vs. Unexecuted Labels, see
https://www.dostips.com/forum/viewtopic.php?f=3&t=3803&p=55405#p55405

Phase 3) Echo the parsed command(s) Only if the command block did not begin with @, and ECHO was
ON at the start of the preceding step.

Phase 4) FOR %X variable expansion: Only if a FOR command is active and the commands after DO are
being processed.

At this point, phase 1 of batch processing will have already converted a FOR variable like %%X into %X.
The command line has different percent expansion rules for phase 1. This is the reason that command
lines use %X but batch files use %%X for FOR variables.
FOR variable names are case sensitive, but ~modifiers are not case sensitive.
~modifiers take precedence over variable names. If a character following ~ is both a modifier and a
valid FOR variable name, and there exists a subsequent character that is an active FOR variable
name, then the character is interpreted as a modifier.
FOR variable names are global, but only within the context of a DO clause. If a routine is CALLed from
within a FOR DO clause, then the FOR variables are not expanded within the CALLed routine. But if
the routine has its own FOR command, then all currently defined FOR variables are accessible to the
inner DO commands.
FOR variable names can be reused within nested FORs. The inner FOR value takes precedence, but
once the INNER FOR closes, then the outer FOR value is restored.
If ECHO was ON at the start of this phase, then phase 3) is repeated to show the parsed DO
commands after the FOR variables have been expanded.

---- From this point onward, each command identified in phase 2 is processed separately.
---- Phases 5 through 7 are completed for one command before moving on to the next.

Phase 5) Delayed Expansion: Only if delayed expansion is on, the command is not in a parenthesized
block on either side of a pipe, and the command is not a "naked" batch script (script name without
parentheses, CALL, command concatenation, or pipe).

Each token for a command is parsed for delayed expansion independently.

Most commands parse two or more tokens - the command token, the arguments token, and
each redirection destination token.
The FOR command parses the IN clause token only.
The IF command parses the comparison values only - either one or two, depending on the
comparison operator.

7/31
 For each parsed token, first check if it contains any !. If not, then the token is not parsed - important
for ^ characters. If the token does contain !, then scan each character from left to right:
If it is a caret (^) the next character has no special meaning, the caret itself is removed
If it is an exclamation mark, search for the next exclamation mark (carets are not observed
anymore), expand to the value of the variable.
Consecutive opening ! are collapsed into a single !
Any remaining unpaired ! is removed
Expanding vars at this stage is "safe", because special characters are not detected anymore
(even <CR> or <LF>)
For a more complete explanation, read the 2nd half of this from dbenham same thread -
Exclamation Point Phase

Phase 5.3) Pipe processing: Only if commands are on either side of a pipe
Each side of the pipe is processed independently and asynchronously.

If command is internal to cmd.exe, or it is a batch file, or if it is a parenthesized command block, then it

is executed in a new cmd.exe thread via %comspec% /S /D /c" commandBlock", so the
command block gets a phase restart, but this time in command line mode.
If a parenthesized command block, then all <LF> with a command before and after are
converted to <space>&. Other <LF> are stripped.
This is the end of processing for the pipe commands.
See Why does delayed expansion fail when inside a piped block of code? for more about pipe parsing
and processing

Phase 5.5) Execute Redirection: Any redirection that was discovered in phase 2 is now executed.

The results of phases 4 and 5 can impact the redirection that was discovered in phase 2.
If the redirection fails, then the remainder of the command is aborted. Note that failed redirection does
not set ERRORLEVEL to 1 unless || is used.

Phase 6) CALL processing/Caret doubling: Only if the command token is CALL, or if the text before the
first occurring standard token delimiter is CALL. If CALL is parsed from a larger command token, then the
unused portion is prepended to the arguments token before proceeding.

Scan the arguments token for an unquoted /?. If found anywhere within the tokens, then abort phase
6 and proceed to Phase 7, where the HELP for CALL will be printed.
Remove the first CALL, so multiple CALL's can be stacked
Double all carets
Restart phases 1, 1.5, and 2, but do not continue to phase 3
Any doubled carets are reduced back to one caret as long as they are not quoted. But
unfortunately, quoted carets remain doubled.

8/31
 Phase 1 changes a bit - Expansion errors in step 1.2 or 1.3 abort the CALL, but the error is not
fatal - batch processing continues.
Phase 2 tasks are altered a bit
Any newly appearing unquoted, unescaped redirection that was not detected in the first
round of phase 2 is detected, but it is removed (including the file name) without actually
performing the redirection
Any newly appearing unquoted, unescaped caret at the end of the line is removed without
performing line continuation
The CALL is aborted without error if any of the following are detected
Newly appearing unquoted, unescaped & or |
The resultant command token begins with unquoted, unescaped (
The very first token after the removed CALL began with @
If the resultant command is a seemingly valid IF or FOR, then execution will subsequently
fail with an error stating that IF or FOR is not recognized as an internal or external
command.
Of course the CALL is not aborted in this 2nd round of phase 2 if the resultant command
token is a label beginning with :.
If the resultant command token is CALL, then restart Phase 6 (repeats until no more CALL)
If the resultant command token is a batch script or a :label, then execution of the CALL is fully handled
by the remainder of Phase 6.
Push the current batch script file position on the call stack so that execution can resume from the
correct position when the CALL is completed.
Setup the %0, %1, %2, ...%N and %* argument tokens for the CALL, using all resultant tokens
If the command token is a label that begins with :, then
Restart Phase 5. This can impact what :label is CALLed. But since the %0 etc. tokens
have already been setup, it will not alter the arguments that are passed to the CALLed
routine.
Execute GOTO label to position the file pointer at the beginning of the subroutine (ignore
any other tokens that may follow the :label) See Phase 7 for rules on how GOTO works.
If the :label token is missing, or the :label is not found, then the call stack is
immediately popped to restore the saved file position, and the CALL is aborted.
If the :label happens to contain /?, then GOTO help is printed instead of searching
for the :label. The file pointer does not move, such that code after the CALL is
executed twice, once in the CALL context, and then again after the CALL return. See
Why CALL prints the GOTO help message in this script?And why command after
that are executed twice? for more info.
Else transfer control to the specified batch script.
Execution of the CALLed :label or script continues until either EXIT /B or end-of-file is reached,
at which point the CALL stack is popped and execution resumes from the saved file position.
Phase 7 is not executed for CALLed scripts or :labels.
Else the result of phase 6 falls through into phase 7 for execution.

9/31
Phase 7) Execute: The command is executed

7.1 - Execute internal command - If the command token is quoted, then skip this step. Otherwise,
attempt to parse out an internal command and execute.
The following tests are made to determine if an unquoted command token represents an internal
command:
If the command token exactly matches an internal command, then execute it.
Else break the command token before the first occurrence of + / [ ] <space> <tab> , ;
or =
If the preceding text is an internal command, then remember that command
If in command line mode, or if the command is from a parenthesized block, IF true or
false command block, FOR DO command block, or involved with command
concatenation, then execute the internal command
Else (must be a stand-alone command in batch mode) scan the current folder and
the PATH for a .COM, .EXE, .BAT, or .CMD file whose base name matches the
original command token
If the first matching file is a .BAT or .CMD, then goto 7.3.exec and execute that
script
Else (match not found or first match is .EXE or .COM) execute the
remembered internal command
Else break the command token before the first occurrence of . \ or :
If the preceding text is not an internal command, then goto 7.2
Else the preceding text may be an internal command. Remember this command.
Break the command token before the first occurrence of + / [ ] <space> <tab> , ; or =
If the preceding text is a path to an existing file, then goto 7.2
Else execute the remembered internal command.
If an internal command is parsed from a larger command token, then the unused portion of the
command token is included in the argument list
Just because a command token is parsed as an internal command does not mean that it will
execute successfully. Each internal command has its own rules as to how the arguments and
options are parsed, and what syntax is allowed.
All internal commands will print help instead of performing their function if /? is detected. Most
recognize /? if it appears anywhere in the arguments. But a few commands like ECHO and SET
only print help if the first argument token begins with /?.
SET has some interesting semantics:
If a SET command has a quote before the variable name and extensions are enabled
set "name=content" ignored --> value=content
then the text between the first equal sign and the last quote is used as the content (first
equal and last quote excluded). Text after the last quote is ignored. If there is no quote
after the equal sign, then the rest of the line is used as content.

10/31
 If a SET command does not have a quote before the name
set name="content" not ignored --> value="content" not ignored
then the entire remainder of the line after the equal is used as content, including any and
all quotes that may be present.
An IF comparison is evaluated, and depending on whether the condition is true or false, the
appropriate already parsed dependent command block is processed, starting with phase 5.
The IN clause of a FOR command is iterated appropriately.
If this is a FOR /F that iterates the output of a command block, then:
The IN clause is executed in a new cmd.exe process via CMD /C.
The command block must go through the entire parsing process a second time, but
this time in a command line context
ECHO will start out ON, and delayed expansion will usually start out disabled
(dependent on the registry setting)
All environment changes made by the IN clause command block will be lost once the
child cmd.exe process terminates
For each iteration:
The FOR variable values are defined
The already parsed DO command block is then processed, starting with phase 4.
GOTO uses the following logic to locate the :label
Parse the label from the first argument token
Scan for the next occurrence of the label
Start from the current file position
If end of file is reached, then loop back to the beginning of file and continue to the
original starting point.
The scan stops at the first occurrence of the label that it finds, and the file pointer is set to
the line immediately following the label. Execution of the script resumes from that point.
Note that a successful true GOTO will immediately abort any parsed block of code,
including FOR loops.
If the label is not found, or the label token is missing, then the GOTO fails, an error
message is printed, and the call stack is popped. This effectively functions as an EXIT /B,
except any already parsed commands in the current command block that follow the GOTO
are still executed, but in the context of the CALLer (the context that exists after EXIT /B)
See https://www.dostips.com/forum/viewtopic.php?t=3803 for a more precise description
of label parsing rules, and https://www.dostips.com/forum/viewtopic.php?t=8988 for label
scanning rules.
RENAME and COPY both accept wildcards for the source and target paths. But Microsoft does a
terrible job documenting how the wildcards work, especially for the target path. A useful set of
wildcard rules may be found at How does the Windows RENAME command interpret wildcards?
7.2 - Execute volume change - Else if the command token does not begin with a quote, is exactly two
characters long, and the 2nd character is a colon, then change the volume
All argument tokens are ignored

11/31
 If the volume specified by the first character cannot be found, then abort with an error
A command token of :: will always result in an error unless SUBST is used to define a volume
for ::
If SUBST is used to define a volume for ::, then the volume will be changed, it will not be
treated as a label.
7.3 - Execute external command - Else try to treat the command as an external command.
If in command line mode and the command is not quoted and does not begin with a volume
specification, white-space, ,, ;, = or + then break the command token at the first occurrence of
<space> , ; or = and prepend the remainder to the argument token(s).
If the 2nd character of the command token is a colon, then verify the volume specified by the 1st
character can be found.
If the volume cannot be found, then abort with an error.
If in batch mode and the command token begins with :, then goto 7.4
Note that if the label token begins with ::, then this will not be reached because the preceding
step will have aborted with an error unless SUBST is used to define a volume for ::.
Identify the external command to execute.
This is a complex process that may involve the current volume, current directory, PATH
variable, PATHEXT variable, and or file associations.
If a valid external command cannot be identified, then abort with an error.
If in command line mode and the command token begins with :, then goto 7.4
Note that this is rarely reached because the preceding step will have aborted with an error
unless the command token begins with ::, and SUBST is used to define a volume for ::, and
the entire command token is a valid path to an external command.
7.3.exec - Execute the external command.
7.4 - Ignore a label - Ignore the command and all its arguments if the command token begins with :.
Rules in 7.2 and 7.3 may prevent a label from reaching this point.

Command Line Parser:

Works like the BatchLine-Parser, except:

Phase 1) Percent Expansion:

No %*, %1 etc. argument expansion

If var is undefined, then %var% is left unchanged.
No special handling of %%. If var=content, then %%var%% expands to %content%.

Phase 3) Echo the parsed command(s)

This is not performed after phase 2. It is only performed after phase 4 for the FOR DO command
block.

12/31
Phase 5) Delayed Expansion: only if DelayedExpansion is enabled

If var is undefined, then !var! is left unchanged.

Phase 7) Execute Command

Attempts to CALL or GOTO a :label result in an error.

As already documented in phase 7, an executed label may result in an error under different scenarios.
Batch executed labels can only cause an error if they begin with ::
Command line executed labels almost always result in an error

Parsing of integer values

There are many different contexts where cmd.exe parses integer values from strings, and the rules are
inconsistent:

SET /A
IF
%var:~n,m% (variable substring expansion)
FOR /F "TOKENS=n"
FOR /F "SKIP=n"
FOR /L %%A in (n1 n2 n3)
EXIT [/B] n

Details for these rules may be found at Rules for how CMD.EXE parses numbers

For anyone wishing to improve the cmd.exe parsing rules, there is a discussion topic on the DosTips forum
where issues can be reported and suggestions made.

Jan Erik (jeb) - Original author and discoverer of phases

Dave Benham (dbenham) - Much additional content and editing

30

Hello jeb, thank you for your insight… It might be hard to understand, but I will try to think it through!
You seem to have performed much tests! Thank you for translating (administrator.de/…)
– Benoit
Nov 4, 2010 at 9:19

13/31
 Batch phase 5) - %%a will have already been changed to %a in Phase 1, so for-loop expansion really
expands %a. Also, I added a more detailed explanation of Batch phase 1 in an answer below (I don't
have edit privilege)
– dbenham
Nov 1, 2011 at 18:26

Jeb - perhaps phase 0 could be moved and combined with phase 6? That makes more sense to me,
or is there a reason why they are separated like that?
– dbenham
Mar 19, 2012 at 14:00

@aschipfl - I updated that section. The ) really does function almost like a REM command when the
parenthesis counter is 0. Try both of these from the command line: ) Ignore this, and echo OK &
) Ignore this
– dbenham
Jun 29, 2016 at 15:29

@aschipfl yes that's correct, therfore you see sometimes 'set "var=%expr%" ! ' the last exclamation
mark will be removed but forces phase 5
– jeb
Jul 18, 2016 at 12:56

| Show 25 more comments

This answer is useful

65

This answer is not useful

Save this answer.

Show activity on this post.

When invoking a command from a command window, tokenization of the command line arguments is not
done by cmd.exe (a.k.a. "the shell"). Most often the tokenization is done by the newly formed processes'
C/C++ runtime, but this is not necessarily so -- for example, if the new process was not written in C/C++, or if

14/31
the new process chooses to ignore argv and process the raw commandline for itself (e.g. with
GetCommandLine()). At the OS level, Windows passes command lines untokenized as a single string to new
processes. This is in contrast to most *nix shells, where the shell tokenizes arguments in a consistent,
predictable way before passing them to the newly formed process. All this means that you may experience
wildly divergent argument tokenization behavior across different programs on Windows, as individual
programs often take argument tokenization into their own hands.

If it sounds like anarchy, it kind of is. However, since a large number of Windows programs do utilize the
Microsoft C/C++ runtime's argv, it may be generally useful to understand how the MSVCRT tokenizes
arguments. Here is an excerpt:

Arguments are delimited by white space, which is either a space or a tab.

A string surrounded by double quotation marks is interpreted as a single argument, regardless of white
space contained within. A quoted string can be embedded in an argument. Note that the caret (^) is
not recognized as an escape character or delimiter.
A double quotation mark preceded by a backslash, \", is interpreted as a literal double quotation mark
(").
Backslashes are interpreted literally, unless they immediately precede a double quotation mark.
If an even number of backslashes is followed by a double quotation mark, then one backslash () is
placed in the argv array for every pair of backslashes (\), and the double quotation mark (") is
interpreted as a string delimiter.
If an odd number of backslashes is followed by a double quotation mark, then one backslash () is
placed in the argv array for every pair of backslashes (\) and the double quotation mark is interpreted
as an escape sequence by the remaining backslash, causing a literal double quotation mark (") to be
placed in argv.

The Microsoft "batch language" (.bat) is no exception to this anarchic environment, and it has developed its
own unique rules for tokenization and escaping. It also looks like cmd.exe's command prompt does do some
preprocessing of the command line argument (mostly for variable substitution and escaping) before passing
the argument off to the newly executing process. You can read more about the low-level details of the batch
language and cmd escaping in the excellent answers by jeb and dbenham on this page.

Let's build a simple command line utility in C and see what it says about your test cases:

int main(int argc, char* argv[]) {

int i;
for (i = 0; i < argc; i++) {
printf("argv[%d][%s]\n", i, argv[i]);
}
return 0;
}

15/31
(Notes: argv[0] is always the name of the executable, and is omitted below for brevity. Tested on Windows
XP SP3. Compiled with Visual Studio 2005.)

> test.exe "a ""b"" c"

argv[1][a "b" c]

> test.exe """a b c"""

argv[1]["a b c"]

> test.exe "a"" b c

argv[1][a" b c]

And a few of my own tests:

> test.exe a "b" c

argv[1][a]
argv[2][b]
argv[3][c]

> test.exe a "b c" "d e

argv[1][a]
argv[2][b c]
argv[3][d e]

> test.exe a \"b\" c

argv[1][a]
argv[2]["b"]
argv[3][c]

Thank you for your answer. It puzzles me even more to see that TinyPerl will not output what your
program outputs, and I have difficulties to understand how [a "b" c] could become [a "b] [c]
doing post-processing.
– Benoit
Nov 4, 2010 at 8:56

Now that I think about it, this tokenization of the command line is probably done entirely by the C
runtime. An executable could be written such that it doesn't even use the C runtime, in which case I
think it would have to deal with the command line verbatim, and be responsible for doing its own
tokenization (if it wanted to.) Or even if your application does use the C runtime, you could choose to

16/31
 ignore argc and argv and just get the raw command line via e.g. Win32 GetCommandLine. Perhaps
TinyPerl is ignoring argv and simply tokenizing the raw command line by its own rules.
– Mike Clark
Nov 4, 2010 at 9:46

"Remember that from Win32's point of view, the command line is just a string that is copied into the
address space of the new process. How the launching process and the new process interpret this
string is governed not by rules but by convention." -Raymond Chen
blogs.msdn.com/b/oldnewthing/archive/2009/11/25/9928372.aspx
– Mike Clark
Nov 4, 2010 at 9:51

Thank you for that truly nice answer. That explains a lot in my opinion. And that also explains why I
sometimes find that truly crappy to work with Windows…
– Benoit
Nov 4, 2010 at 14:19

This is great information, but the Microsoft documentation is incomplete! (big surprise) The actual
missing rules are documented at
daviddeley.com/autohotkey/parameters/parameters.htm#WINCRULES.
– dbenham
Feb 18, 2019 at 19:53

| Show 2 more comments

This answer is useful

57

This answer is not useful

Save this answer.

Show activity on this post.

Percent Expansion Rules

17/31
Here is an expanded explanation of Phase 1 in jeb's answer (valid for both batch mode and command line
mode).

Phase 1) Percent Expansion Starting from left, scan each character for % or <LF>. If found then

1.05 (truncate line at <LF>)

If the character is <LF> then
Drop (ignore) the remainder of the line from the <LF> onward
Goto Phase 2.0
Else the character must be %, so proceed to 1.1
1.1 (escape %) skipped if command line mode
If batch mode and followed by another % then
Replace %% with single % and continue scan
1.2 (expand argument) skipped if command line mode
Else if batch mode then
If followed by * and command extensions are enabled then
Replace %* with the text of all command line arguments (Replace with nothing if there are no
arguments) and continue scan.
Else if followed by <digit> then
Replace %<digit> with argument value (replace with nothing if undefined) and continue scan.
Else if followed by ~ and command extensions are enabled then
If followed by optional valid list of argument modifiers followed by required <digit> then
Replace %~[modifiers]<digit> with modified argument value (replace with nothing if
not defined or if specified $PATH: modifier is not defined) and continue scan.
Note: modifiers are case insensitive and can appear multiple times in any order, except
$PATH: modifier can only appear once and must be the last modifier before the <digit>
Else invalid modified argument syntax raises fatal error: All parsed commands are
aborted, and batch processing aborts if in batch mode!
1.3 (expand variable)
Else if command extensions are disabled then
Look at next string of characters, breaking before % or end of buffer, and call them VAR (may be an
empty list)
If next character is % then
If VAR is defined then
Replace %VAR% with value of VAR and continue scan
Else if batch mode then
Remove %VAR% and continue scan
Else goto 1.4
Else goto 1.4

18/31
 Else if command extensions are enabled then
Look at next string of characters, breaking before % : or end of buffer, and call them VAR (may be an
empty list). If VAR breaks before : and the subsequent character is % then include : as the last
character in VAR and break before %.
If next character is % then
If VAR is defined then
Replace %VAR% with value of VAR and continue scan
Else if batch mode then
Remove %VAR% and continue scan
Else goto 1.4
Else if next character is : then
If VAR is undefined then
If batch mode then
Remove %VAR: and continue scan.
Else goto 1.4
Else if next character is ~ then
If next string of characters matches pattern of [integer][,[integer]]% then
Replace %VAR:~[integer][,[integer]]% with substring of value of VAR
(possibly resulting in empty string) and continue scan.
Else goto 1.4
Else if followed by = or *= then
Invalid variable search and replace syntax raises fatal error: All parsed commands are
aborted, and batch processing aborts if in batch mode!
Else if next string of characters matches pattern of [*]search=[replace]%, where
search may include any set of characters except =, and replace may include any set of
characters except %, then
Replace %VAR:[*]search=[replace]% with value of VAR after performing search and
replace (possibly resulting in empty string) and continue scan
Else goto 1.4
1.4 (strip %)
Else If batch mode then
Remove % and continue scan starting with the next character after the %
Else preserve the leading % and continue scan starting with the next character after the
preserved leading %

The above helps explain why this batch

@echo off
setlocal enableDelayedExpansion
set "1var=varA"

19/31
 set "~f1var=varB"
call :test "arg1"
exit /b
::
:test "arg1"
echo %%1var%% = %1var%
echo ^^^!1var^^^! = !1var!
echo --------
echo %%~f1var%% = %~f1var%
echo ^^^!~f1var^^^! = !~f1var!
exit /b

Gives these results:

%1var% = "arg1"var
!1var! = varA
--------
%~f1var% = P:\arg1var
!~f1var! = varB

Note 1 - Phase 1 occurs prior to the recognition of REM statements. This is very important because it means
even a remark can generate a fatal error if it has invalid argument expansion syntax or invalid variable
search and replace syntax!

@echo off
rem %~x This generates a fatal argument expansion error
echo this line is never reached

Note 2 - Another interesting consequence of the % parsing rules: Variables containing : in the name can be
defined, but they cannot be expanded unless command extensions are disabled. There is one exception - a
variable name containing a single colon at the end can be expanded while command extensions are
enabled. However, you cannot perform substring or search and replace operations on variable names
ending with a colon. The batch file below (courtesy of jeb) demonstrates this behavior

@echo off
setlocal
set var=content
set var:=Special
set var::=double colon
set var:~0,2=tricky
set var::~0,2=unfortunate
echo %var%

20/31
 echo %var:%
echo %var::%
echo %var:~0,2%
echo %var::~0,2%
echo Now with DisableExtensions
setlocal DisableExtensions
echo %var%
echo %var:%
echo %var::%
echo %var:~0,2%
echo %var::~0,2%

Note 3 - An interesting outcome of the order of the parsing rules that jeb lays out in his post: When
performing find and replace with delayed expansion, special characters in both the find and replace terms
must be escaped or quoted. But the situation is different for percent expansion - the find term must not be
escaped (though it can be quoted). The percent replace string may or may not require escape or quote,
depending on your intent.

@echo off
setlocal enableDelayedExpansion
set "var=this & that"
echo %var:&=and%
echo "%var:&=and%"
echo !var:^&=and!
echo "!var:&=and!"

Delayed Expansion Rules

Here is an expanded, and more accurate explanation of Phase 5 in jeb's answer (valid for both batch mode
and command line mode)

Phase 5) Delayed Expansion

This phase is skipped if any of the following conditions apply:

Delayed expansion is disabled.

The command is within a parenthesized block on either side of a pipe.
The incoming command token is a "naked" batch script, meaning it is not associated with CALL,
parenthesized block, any form of command concatenation (&, && or ||), or a pipe |.

The delayed expansion process is applied to tokens independently. A command may have multiple tokens:

21/31
 The command token. For most commands the command name itself is a token. But a few commands
have specialized regions that are considered a TOKEN for Phase 5.
for ... in(TOKEN) do
if defined TOKEN
if exists TOKEN
if errorlevel TOKEN
if cmdextversion TOKEN
if TOKEN comparison TOKEN, where comparison is one of ==, equ, neq, lss, leq, gtr, or
geq
The arguments token
The destination token of redirection (one per redirection)

No change is made to tokens that do not contain !.

For each token that does contain at least one !, scan each character from left to right for ^ or !, and if found,
then

5.1 (caret escape) Needed for ! or ^ literals

If character is a caret ^ then
Remove the ^
Scan the next character and preserve it as a literal
Continue the scan
5.2 (expand variable)
If character is !, then
If command extensions are disabled then
Look at next string of characters, breaking before ! or <LF>, and call them VAR (may be an
empty list)
If next character is ! then
If VAR is defined, then
Replace !VAR! with value of VAR and continue scan
Else if batch mode then
Remove !VAR! and continue scan
Else goto 5.2.1
Else goto 5.2.1
Else if command extensions are enabled then
Look at next string of characters, breaking before !, :, or <LF>, and call them VAR (may be an
empty list). If VAR breaks before : and the subsequent character is ! then include : as the last
character in VAR and break before !
If next character is ! then

22/31
 If VAR exists, then
Replace !VAR! with value of VAR and continue scan
Else if batch mode then
Remove !VAR! and continue scan
Else goto 5.2.1
Else if next character is : then
If VAR is undefined then
If batch mode then
Remove !VAR: and continue scan
Else goto 5.2.1
Else if next character is ~ then
If next string of characters matches pattern of [integer][,[integer]]!
then Replace !VAR:~[integer][,[integer]]! with substring of value of
VAR (possibly resulting in empty string) and continue scan.
Else goto 5.2.1
Else if next string of characters matches pattern of [*]search=[replace]!,
where search may include any set of characters except =, and replace may include
any set of characters except !, then
Replace !VAR:[*]search=[replace]! with value of VAR after performing
search and replace (possibly resulting in an empty string) and continue scan
Else goto 5.2.1
Else goto 5.2.1
5.2.1
If batch mode then remove the leading !
Else preserve the leading !
Continue the scan starting with the next character after the preserved leading !

edited Aug 5, 2022 at 2:53

30

+1, Only the colon syntax and rules are missing here for %definedVar:a=b% vs
%undefinedVar:a=b% and the %var:~0x17,-010% forms
– jeb
Nov 1, 2011 at 19:03

23/31
 After getting some additional private feedback from jeb, I added a rule for variable names ending with
colon, and added note 2. I also added note 3 simply because I thought it was interesting and
important.
– dbenham
Nov 3, 2011 at 18:31

@aschipfl - Yeah, I considered going into more detail about that, but didn't want to go down that rabbit
hole. I was intentionally non-committal when I used the term [integer].There is more info at Rules for
how does CMD.EXE parses numbers.
– dbenham
Jul 18, 2016 at 10:39

I'm missing the expansion rules for the cmd context, like that there are no reserved characters for the
first character of the variable name like %<digit>, %* or %~. And the behaviour changes for
undefined variables. Perhaps you need to open a second answer
– jeb
Mar 16, 2017 at 10:22

| Show 25 more comments

This answer is useful

This answer is not useful

Save this answer.

Show activity on this post.

As pointed out, commands are passed the entire argument string in μSoft land, and it is up to them to parse
this into separate arguments for their own use. There is no consistencty in this between different programs,
and therefore there is no one set of rules to describe this process. You really need to check each corner
case for whatever C library your program uses.

As far as the system .bat files go, here is that test:

c> type args.cmd

@echo off

24/31
 echo cmdcmdline:[%cmdcmdline%]
echo 0:[%0]
echo *:[%*]
set allargs=%*
if not defined allargs goto :eof
setlocal
@rem Wot about a nice for loop?
@rem Then we are in the land of delayedexpansion, !n!, call, etc.
@rem Plays havoc with args like %t%, a"b etc. ugh!
set n=1
:loop
echo %n%:[%1]
set /a n+=1
shift
set param=%1
if defined param goto :loop
endlocal

Now we can run some tests. See if you can figure out just what μSoft are trying to do:

C>args a b c
cmdcmdline:[cmd.exe ]
0:[args]
*:[a b c]
1:[a]
2:[b]
3:[c]

Fine so far. (I'll leave out the uninteresting %cmdcmdline% and %0 from now on.)

C>args *.*
*:[*.*]
1:[*.*]

No filename expansion.

C>args "a b" c

*:["a b" c]
1:["a b"]
2:[c]

No quote stripping, though quotes do prevent argument splitting.

25/31
 c>args ""a b" c
*:[""a b" c]
1:[""a]
2:[b" c]

Consecutive double quotes causes them to lose any special parsing abilities they may have had. @Beniot's
example:

C>args "a """ b "" c"""

*:["a """ b "" c"""]
1:["a """]
2:[b]
3:[""]
4:[c"""]

Quiz: How do you pass the value of any environment var as a single argument (i.e., as %1) to a bat file?

c>set t=a "b c

c>set t
t=a "b c
c>args %t%
1:[a]
2:["b c]
c>args "%t%"
1:["a "b]
2:[c"]
c>Aaaaaargh!

Sane parsing seems forever broken.

For your entertainment, try adding miscellaneous ^, \, ', & (&c.) characters to these examples.

edited Aug 5, 2014 at 18:48

@Toughy So, in my example, t is a "b c. Do you have a recipe for getting those 6 characters (a, 2 ×
space, ", b, and c) to appear as %1 inside a .cmd? I like your thinking though. args "%t:"=""%" is
pretty close :-)
– bobbogo
Oct 30, 2018 at 13:58

Add a comment |

26/31
This answer is useful

This answer is not useful

Save this answer.

Show activity on this post.

You have some great answers above already, but to answer one part of your question:

set a =b, echo %a %b% c% → bb c%

What is happening there is that because you have a space before the =, a variable is created called
%a<space>% so when you echo %a % that is evaluated correctly as b.

The remaining part b% c% is then evaluated as plain text + an undefined variable % c%, which should be
echoed as typed, for me echo %a %b% c% returns bb% c%

I suspect that the ability to include spaces in variable names is more of an oversight than a planned 'feature'

answered Aug 11, 2014 at 21:01

community wiki

SS64

Add a comment |

This answer is useful

This answer is not useful

Save this answer.

Show activity on this post.

27/31
FOR-Loop Meta-Variable Expansion
This is an extended explanation of Phase 4) in the accepted answer (applicable for both batch file mode and
command line mode). Of course a for command must be active. The following describes the processing of
the command line portion after the do clause. Note that in batch file mode, %% has already been converted to
% due to the foregoing immediate %-expansion phase (Phase 1)).

scan for %-sign, beginning from the left up to the end of the line; if one is found, then:
if Command Extensions are enabled (default), check if next character is ~; if yes, then:
take as many as possible of the following characters in the case-insensitive set
fdpnxsatz (even multiple times each) that are preceding a character that defines a for
variable reference or a $-sign; if such a $-sign is encountered, then:

scan for a :1; if found, then:

if there is a character following the :, use it as a for variable reference and
expand as expected, unless it is not defined, then do not expand and continue
scan at that character position;
if the : is the last character, cmd.exe will crash!
else (no : is found) do not expand anything;
else (if no $-sign is encountered) expand the for variable using all the modifiers, unless it
is not defined, then do not expand and continue scan at that character position;
else (if no ~ is found or Command Extensions are disabled) check the next character:
if there is no more character available, do not expand anything;
if the next character is %, do not expand anything and go back to the beginning of the scan
at this character position2;
else use the next character as a for variable reference and expand, unless such is not
defined, then do not expand;
go back to the beginning of the scan at the next character position (as long as there still characters
available);

1) The string between $ and : is considered as the name of an environment variable, which may even be empty; since an

environment variable cannot have an empty name, the behaviour is just the same as for an undefined environment variable.
2) This implies that a for meta-variable named % cannot be expanded without a ~-modifier.

Original source: How to safely echo FOR variable %%~p followed by a string literal

28/31
Add a comment |

This answer is useful

This answer is not useful

Save this answer.

Show activity on this post.

edit: see accepted answer, what follows is wrong and explains only how to pass a command line to TinyPerl.

Regarding quotes, I have the feeling that the behaviour is the following:

when a " is found, string globbing begins

when string globbing occurs:
every character that is not a " is globbed
when a " is found:
if it is followed by "" (thus a triple ") then a double quote is added to the string
if it is followed by " (thus a double ") then a double quote is added to the string and string
globbing ends
if the next character is not ", string globbing ends
when line ends, string globbing ends.

In short:

"a """ b "" c""" consists of two strings: a " b " and c"

"a"", "a""" and"a"""" are all the same string if at the end of a line

29/31
 cmd.exe passes always the expansion result as a string not the tokens to an external command. It
depends on the external command how to escape and tokenize it, findstr uses backslash the next one
can use something else
– jeb
Nov 4, 2010 at 9:55

Add a comment |

This answer is useful

-3

This answer is not useful

Save this answer.

Show activity on this post.

Note that Microsoft has published its Terminal's source code. It may work similar to the command line with
respect to syntax parsing. Maybe someone is interested in testing the reverse-engineered parsing rules on
accordance with the terminal's parsing rules.

Link to the source code.

answered Jun 8, 2020 at 18:00

community wiki

user7427029
1
Add a comment |

Highly active question. Earn 10 reputation (not counting the association bonus) in order to answer this
question. The reputation requirement helps protect this question from spam and non-answer activity.

Not the answer you're looking for? Browse other questions tagged

windows
parsing
batch-file
cmd

30/31
 variable-expansion

or ask your own question.

31/31