4122284075
git-svn-id: svn://bknr.net/svn/trunk/thirdparty/cl-ppcre@12 4281704c-cde7-0310-8518-8e2dc76b1ff0
1934 lines
85 KiB
HTML
1934 lines
85 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
|
|
<html>
|
|
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
|
|
<title>CL-PPCRE - portable Perl-compatible regular expressions for Common Lisp</title>
|
|
<style type="text/css">
|
|
pre { padding:5px; background-color:#e0e0e0 }
|
|
a.none { text-decoration: none; color:black }
|
|
a.none:visited { text-decoration: none; color:black }
|
|
a.none:active { text-decoration: none; color:black }
|
|
a.none:hover { text-decoration: none; color:black }
|
|
a { text-decoration: none; }
|
|
a:visited { text-decoration: none; }
|
|
a:active { text-decoration: underline; }
|
|
a:hover { text-decoration: underline; }
|
|
</style>
|
|
</head>
|
|
|
|
<body bgcolor=white>
|
|
|
|
<h2>CL-PPCRE - portable Perl-compatible regular expressions for Common Lisp</h2>
|
|
|
|
<blockquote>
|
|
<br> <br><h3>Abstract</h3>
|
|
|
|
CL-PPCRE is a portable regular expression library for Common Lisp
|
|
which has the following features:
|
|
|
|
<ul>
|
|
|
|
<li>It is <b>compatible with Perl</b>. (Well - as far as you can be
|
|
compatible with a language defined by its ever-changing
|
|
implementation. Currently, as of December 2002, CL-PPCRE is more
|
|
compatible with the regex semantics of Perl 5.8.0 than, say,
|
|
Perl 5.6.1 is...:) It even correctly parses and applies <a
|
|
href="http://www.oreilly.com/catalog/regex2/">Jeffrey Friedl's</a>
|
|
famous 6600-byte long RFC822 address pattern.
|
|
|
|
<li>It is <b>fast</b>. If compiled with <a
|
|
href="http://www.cons.org/cmucl/">CMUCL</a> it <a
|
|
href="#performance">outperforms</a> Perl's highly optimized regex engine (written
|
|
in C) which to my knowledge is faster than most other regex engines
|
|
around. If compiled with <a
|
|
href="http://clisp.sourceforge.net/">CLISP</a> it is still comparable
|
|
to CLISP's own regex implementation which is also written in
|
|
C.
|
|
|
|
<li>It is <b>portable</b>, i.e. the code aims to be strictly <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Front/index.htm">ANSI-compliant</a>. If
|
|
you encounter any deviations this is an error and should be
|
|
reported to <a
|
|
href="#mail">the mailing list</a>. CL-PPCRE has been
|
|
successfully tested with the following Common Lisp implementations:
|
|
|
|
<ul>
|
|
|
|
<li><a href="http://www.franz.com/products/allegrocl/">Allegro Common Lisp</a> (6.2 trial on Gentoo Linux 1.1a)
|
|
<li><a href="http://clisp.sourceforge.net/">CLISP</a> (2.30 on Gentoo Linux 1.1a and 2.29 on Windows XP pro)
|
|
<li><a href="http://www.cons.org/cmucl/">CMUCL</a> (18e on Gentoo Linux 1.1a)
|
|
<li><a href="http://www.cormanlisp.com/">Corman Lisp</a> (2.5 on Windows XP pro)
|
|
<li><a href="http://ecls.sourceforge.net/">ECL</a> (0.9c on Gentoo Linux 1.1a)
|
|
<li><a href="http://www.digitool.com/">Macintosh Common Lisp</a> (4.3 demo on MacOS 9.1 - only tested with CL-PPCRE 0.1.x)
|
|
<li><a href="http://openmcl.clozure.com/">OpenMCL</a> (0.13.4 on MacOS X 10.2.2 - only tested with CL-PPCRE 0.1.x)
|
|
<li><a href="http://sbcl.sourceforge.net/">SBCL</a> (0.8.4 on Gentoo Linux 1.1a)
|
|
<li><a href="http://www.scieneer.com/scl/">Scieneer Common Lisp</a> (1.1.1 evaluation on Gentoo Linux 1.1a - only tested with CL-PPCRE 0.1.x)
|
|
<li><a href="http://www.lispworks.com/">Xanalys LispWorks</a> (4.2.7 professional on Gentoo Linux 1.1a and 4.3.6 professional on Windows XP pro)
|
|
|
|
</ul>
|
|
|
|
If you succeed in using CL-PPCRE on other platforms please <a
|
|
href="#mail">let us know</a>.
|
|
|
|
<br>
|
|
Note that the tests mainly made sure that the package compiled
|
|
without errors and that the <a href="#test">test suite</a> - which
|
|
compiles about 1,500 regex strings into scanners and applies these to
|
|
target strings with the <a href="#scan"><code>SCAN</code></a> function
|
|
- yields the expected results. Other functions like <a
|
|
href="#split"><code>SPLIT</code></a>, <a
|
|
href="#all-matches"><code>ALL-MATCHES</code></a>, <a
|
|
href="#regex-replace"><code>REGEX-REPLACE</code></a>, <a
|
|
href="#regex-apropos"><code>REGEX-APROPOS</code></a>, or <a
|
|
href="#do-scans">the <code>DO</code>-macros</a> have only been tested
|
|
on CMUCL and LispWorks which were my main development platforms.
|
|
|
|
<br>Also, I don't have the time to re-test any implementation with
|
|
every new release of CL-PPCRE. <a href="#mail">Let us
|
|
know</a> if your implementation is listed above and fails with a
|
|
recent version and I'll try to fix it.
|
|
|
|
<li>It is <b>thread-safe</b>. Although the code uses closures
|
|
extensively, no state which dynamically changes during the scanning
|
|
process is stored in the lexical environments of the closures, so it
|
|
should be safe to use CL-PPCRE in a multi-threaded program. Tests with
|
|
LispWorks and Scieneer Common Lisp seem to confirm this.
|
|
|
|
<li>It comes with <b>convenient features</b> like a <a
|
|
href="#split"><code>SPLIT</code></a> function, a couple of <a
|
|
href="#do-scans"><code>DO</code>-like loop constructs</a>, and a <a
|
|
href="#regex-apropos"><code>regex-based APROPOS feature</code></a>
|
|
similar to the one found in <a
|
|
href="http://www.gnu.org/software/emacs/emacs.html">Emacs</a>.
|
|
|
|
<li>In addition to specifying regular expressions as strings like in
|
|
Perl you can also use <a
|
|
href="#create-scanner2"><b>S-expressions</b></a> which obviously is
|
|
more Lisp-y.
|
|
|
|
<li>Is it is fully <b>documented</b> so I might have a chance to
|
|
understand my own code in about six months... :)
|
|
|
|
<li>It comes with a <a
|
|
href="http://www.opensource.org/licenses/bsd-license.php"><b>BSD-style
|
|
license</b></a> so you can basically do with it whatever you want.
|
|
|
|
</ul>
|
|
|
|
</blockquote>
|
|
|
|
<br> <br><h3><a class=none name="contents">Contents</a></h3>
|
|
<ol>
|
|
<li><a href="#howto">How to use CL-PPCRE</a>
|
|
<ol>
|
|
<li><a href="#create-scanner1"><code>create-scanner</code></a> (for Perl regex strings)
|
|
<li><a href="#create-scanner2"><code>create-scanner</code></a> (for parse trees)
|
|
<li><a href="#scan"><code>scan</code></a>
|
|
<li><a href="#scan-to-strings"><code>scan-to-strings</code></a>
|
|
<li><a href="#register-groups-bind"><code>register-groups-bind</code></a>
|
|
<li><a href="#do-scans"><code>do-scans</code></a>
|
|
<li><a href="#do-matches"><code>do-matches</code></a>
|
|
<li><a href="#do-matches-as-strings"><code>do-matches-as-strings</code></a>
|
|
<li><a href="#do-register-groups"><code>do-register-groups</code></a>
|
|
<li><a href="#all-matches"><code>all-matches</code></a>
|
|
<li><a href="#all-matches-as-strings"><code>all-matches-as-strings</code></a>
|
|
<li><a href="#split"><code>split</code></a>
|
|
<li><a href="#regex-replace"><code>regex-replace</code></a>
|
|
<li><a href="#regex-replace-all"><code>regex-replace-all</code></a>
|
|
<li><a href="#regex-apropos"><code>regex-apropos</code></a>
|
|
<li><a href="#regex-apropos-list"><code>regex-apropos-list</code></a>
|
|
<li><a href="#regex-char-code-limit"><code>*regex-char-code-limit*</code></a>
|
|
<li><a href="#use-bmh-matchers"><code>*use-bmh-matchers*</code></a>
|
|
<li><a href="#*allow-quoting*"><code>*allow-quoting*</code></a>
|
|
<li><a href="#quote-meta-chars"><code>quote-meta-chars</code></a>
|
|
<li><a href="#ppcre-error"><code>ppcre-error</code></a>
|
|
<li><a href="#ppcre-invocation-error"><code>ppcre-invocation-error</code></a>
|
|
<li><a href="#ppcre-syntax-error"><code>ppcre-syntax-error</code></a>
|
|
<li><a href="#ppcre-syntax-error-string"><code>ppcre-syntax-error-string</code></a>
|
|
<li><a href="#ppcre-syntax-error-pos"><code>ppcre-syntax-error-pos</code></a>
|
|
</ol>
|
|
<li><a href="#install">Download and installation</a>
|
|
<li><a href="#mail">Support and mailing lists</a>
|
|
<li><a href="#test">Testing CL-PPCRE</a>
|
|
<li><a href="#perl">Compatibility with Perl</a>
|
|
<ol>
|
|
<li><a href="#empty">Empty strings instead of <code>undef</code> in <code>$1</code>, <code>$2</code>, etc.</a>
|
|
<li><a href="#scope">Strange scoping of embedded modifiers</a>
|
|
<li><a href="#inconsistent">Inconsistent capturing of <code>$1</code>, <code>$2</code>, etc.</a>
|
|
<li><a href="#lookaround">Captured groups not available outside of look-aheads and look-behinds</a>
|
|
<li><a href="#order">Alternations don't always work from left to right</a>
|
|
<li><a href="#mac"><code>"\r"</code> doesn't work with MCL</a>
|
|
<li><a href="#alpha">What about <code>"\w"</code>?</a>
|
|
</ol>
|
|
<li><a href="#performance">Performance</a>
|
|
<ol>
|
|
<li><a href="#bench">Benchmarking</a>
|
|
<li><a href="#other">Other performance issues</a>
|
|
</ol>
|
|
<li><a href="#bugs">Bugs and problems</a>
|
|
<ol>
|
|
<li><a href="#stack">Stack overflow</a>
|
|
<li><a href="#quote"><code>"\Q"</code> doesn't work, or does it?</a>
|
|
<li><a href="#backslash">Backslashes may confuse you...</a>
|
|
</ol>
|
|
<li><a href="#remarks">Remarks</a>
|
|
<li><a href="#ack">Acknowledgements</a>
|
|
</ol>
|
|
|
|
<br> <br><h3><a class=none name="howto">How to use CL-PPCRE</a></h3>
|
|
|
|
CL-PPCRE exports the following symbols:
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="create-scanner1"><b>create-scanner</b> <i>string <tt>&key</tt> case-insensitive-mode multi-line-mode single-line-mode extended-mode destructive</i> => <i>scanner</i></a>
|
|
|
|
<blockquote><br> Accepts a string which is a regular expression in
|
|
Perl syntax and returns a closure which will scan strings for this
|
|
regular expression. The mode keyboard arguments are equivalent to the
|
|
<code>"imsx"</code> modifiers in Perl. The
|
|
<code>destructive</code> keyword will be ignored.
|
|
<p>
|
|
The function accepts most of the regex syntax of Perl 5 as described
|
|
in <a
|
|
href="http://www.perldoc.com/perl5.8.0/pod/perlre.html"><code>man
|
|
perlre</code></a> including extended features like non-greedy
|
|
repetitions, positive and negative look-ahead and look-behind
|
|
assertions, "standalone" subexpressions, and conditional
|
|
subpatterns. The following Perl features are (currently) <b>not</b>
|
|
supported:
|
|
|
|
<ul>
|
|
|
|
<li><code>(?{ code })</code> and <code>(??{ code })</code> because
|
|
they obviously don't make sense in Lisp.
|
|
|
|
<li><code>\N{name}</code> (named characters), <code>\x{263a}</code>
|
|
(wide hex characters), <code>\l</code>, <code>\u</code>,
|
|
<code>\L</code>, and <code>\U</code>
|
|
because they're actually not part of Perl's regex syntax and
|
|
(honestly) because I was too lazy - but see <a href="http://weitz.de/cl-interpol/">CL-INTERPOL</a>.
|
|
|
|
<li><code>\pP</code> and <code>\PP</code> (named properties),
|
|
<code>\X</code> (extended Unicode), and <code>\C</code> (single
|
|
character). But you can of course use all characters
|
|
supported by your CL implementation.
|
|
|
|
<li>Posix character classes like <code>[[:alpha]]</code>. I
|
|
<em>might</em> add this in the future.
|
|
|
|
<li><code>\G</code> for Perl's <code>pos()</code> because we don't have it.
|
|
|
|
</ul>
|
|
|
|
Note, however, that <code>\t</code>, <code>\n</code>, <code>\r</code>,
|
|
<code>\f</code>, <code>\a</code>, <code>\e</code>, <code>\033</code>
|
|
(octal character codes), <code>\x1B</code> (hexadecimal character
|
|
codes), <code>\c[</code> (control characters), <code>\w</code>,
|
|
<code>\W</code>, <code>\s</code>, <code>\S</code>, <code>\d</code>,
|
|
<code>\D</code>, <code>\b</code>, <code>\B</code>, <code>\A</code>,
|
|
<code>\Z</code>, and <code>\z</code> <b>are</b> supported.
|
|
<p>
|
|
Since version 0.6.0 CL-PPCRE also supports Perl's <code>\Q</code> and <code>\E</code> - see <a
|
|
href="#*allow-quoting*"><code>*ALLOW-QUOTING*</code></a> below. Make sure you also read <a href="#quote">the relevant section</a> in "<a href="#bugs">Bugs and problems</a>."
|
|
<p>
|
|
The keyword arguments are just for your
|
|
convenience. You can always use embedded modifiers like
|
|
<code>"(?i-s)"</code> instead.</blockquote>
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="create-scanner2"><b>create-scanner</b> <i>parse-tree <tt>&key</tt> case-insensitive-mode multi-line-mode single-line-mode extended-mode destructive</i> => <i>scanner</i></a>
|
|
<blockquote><br>
|
|
This is similar to <a
|
|
href="#create-scanner1"><code>CREATE-SCANNER</code></a> above but
|
|
accepts a <em>parse tree</em> as its first argument. A parse tree is an S-expression
|
|
conforming to the following syntax:
|
|
|
|
<ul>
|
|
|
|
<li>Every string and character is a parse tree and is treated
|
|
<em>literally</em> as a part of the regular expression,
|
|
i.e. parentheses, brackets, asterisks and such aren't special.
|
|
|
|
<li>The symbol <code>:VOID</code> is equivalent to the empty string.
|
|
|
|
<li>The symbol <code>:EVERYTHING</code> is equivalent to Perl's dot,
|
|
i.e it matches everything (except maybe a newline character depending
|
|
on the mode).
|
|
|
|
<li>The symbols <code>:WORD-BOUNDARY</code> and
|
|
<code>:NON-WORD-BOUNDARY</code> are equivalent to Perl's
|
|
<code>"\b"</code> and <code>"\B"</code>.
|
|
|
|
<li>The symbols <code>:DIGIT-CLASS</code>,
|
|
<code>:NON-DIGIT-CLASS</code>, <code>:WORD-CHAR-CLASS</code>,
|
|
<code>:NON-WORD-CHAR-CLASS</code>,
|
|
<code>:WHITESPACE-CHAR-CLASS</code>, and
|
|
<code>:NON-WHITESPACE-CHAR-CLASS</code> are equivalent to Perl's
|
|
<em>special character classes</em> <code>"\d"</code>,
|
|
<code>"\D"</code>, <code>"\w"</code>,
|
|
<code>"\W"</code>, <code>"\s"</code>, and
|
|
<code>"\S"</code> respectively.
|
|
|
|
<li>The symbols <code>:START-ANCHOR</code>, <code>:END-ANCHOR</code>,
|
|
<code>:MODELESS-START-ANCHOR</code>,
|
|
<code>:MODELESS-END-ANCHOR</code>, and
|
|
<code>:MODELESS-END-ANCHOR-NO-NEWLINE</code> are equivalent to Perl's
|
|
<code>"^"</code>, <code>"$"</code>,
|
|
<code>"\A"</code>, <code>"\Z"</code>, and
|
|
<code>"\z"</code> respectively.
|
|
|
|
<li>The symbols <code>:CASE-INSENSITIVE-P</code>,
|
|
<code>:CASE-SENSITIVE-P</code>, <code>:MULTI-LINE-MODE-P</code>,
|
|
<code>:NOT-MULTI-LINE-MODE-P</code>, <code>:SINGLE-LINE-MODE-P</code>,
|
|
and <code>:NOT-SINGLE-LINE-MODE-P</code> are equivalent to Perl's
|
|
<em>embedded modifiers</em> <code>"(?i)"</code>,
|
|
<code>"(?-i)"</code>, <code>"(?m)"</code>,
|
|
<code>"(?-m)"</code>, <code>"(?s)"</code>, and
|
|
<code>"(?-s)"</code>. As usual, changes applied to modes are
|
|
kept local to the innermost enclosing grouping or clustering
|
|
construct.
|
|
|
|
<li><code>(:FLAGS {<modifier>}*)</code> where
|
|
<code><modifier></code> is one of the modifier symbols from
|
|
above is used to group modifier symbols. The modifiers are applied
|
|
from left to right. (This construct is obviously redundant. It is only
|
|
there because it's used by the parser.)
|
|
|
|
<li><code>(:SEQUENCE {<<i>parse-tree</i>>}*)</code> means a
|
|
sequence of parse trees, i.e. the parse trees must match one after
|
|
another. Example: <code>(:SEQUENCE #\f #\o #\o)</code> is equivalent
|
|
to the parse tree <code>"foo"</code>.
|
|
|
|
<li><code>(:GROUP {<<i>parse-tree</i>>}*)</code> is like
|
|
<code>:SEQUENCE</code> but changes applied to modifier flags (see
|
|
above) are kept local to the parse trees enclosed by this
|
|
construct. Think of it as the S-expression variant of Perl's
|
|
<code>"(?:<<i>pattern</i>>)"</code> construct.
|
|
|
|
<li><code>(:ALTERNATION {<<i>parse-tree</i>>}*)</code> means an
|
|
alternation of parse trees, i.e. one of the parse trees must
|
|
match. Example: <code>(:ALTERNATION #\b #\a #\z)</code> is equivalent
|
|
to the Perl regex string <code>"b|a|z"</code>.
|
|
|
|
<li><code>(:BRANCH <<i>test</i>>
|
|
<<i>parse-tree</i>>)</code> is for conditional regular
|
|
expressions. <code><<i>test</i>></code> is either a number which
|
|
stands for a register or a parse tree which is a look-ahead or
|
|
look-behind assertion. See the entry for
|
|
<code>(?(<<i>condition</i>>)<<i>yes-pattern</i>>|<<i>no-pattern</i>>)</code>
|
|
in <a
|
|
href="http://www.perldoc.com/perl5.8.0/pod/perlre.html#Extended-Patterns"><code>man
|
|
perlre</code></a> for the semantics of this construct. If
|
|
<code><<i>parse-tree</i>></code> is an alternation is
|
|
<em>must</em> enclose exactly one or two parse trees where the second
|
|
one (if present) will be treated as the "no-pattern" - in
|
|
all other cases <code><<i>parse-tree</i>></code> will be treated
|
|
as the "yes-pattern".
|
|
|
|
<li><code>(:POSITIVE-LOOKAHEAD|:NEGATIVE-LOOKAHEAD|:POSITIVE-LOOKBEHIND|:NEGATIVE-LOOKBEHIND
|
|
<<i>parse-tree</i>>)</code> should be pretty obvious...
|
|
|
|
<li><code>(:GREEDY-REPETITION|:NON-GREEDY-REPETITION
|
|
<<i>min</i>> <<i>max</i>>
|
|
<<i>parse-tree</i>>)</code> where
|
|
<code><<i>min</i>></code> is a non-negative integer and
|
|
<code><<i>max</i>></code> is either a non-negative integer not
|
|
smaller than <code><<i>min</i>></code> or <code>NIL</code> will
|
|
result in a regular expression which tries to match
|
|
<code><<i>parse-tree</i>></code> at least
|
|
<code><<i>min</i>></code> times and at most
|
|
<code><<i>max</i>></code> times (or as often as possible if
|
|
<code><<i>max</i>></code> is <code>NIL</code>). So, e.g.,
|
|
<code>(:NON-GREEDY-REPETITION 0 1 "ab")</code> is equivalent
|
|
to the Perl regex string <code>"(?:ab)??"</code>.
|
|
|
|
<li><code>(:STANDALONE <<i>parse-tree</i>>)</code> is an
|
|
"independent" subexpression, i.e. <code>(:STANDALONE
|
|
"bar")</code> is equivalent to the Perl regex string
|
|
<code>"(?>bar)"</code>.
|
|
|
|
<li><code>(:REGISTER <<i>parse-tree</i>>)</code> is a capturing
|
|
register group. As usual, registers are counted from left to right
|
|
beginning with 1.
|
|
|
|
<li><code>(:BACK-REFERENCE <<i>number</i>>)</code> where
|
|
<code><<i>number</i>></code> is a positive integer is a back-reference to a
|
|
register group.
|
|
|
|
<li><code>(:CHAR-CLASS|:INVERTED-CHAR-CLASS
|
|
{<<i>item</i>>}*)</code> where <code><<i>item</i>></code>
|
|
is either a character, a <em>character range</em>, or a symbol for a
|
|
special character class (see above) will be translated into a (one
|
|
character wide) character class. A <em>character range</em> looks like
|
|
<code>(:RANGE <<i>char1</i>> <<i>char2</i>>)</code> where
|
|
<code><<i>char1</i>></code> and
|
|
<code><<i>char2</i>></code> are characters such that
|
|
<code>(CHAR<= <<i>char1</i>> <<i>char2</i>>)</code> is
|
|
true. Example: <code>(:INVERTED-CHAR-CLASS #\a (:RANGE #\D #\G)
|
|
:DIGIT-CLASS)</code> is equivalent to the Perl regex string
|
|
<code>"[^aD-G\d]"</code>.
|
|
|
|
</ul>
|
|
|
|
Because <code>CREATE-SCANNER</code> is defined as a generic function
|
|
which dispatches on its first argument there's a certain ambiguity:
|
|
Although strings are valid parse trees they will be interpreted as
|
|
Perl regex strings when given to <code>CREATE-SCANNER</code>. To
|
|
circumvent this you can always use the equivalent parse tree <code>(:GROUP
|
|
<<i>string</i>>)</code> instead.
|
|
<p>
|
|
Note that currently <code>CREATE-SCANNER</code> doesn't always check
|
|
for the well-formedness of its first argument, i.e. you are expected
|
|
to provide <em>correct</em> parse trees. This will most likely change in
|
|
future releases.
|
|
<p>
|
|
The usage of the keyword argument <code>extended-mode</code> obviously
|
|
doesn't make sense if <code>CREATE-SCANNER</code> is applied to parse
|
|
trees and will signal an error.
|
|
<p>
|
|
If <code>destructive</code> is not <code>NIL</code> (the default is
|
|
<code>NIL</code>) the function is allowed to destructively modify
|
|
<code><i>parse-tree</i></code> while creating the scanner.
|
|
<p>
|
|
If you want to find out how parse trees are related to Perl regex
|
|
strings you should play around with
|
|
<code>CL-PPCRE::PARSE-STRING</code> - a function which converts Perl
|
|
regex strings to parse trees. Here are some examples:
|
|
|
|
<pre>
|
|
* (cl-ppcre::parse-string "(ab)*")
|
|
(:GREEDY-REPETITION 0 NIL (:REGISTER "ab"))
|
|
|
|
* (cl-ppcre::parse-string "(a(b))")
|
|
(:REGISTER (:SEQUENCE #\a (:REGISTER #\b)))
|
|
|
|
* (cl-ppcre::parse-string "(?:abc){3,5}")
|
|
(:GREEDY-REPETITION 3 5 (:GROUP "abc"))
|
|
<font color=orange>;; (:GREEDY-REPETITION 3 5 "abc") would also be OK</font>
|
|
|
|
* (cl-ppcre::parse-string "a(?i)b(?-i)c")
|
|
(:SEQUENCE #\a
|
|
(:SEQUENCE (:FLAGS :CASE-INSENSITIVE-P)
|
|
(:SEQUENCE #\b (:SEQUENCE (:FLAGS :CASE-SENSITIVE-P) #\c))))
|
|
<font color=orange>;; same as (:SEQUENCE #\a :CASE-INSENSITIVE-P #\b :CASE-SENSITIVE-P #\c)</font>
|
|
|
|
* (cl-ppcre::parse-string "(?=a)b")
|
|
(:SEQUENCE (:POSITIVE-LOOKAHEAD #\a) #\b)
|
|
</pre></blockquote>
|
|
|
|
<p><br>
|
|
<b>For the rest of this section </b><code><i>regex</i></code><b> can
|
|
always be a string (which is interpreted as a Perl regular
|
|
expression), a parse tree, or a scanner created by
|
|
</b><code>CREATE-SCANNER</code><b>. The
|
|
</b><code><i>start</i></code><b> and </b><code><i>end</i></code><b>
|
|
keyword parameters are always used as in </b><a
|
|
href="#scan"><code>SCAN</code></a><b>.</b>
|
|
|
|
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="scan"><b>scan</b> <i>regex target-string <tt>&key</tt> start end</i> => <i>match-start, match-end, reg-starts, reg-ends</i></a>
|
|
|
|
<blockquote><br>
|
|
Searches the string <code><i>target-string</i></code> from
|
|
<code><i>start</i></code> (which defaults to 0) to
|
|
<code><i>end</i></code> (which default to the length of
|
|
<code><i>target-string</i></code>) and tries to match
|
|
<code><i>regex</i></code>. On success returns four values - the start
|
|
of the match, the end of the match, and two arrays denoting the
|
|
beginnings and ends of register matches. On failure returns
|
|
<code>NIL</code>. <code><i>target-string</i></code> will be coerced to a
|
|
simple string if it isn't one already.
|
|
<p>
|
|
<code>SCAN</code> acts as if the part of
|
|
<code><i>target-string</i></code> between <code><i>start</i></code>
|
|
and <code><i>end</i></code> were a standalone string, i.e. look-aheads
|
|
and look-behinds can't look beyond these boundaries.
|
|
<p>
|
|
Examples:
|
|
<pre>
|
|
* (cl-ppcre:scan "(a)*b" "xaaabd")
|
|
1
|
|
5
|
|
#(3)
|
|
#(4)
|
|
|
|
* (cl-ppcre:scan "(a)*b" "xaaabd" :start 1)
|
|
1
|
|
5
|
|
#(3)
|
|
#(4)
|
|
|
|
* (cl-ppcre:scan "(a)*b" "xaaabd" :start 2)
|
|
2
|
|
5
|
|
#(3)
|
|
#(4)
|
|
|
|
* (cl-ppcre:scan "(a)*b" "xaaabd" :end 4)
|
|
NIL
|
|
|
|
* (cl-ppcre:scan '(:GREEDY-REPETITION 0 NIL #\b) "bbbc")
|
|
0
|
|
3
|
|
#()
|
|
#()
|
|
|
|
* (cl-ppcre:scan '(:GREEDY-REPETITION 4 6 #\b) "bbbc")
|
|
NIL
|
|
|
|
* (let ((s (cl-ppcre:create-scanner "(([a-c])+)x")))
|
|
(cl-ppcre:scan s "abcxy"))
|
|
0
|
|
4
|
|
#(0 2)
|
|
#(3 3)
|
|
</pre></blockquote>
|
|
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="scan-to-strings"><b>scan-to-strings</b> <i>regex target-string <tt>&key</tt> start end sharedp</i> => <i>match, regs</i></a>
|
|
|
|
<blockquote><br>
|
|
Like <a href="#scan"><code>SCAN</code></a> but returns substrings of
|
|
<code><i>target-string</i></code> instead of positions, i.e. this
|
|
function returns two values on success: the whole match as a string
|
|
plus an array of substrings (or <code>NIL</code>s) corresponding to
|
|
the matched registers. If <code><i>sharedp</i></code> is true, the substrings may share structure with
|
|
<code><i>target-string</i></code>.
|
|
<p>
|
|
Examples:
|
|
<pre>
|
|
* (cl-ppcre:scan-to-strings "[^b]*b" "aaabd")
|
|
"aaab"
|
|
#()
|
|
|
|
* (cl-ppcre:scan-to-strings "([^b])*b" "aaabd")
|
|
"aaab"
|
|
#("a")
|
|
|
|
* (cl-ppcre:scan-to-strings "(([^b])*)b" "aaabd")
|
|
"aaab"
|
|
#("aaa" "a")
|
|
</pre></blockquote>
|
|
|
|
|
|
<p><br>[Macro]
|
|
<br><a class=none name="register-groups-bind"><b>register-groups-bind</b> <i>var-list (regex target-string <tt>&key</tt> start end sharedp) declaration* statement*</i> => <i>result*</i></a>
|
|
|
|
<blockquote><br>
|
|
Evaluates <code><i>statement*</i></code> with the variables in <code><i>var-list</i></code> bound to the
|
|
corresponding register groups after <code><i>target-string</i></code> has been matched
|
|
against <code><i>regex</i></code>, i.e. each variable is either
|
|
bound to a string or to <code>NIL</code>. If there is no match, the <code><i>statement*</i></code> forms are <em>not</em>
|
|
executed. For each element of
|
|
<code><i>var-list</i></code> which is <code>NIL</code> there's no binding to the corresponding register
|
|
group. The number of variables in <code><i>var-list</i></code> must not be greater than
|
|
the number of register groups. If <code><i>sharedp</i></code> is true, the substrings may
|
|
share structure with <code><i>target-string</i></code>.
|
|
<p>Examples:
|
|
<pre>
|
|
* (register-groups-bind (first second third fourth)
|
|
("((a)|(b)|(c))+" "abababc" :sharedp t)
|
|
(list first second third fourth))
|
|
("c" "a" "b" "c")
|
|
* (register-groups-bind (nil second third fourth)
|
|
<font color=orange>;; note that we don't bind the first and fifth register group</font>
|
|
("((a)|(b)|(c))()+" "abababc" :start 6)
|
|
(list second third fourth))
|
|
(NIL NIL "c")
|
|
* (register-groups-bind (first)
|
|
("(a|b)+" "accc" :start 1)
|
|
(format t "This will not be printed: ~A" first))
|
|
NIL
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<p><br>[Macro]
|
|
<br><a class=none name="do-scans"><b>do-scans</b> <i>(match-start match-end reg-starts reg-ends regex target-string <tt>&optional</tt> result-form <tt>&key</tt> start end) declaration* statement*</i> => <i>result*</i></a>
|
|
|
|
<blockquote><br>
|
|
A macro which iterates over <code><i>target-string</i></code> and
|
|
tries to match <code><i>regex</i></code> as often as possible
|
|
evaluating <code><i>statement*</i></code> with
|
|
<code><i>match-start</i></code>, <code><i>match-end</i></code>,
|
|
<code><i>reg-starts</i></code>, and <code><i>reg-ends</i></code> bound
|
|
to the four return values of each match (see <a
|
|
href="#scan"><code>SCAN</code></a>) in turn. After the last match,
|
|
returns <code><i>result-form</i></code> if provided or
|
|
<code>NIL</code> otherwise. An implicit block named <code>NIL</code>
|
|
surrounds <code>DO-SCANS</code>; <code>RETURN</code> may be used to
|
|
terminate the loop immediately. If <code><i>regex</i></code> matches
|
|
an empty string the scan is continued one position behind this match.
|
|
<p>
|
|
This is the most general macro to iterate over all matches in a target
|
|
string. See the source code of <a
|
|
href="#do-matches"><code>DO-MATCHES</code></a>, <a
|
|
href="#all-matches"><code>ALL-MATCHES</code></a>, <a
|
|
href="#split"><code>SPLIT</code></a>, or <a
|
|
href="#regex-replace-all"><code>REGEX-REPLACE-ALL</code></a> for examples of its
|
|
usage.</blockquote>
|
|
|
|
|
|
|
|
|
|
<p><br>[Macro]
|
|
<br><a class=none name="do-matches"><b>do-matches</b> <i>(match-start match-end regex target-string <tt>&optional</tt> result-form <tt>&key</tt> start end) declaration* statement*</i> => <i>result*</i></a>
|
|
|
|
<blockquote><br>
|
|
Like <a href="#do-scans"><code>DO-SCANS</code></a> but doesn't bind
|
|
variables to the register arrays.
|
|
<p>Example:
|
|
<pre>
|
|
* (defun foo (regex target-string &key (start 0) (end (length target-string)))
|
|
(let ((sum 0))
|
|
(cl-ppcre:do-matches (s e regex target-string nil :start start :end end)
|
|
(incf sum (- e s)))
|
|
(format t "~,2F% of the string was inside of a match~%"
|
|
<font color=orange>;; note: doesn't check for division by zero</font>
|
|
(float (* 100 (/ sum (- end start)))))))
|
|
|
|
FOO
|
|
|
|
* (foo "a" "abcabcabc")
|
|
33.33% of the string was inside of a match
|
|
NIL
|
|
* (foo "aa|b" "aacabcbbc")
|
|
55.56% of the string was inside of a match
|
|
NIL
|
|
</pre></blockquote>
|
|
|
|
|
|
|
|
|
|
<p><br>[Macro]
|
|
<br><a class=none name="do-matches-as-strings"><b>do-matches-as-strings</b> <i>(match-var regex target-string <tt>&optional</tt> result-form <tt>&key</tt> start end sharedp) declaration* statement*</i> => <i>result*</i></a>
|
|
|
|
<blockquote><br>
|
|
Like <a href="#do-matches"><code>DO-MATCHES</code></a> but binds
|
|
<code><i>match-var</i></code> to the substring of
|
|
<code><i>target-string</i></code> corresponding to each match in turn. If <code><i>sharedp</i></code> is true, the substrings may share structure with
|
|
<code><i>target-string</i></code>.
|
|
<p>
|
|
Example:
|
|
<pre>
|
|
* (defun crossfoot (target-string &key (start 0) (end (length target-string)))
|
|
(let ((sum 0))
|
|
(cl-ppcre:do-matches-as-strings (m :digit-class
|
|
target-string nil
|
|
:start start :end end)
|
|
(incf sum (parse-integer m)))
|
|
(if (< sum 10)
|
|
sum
|
|
(crossfoot (format nil "~A" sum)))))
|
|
|
|
CROSSFOOT
|
|
|
|
* (crossfoot "bar")
|
|
0
|
|
|
|
* (crossfoot "a3x")
|
|
3
|
|
|
|
* (crossfoot "12345")
|
|
6
|
|
</pre>
|
|
|
|
Of course, in real life you would do this with <a href="#do-matches"><code>DO-MATCHES</code></a> and use the <code><i>start</i></code> and <code><i>end</i></code> keyword parameters of <a href="http://www.lispworks.com/reference/HyperSpec/Body/f_parse_.htm"><code>PARSE-INTEGER</code></a>.</blockquote>
|
|
|
|
<p><br>[Macro]
|
|
<br><a class=none name="do-register-groups"><b>do-register-groups</b> <i>var-list (regex target-string <tt>&optional</tt> result-form <tt>&key</tt> start end sharedp) declaration* statement*</i> => <i>result*</i></a>
|
|
|
|
<blockquote><br>
|
|
Iterates over <code><i>target-string</i></code> and tries to match <code><i>regex</i></code> as often as
|
|
possible evaluating <code><i>statement*</i></code> with the variables in <code><i>var-list</i></code> bound to the
|
|
corresponding register groups for each match in turn, i.e. each
|
|
variable is either bound to a string or to <code>NIL</code>. The number of
|
|
variables in <code><i>var-list</i></code> must not be greater than the number of register
|
|
groups. For each element of
|
|
<code><i>var-list</i></code> which is <code>NIL</code> there's no binding to the corresponding register
|
|
group. After the last match, returns <code><i>result-form</i></code> if provided or <code>NIL</code>
|
|
otherwise. An implicit block named <code>NIL</code> surrounds <code>DO-REGISTER-GROUPS</code>;
|
|
<code>RETURN</code> may be used to terminate the loop immediately. If <code><i>regex</i></code> matches
|
|
an empty string the scan is continued one position behind this
|
|
match. If <code><i>sharedp</i></code> is true, the substrings may share structure with
|
|
<code><i>target-string</i></code>.
|
|
<p>Example:
|
|
<pre>
|
|
* (do-register-groups (first second third fourth)
|
|
("((a)|(b)|(c))" "abababc" nil :start 2 :sharedp t)
|
|
(print (list first second third fourth)))
|
|
("a" "a" NIL NIL)
|
|
("b" NIL "b" NIL)
|
|
("a" "a" NIL NIL)
|
|
("b" NIL "b" NIL)
|
|
("c" NIL NIL "c")
|
|
NIL
|
|
</pre>
|
|
</blockquote>
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="all-matches"><b>all-matches</b> <i>regex target-string <tt>&key</tt> start end</i> => <i>list</i></a>
|
|
|
|
<blockquote><br>
|
|
Returns a list containing the start and end positions of all matches
|
|
of <code><i>regex</i></code> against
|
|
<code><i>target-string</i></code>, i.e. if there are <code>N</code>
|
|
matches the list contains <code>(* 2 N)</code> elements. If
|
|
<code><i>regex</i></code> matches an empty string the scan is
|
|
continued one position behind this match.
|
|
<p>
|
|
Examples:
|
|
<pre>
|
|
* (cl-ppcre:all-matches "a" "foo bar baz")
|
|
(5 6 9 10)
|
|
|
|
* (cl-ppcre:all-matches "\\w*" "foo bar baz")
|
|
(0 3 3 3 4 7 7 7 8 11 11 11)
|
|
</pre></blockquote>
|
|
|
|
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="all-matches-as-strings"><b>all-matches-as-strings</b> <i>regex target-string <tt>&key</tt> start end sharedp</i> => <i>list</i></a>
|
|
|
|
<blockquote><br>
|
|
Like <a href="#all-matches"><code>ALL-MATCHES</code></a> but
|
|
returns a list of substrings instead. If <code><i>sharedp</i></code> is true, the substrings may share structure with
|
|
<code><i>target-string</i></code>.
|
|
<p>
|
|
Examples:
|
|
<pre>
|
|
* (cl-ppcre:all-matches-as-strings "a" "foo bar baz")
|
|
("a" "a")
|
|
|
|
* (cl-ppcre:all-matches-as-strings "\\w*" "foo bar baz")
|
|
("foo" "" "bar" "" "baz" "")
|
|
</pre></blockquote>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="split"><b>split</b> <i>regex target-string <tt>&key</tt> start end limit with-registers-p omit-unmatched-p sharedp</i> => <i>list</i></a>
|
|
|
|
<blockquote><br>
|
|
Matches <code><i>regex</i></code> against
|
|
<code><i>target-string</i></code> as often as possible and returns a
|
|
list of the substrings between the matches. If
|
|
<code><i>with-registers-p</i></code> is true, substrings corresponding
|
|
to matched registers are inserted into the list as well. If
|
|
<code><i>omit-unmatched-p</i></code> is true, unmatched registers will
|
|
simply be left out, otherwise they will show up as
|
|
<code>NIL</code>. <code><i>limit</i></code> limits the number of
|
|
elements returned - registers aren't counted. If
|
|
<code><i>limit</i></code> is <code>NIL</code> (or 0 which is
|
|
equivalent), trailing empty strings are removed from the result list.
|
|
If <code><i>regex</i></code> matches an empty string the scan is
|
|
continued one position behind this match. If <code><i>sharedp</i></code> is true, the substrings may share structure with
|
|
<code><i>target-string</i></code>.
|
|
<p>
|
|
Beginning with CL-PPCRE 0.2.0, this function also tries hard to be
|
|
Perl-compatible - thus the somewhat peculiar behaviour. But note that
|
|
it hasn't been as extensively tested as <a
|
|
href="#scan"><code>SCAN</code></a>.
|
|
<p>
|
|
Examples:
|
|
<pre>
|
|
* (cl-ppcre:split "\\s+" "foo bar baz
|
|
frob")
|
|
("foo" "bar" "baz" "frob")
|
|
|
|
* (cl-ppcre:split "\\s*" "foo bar baz")
|
|
("f" "o" "o" "b" "a" "r" "b" "a" "z")
|
|
|
|
* (cl-ppcre:split "(\\s+)" "foo bar baz")
|
|
("foo" "bar" "baz")
|
|
|
|
* (cl-ppcre:split "(\\s+)" "foo bar baz" :with-registers-p t)
|
|
("foo" " " "bar" " " "baz")
|
|
|
|
* (cl-ppcre:split "(\\s)(\\s*)" "foo bar baz" :with-registers-p t)
|
|
("foo" " " "" "bar" " " " " "baz")
|
|
|
|
* (cl-ppcre:split "(,)|(;)" "foo,bar;baz" :with-registers-p t)
|
|
("foo" "," NIL "bar" NIL ";" "baz")
|
|
|
|
* (cl-ppcre:split "(,)|(;)" "foo,bar;baz" :with-registers-p t :omit-unmatched-p t)
|
|
("foo" "," "bar" ";" "baz")
|
|
|
|
* (cl-ppcre:split ":" "a:b:c:d:e:f:g::")
|
|
("a" "b" "c" "d" "e" "f" "g")
|
|
|
|
* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 1)
|
|
("a:b:c:d:e:f:g::")
|
|
|
|
* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 2)
|
|
("a" "b:c:d:e:f:g::")
|
|
|
|
* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 3)
|
|
("a" "b" "c:d:e:f:g::")
|
|
|
|
* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 1000)
|
|
("a" "b" "c" "d" "e" "f" "g" "" "")
|
|
</pre></blockquote>
|
|
|
|
|
|
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="regex-replace"><b>regex-replace</b> <i>regex target-string replacement <tt>&key</tt> start end preserve-case</i> => <i>list</i></a>
|
|
|
|
<blockquote><br> Try to match <code><i>target-string</i></code>
|
|
between <code><i>start</i></code> and <code><i>end</i></code> against
|
|
<code><i>regex</i></code> and replace the first match with
|
|
<code><i>replacement</i></code>.
|
|
<p>
|
|
<code><i>replacement</i></code> can be a string which may contain the
|
|
special substrings <code>"\&"</code> for the whole
|
|
match, <code>"\`"</code> for the part of
|
|
<code><i>target-string</i></code> before the match,
|
|
<code>"\'"</code> for the part of
|
|
<code><i>target-string</i></code> after the match,
|
|
<code>"\N"</code> or <code>"\{N}"</code> for the
|
|
<code>N</code>th register where <code>N</code> is a positive integer.
|
|
<p>
|
|
<code><i>replacement</i></code> can also be a <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/26_glo_f.htm#function_designator">function
|
|
designator</a> in which case the match will be replaced with the
|
|
result of calling the function designated by
|
|
<code><i>replacement</i></code> with the arguments
|
|
<code><i>target-string</i></code>, <code><i>start</i></code>,
|
|
<code><i>end</i></code>, <code><i>match-start</i></code>,
|
|
<code><i>match-end</i></code>, <code><i>reg-starts</i></code>, and
|
|
<code><i>reg-ends</i></code>. (<code><i>reg-starts</i></code> and
|
|
<code><i>reg-ends</i></code> are arrays holding the start and end
|
|
positions of matched registers (or <code>NIL</code>) - the meaning of
|
|
the other arguments should be obvious.)
|
|
<p>
|
|
Finally, <code><i>replacement</i></code> can be a list where each
|
|
element is a string (which will be inserted verbatim), one of the
|
|
symbols <code>:match</code>, <code>:before-match</code>, or
|
|
<code>:after-match</code> (corresponding to
|
|
<code>"\&"</code>, <code>"\`"</code>, and
|
|
<code>"\'"</code> above), an integer <code>N</code>
|
|
(representing register <code>(1+ N)</code>), or a function
|
|
designator.
|
|
<p>
|
|
If <code><i>preserve-case</i></code> is true (default is
|
|
<code>NIL</code>), the replacement will try to preserve the case (all
|
|
upper case, all lower case, or capitalized) of the match. The result
|
|
will always be a <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/26_glo_f.htm#fresh">fresh</a>
|
|
string, even if <code><i>regex</i></code> doesn't match.
|
|
<p>
|
|
Examples:
|
|
|
|
<pre>
|
|
* (cl-ppcre:regex-replace "fo+" "foo bar" "frob")
|
|
"frob bar"
|
|
|
|
* (cl-ppcre:regex-replace "fo+" "FOO bar" "frob")
|
|
"FOO bar"
|
|
|
|
* (cl-ppcre:regex-replace "(?i)fo+" "FOO bar" "frob")
|
|
"frob bar"
|
|
|
|
* (cl-ppcre:regex-replace "(?i)fo+" "FOO bar" "frob" :preserve-case t)
|
|
"FROB bar"
|
|
|
|
* (cl-ppcre:regex-replace "(?i)fo+" "Foo bar" "frob" :preserve-case t)
|
|
"Frob bar"
|
|
|
|
* (cl-ppcre:regex-replace "bar" "foo bar baz" "[frob (was '\\&' between '\\`' and '\\'')]")
|
|
"foo [frob (was 'bar' between 'foo ' and ' baz')] baz"
|
|
|
|
* (cl-ppcre:regex-replace "bar" "foo bar baz"
|
|
'("[frob (was '" :match "' between '" :before-match "' and '" :after-match "')]"))
|
|
"foo [frob (was 'bar' between 'foo ' and ' baz')] baz"
|
|
</pre></blockquote>
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="regex-replace-all"><b>regex-replace-all</b> <i>regex target-string replacement <tt>&key</tt> start end preserve-case</i> => <i>list</i></a>
|
|
|
|
<blockquote><br>
|
|
Like <a href="#regex-replace"><code>REGEX-REPLACE</code></a> but replaces all matches.
|
|
<p>
|
|
Examples:
|
|
|
|
<pre>
|
|
* (cl-ppcre:regex-replace-all "(?i)fo+" "foo Fooo FOOOO bar" "frob" :preserve-case t)
|
|
"frob Frob FROB bar"
|
|
|
|
* (cl-ppcre:regex-replace-all "(?i)f(o+)" "foo Fooo FOOOO bar" "fr\\1b" :preserve-case t)
|
|
"froob Frooob FROOOOB bar"
|
|
|
|
* (let ((qp-regex (cl-ppcre:create-scanner "[\\x80-\\xff]")))
|
|
(defun encode-quoted-printable (string)
|
|
"Convert 8-bit string to quoted-printable representation."
|
|
<font color=orange>;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there</font>
|
|
(flet ((convert (target-string start end match-start match-end reg-starts reg-ends)
|
|
(declare (ignore start end match-end reg-starts reg-ends))
|
|
(format nil "=~2,'0x" (char-code (char target-string match-start)))))
|
|
(cl-ppcre:regex-replace-all qp-regex string #'convert))))
|
|
Converted ENCODE-QUOTED-PRINTABLE.
|
|
ENCODE-QUOTED-PRINTABLE
|
|
|
|
* (encode-quoted-printable "Fête Sørensen naïve Hühner Straße")
|
|
"F=EAte S=F8rensen na=EFve H=FChner Stra=DFe"
|
|
|
|
* (let ((url-regex (cl-ppcre:create-scanner "[^a-zA-Z0-9_\\-.]")))
|
|
(defun url-encode (string)
|
|
"URL-encode a string."
|
|
<font color=orange>;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there</font>
|
|
(flet ((convert (target-string start end match-start match-end reg-starts reg-ends)
|
|
(declare (ignore start end match-end reg-starts reg-ends))
|
|
(format nil "%~2,'0x" (char-code (char target-string match-start)))))
|
|
(cl-ppcre:regex-replace-all url-regex string #'convert))))
|
|
Converted URL-ENCODE.
|
|
URL-ENCODE
|
|
|
|
* (url-encode "Fête Sørensen naïve Hühner Straße")
|
|
"F%EAte%20S%F8rensen%20na%EFve%20H%FChner%20Stra%DFe"
|
|
|
|
* (defun how-many (target-string start end match-start match-end reg-starts reg-ends)
|
|
(declare (ignore start end match-start match-end))
|
|
(format nil "~A" (- (svref reg-ends 0)
|
|
(svref reg-starts 0))))
|
|
HOW-MANY
|
|
|
|
* (cl-ppcre:regex-replace-all "{(.+?)}"
|
|
"foo{...}bar{.....}{..}baz{....}frob"
|
|
(list "[" 'how-many " dots]"))
|
|
"foo[3 dots]bar[5 dots][2 dots]baz[4 dots]frob"
|
|
</pre></blockquote>
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="regex-apropos"><b>regex-apropos</b> <i>regex <tt>&optional</tt> packages <tt>&key</tt> case-insensitive</i> => <i>list</i></a>
|
|
|
|
<blockquote><br>
|
|
Like <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/f_apropo.htm"><code>APROPOS</code></a>
|
|
but searches for interned symbols which match the regular expression
|
|
<code><i>regex</i></code>. The output is implementation-dependent. If
|
|
<code><i>case-insensitive</i></code> is true (which is the default)
|
|
and <code><i>regex</i></code> isn't already a scanner, a
|
|
case-insensitive scanner is used.
|
|
<p>
|
|
Here are examples for CMUCL:
|
|
|
|
<pre>
|
|
* *package*
|
|
#<The COMMON-LISP-USER package, 16/21 internal, 0/9 external>
|
|
|
|
* (defun foo (n &optional (k 0)) (+ 3 n k))
|
|
FOO
|
|
|
|
* (defparameter foo "bar")
|
|
FOO
|
|
|
|
* (defparameter |foobar| 42)
|
|
|foobar|
|
|
|
|
* (defparameter fooboo 43)
|
|
FOOBOO
|
|
|
|
* (defclass frobar () ())
|
|
#<STANDARD-CLASS FROBAR {4874E625}>
|
|
|
|
* (cl-ppcre:regex-apropos "foo(?:bar)?")
|
|
FOO [variable] value: "bar"
|
|
[compiled function] (N &OPTIONAL (K 0))
|
|
FOOBOO [variable] value: 43
|
|
|foobar| [variable] value: 42
|
|
|
|
* (cl-ppcre:regex-apropos "(?:foo|fro)bar")
|
|
PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]
|
|
FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
|
|
|foobar| [variable] value: 42
|
|
|
|
* (cl-ppcre:regex-apropos "(?:foo|fro)bar" 'cl-user)
|
|
FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
|
|
|foobar| [variable] value: 42
|
|
|
|
* (cl-ppcre:regex-apropos "(?:foo|fro)bar" '(pcl ext))
|
|
PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]
|
|
|
|
* (cl-ppcre:regex-apropos "foo")
|
|
FOO [variable] value: "bar"
|
|
[compiled function] (N &OPTIONAL (K 0))
|
|
FOOBOO [variable] value: 43
|
|
|foobar| [variable] value: 42
|
|
|
|
* (cl-ppcre:regex-apropos "foo" nil :case-insensitive nil)
|
|
|foobar| [variable] value: 42
|
|
</pre></blockquote>
|
|
|
|
|
|
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="regex-apropos-list"><b>regex-apropos-list</b> <i>regex <tt>&optional</tt> packages <tt>&key</tt> upcase</i> => <i>list</i></a>
|
|
|
|
<blockquote><br>
|
|
Like <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/f_apropo.htm"><code>APROPOS-LIST</code></a>
|
|
but searches for interned symbols which match the regular expression
|
|
<code><i>regex</i></code>. If <code><i>case-insensitive</i></code> is
|
|
true (which is the default) and <code><i>regex</i></code> isn't
|
|
already a scanner, a case-insensitive scanner is used.
|
|
<p>
|
|
Example (continued from above):
|
|
|
|
<pre>
|
|
* (cl-ppcre:regex-apropos-list "foo(?:bar)?")
|
|
(|foobar| FOOBOO FOO)
|
|
</pre></blockquote>
|
|
|
|
<p><br>[Special variable]
|
|
<br><a class=none name="regex-char-code-limit"><b>*regex-char-code-limit*</b></a>
|
|
|
|
<blockquote><br>This variable controls whether scanners take into
|
|
account all characters of your CL implementation or only those the <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/f_char_c.htm#char-code"><code>CHAR-CODE</code></a>
|
|
of which is not larger than its value. It is only relevant if the
|
|
regular expression contains certain character classes. The default is
|
|
<a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/v_char_c.htm"><code>CHAR-CODE-LIMIT</code></a>,
|
|
and you might see significant speed and space improvements during
|
|
scanner <em>creation</em> if, say, your target strings only contain <a
|
|
href="http://wwwwbs.cs.tu-berlin.de/user/czyborra/charsets/">ISO-8859-1</a>
|
|
characters and you're using an implementation like AllegroCL,
|
|
LispWorks, or CLISP where <code>CHAR-CODE-LIMIT</code> has a value
|
|
much higher than 255. The <a href="#test">test suite</a> will
|
|
automatically set <code>*REGEX-CHAR-CODE-LIMIT*</code> to 255 while
|
|
you're running the default test.
|
|
<p>
|
|
Here's an example with LispWorks:
|
|
|
|
<pre>
|
|
CL-USER 23 > (time (cl-ppcre:create-scanner "[3\\D]"))
|
|
Timing the evaluation of (CL-PPCRE:CREATE-SCANNER "[3\\D]")
|
|
|
|
user time = 0.443
|
|
system time = 0.001
|
|
Elapsed time = 0:00:01
|
|
Allocation = 546600 bytes standard / 2162611 bytes fixlen
|
|
0 Page faults
|
|
#<closure 20654AF2>
|
|
|
|
CL-USER 24 > (time (let ((cl-ppcre:*regex-char-code-limit* 255)) (cl-ppcre:create-scanner "[3\\D]")))
|
|
Timing the evaluation of (LET ((CL-PPCRE:*REGEX-CHAR-CODE-LIMIT* 255)) (CL-PPCRE:CREATE-SCANNER "[3\\D]"))
|
|
|
|
user time = 0.000
|
|
system time = 0.000
|
|
Elapsed time = 0:00:00
|
|
Allocation = 3336 bytes standard / 8338 bytes fixlen
|
|
0 Page faults
|
|
#<closure 206569DA>
|
|
</pre>
|
|
<p>
|
|
Note: Due to the nature of <code>LOAD-TIME-VALUE</code> and the <a
|
|
href="#compiler-macro">compiler macro for <code>SCAN</code></a> some
|
|
scanners might be created in a <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/26_glo_n.htm#null_lexical_environment">null
|
|
lexical environment</a> at load time or at compile time so be careful
|
|
to which value <code>*REGEX-CHAR-CODE-LIMIT*</code> is bound at that
|
|
time. The default value should always yield correct results unless you
|
|
play dirty tricks with implementation-dependent behaviour, though.</blockquote>
|
|
|
|
<p><br>[Special variable]
|
|
<br><a class=none name="use-bmh-matchers"><b>*use-bmh-matchers*</b></a>
|
|
|
|
<blockquote><br>Usually, the scanners created by <a
|
|
href="#create-scanner1"><code>CREATE-SCANNER</code></a> (or
|
|
implicitely by other functions and macros) will use fast <a
|
|
href="http://www-igm.univ-mlv.fr/~lecroq/string/node18.html">Boyer-Moore-Horspool
|
|
matchers</a> to check for constant strings at the start or end of the
|
|
regular expression. If <code>*USE-BMH-MATCHERS*</code> is
|
|
<code>NIL</code> (the default is <code>T</code>), the standard
|
|
function <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/f_search.htm"><code>SEARCH</code></a>
|
|
will be used instead. This will usually be a bit slower but can save
|
|
lots of space if you're storing many scanners. The <a
|
|
href="#test">test suite</a> will automatically set
|
|
<code>*USE-BMH-MATCHERS*</code> to <code>NIL</code> while you're running
|
|
the default test.
|
|
<p>
|
|
Note: Due to the nature of <code>LOAD-TIME-VALUE</code> and the <a
|
|
href="#compiler-macro">compiler macro for <code>SCAN</code></a> some
|
|
scanners might be created in a <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/26_glo_n.htm#null_lexical_environment">null
|
|
lexical environment</a> at load time or at compile time so be careful
|
|
to which value <code>*USE-BMH-MATCHERS*</code> is bound at that
|
|
time.</blockquote>
|
|
|
|
<p><br>[Special variable]
|
|
<br><a class=none name="*allow-quoting*"><b>*allow-quoting*</b></a>
|
|
|
|
<blockquote><br>
|
|
If this value is <em>true</em> (the default is <code>NIL</code>)
|
|
CL-PPCRE will support <code>\Q</code> and <code>\E</code> in regex
|
|
strings to quote (disable) metacharacters. Note that this entails a
|
|
slight performance penalty when creating scanners because (a copy of) the regex
|
|
string is modified (probably more than once) before it
|
|
is fed to the parser. Also, the parser's <a
|
|
href="#ppcre-syntax-error">syntax error messages</a> will complain
|
|
about the converted string and not about the original regex string.
|
|
|
|
<pre>
|
|
* (cl-ppcre:scan "^a+$" "a+")
|
|
NIL
|
|
|
|
* (let ((cl-ppcre:*allow-quoting* t))
|
|
(cl-ppcre:scan "^\\Qa+\\E$" "a+"))
|
|
0
|
|
2
|
|
#()
|
|
#()
|
|
|
|
* (let ((cl-ppcre:*allow-quoting* t))
|
|
(cl-ppcre:scan "\\Qa()\\E(?#comment\\Q)a**b" "()ab"))
|
|
|
|
Quantifier '*' not allowed at position 19 in string "a\\(\\)(?#commentQ)a**b"
|
|
</pre>
|
|
|
|
Note how in the last example the regex string in the error message is
|
|
different from the first argument to the <code>SCAN</code>
|
|
function. Also note that the second example might be easier to
|
|
understand (and Lisp-ier) if you write it like this:
|
|
|
|
<pre>
|
|
* (cl-ppcre:scan '(:sequence :start-anchor
|
|
"a+" <font color=orange>;; no quoting necessary</font>
|
|
:end-anchor)
|
|
"a+")
|
|
0
|
|
2
|
|
#()
|
|
#()
|
|
</pre>
|
|
|
|
Make sure you also read <a href="#quote">the relevant section</a> in "<a href="#bugs">Bugs and problems</a>."
|
|
|
|
</blockquote>
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="quote-meta-chars"><b>quote-meta-chars</b> <i>string</i> => <i>string'</i></a>
|
|
|
|
<blockquote><br>
|
|
This is a simple utility function used when <a
|
|
href="#*allow-quoting*"><code>*ALLOW-QUOTING*</code></a> is
|
|
<em>true</em>. It returns a string <code>STRING'</code> where all
|
|
non-word characters (everything except ASCII characters, digits and
|
|
underline) of <code>STRING</code> are quoted by prepending a
|
|
backslash similar to Perl's <code>quotemeta</code> function. It always returns a <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/26_glo_f.htm#fresh">fresh</a>
|
|
string.
|
|
<pre>
|
|
* (cl-ppcre:quote-meta-chars "[a-z]*")
|
|
"\\[a\\-z\\]\\*"
|
|
</pre></blockquote>
|
|
|
|
<p><br>[Condition type]
|
|
<br><a class=none name="ppcre-error"><b>ppcre-error</b></a>
|
|
|
|
<blockquote><br>
|
|
Every error signaled by CL-PPCRE is of type
|
|
<code>PPCRE-ERROR</code>. This is a direct subtype of <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/e_smp_er.htm"><code>SIMPLE-ERROR</code></a>
|
|
without any additional slots or options.
|
|
</blockquote>
|
|
|
|
<p><br>[Condition type]
|
|
<br><a class=none name="ppcre-invocation-error"><b>ppcre-invocation-error</b></a>
|
|
|
|
<blockquote><br>
|
|
Errors of type <code>PPCRE-INVOCATION-ERROR</code>
|
|
are signaled if one of the exported functions of CL-PPCRE is called with wrong or
|
|
inconsistent arguments. This is a direct subtype of <a
|
|
href="#ppcre-error"><code>PPCRE-ERROR</code></a> without any
|
|
additional slots or options.
|
|
</blockquote>
|
|
|
|
<p><br>[Condition type]
|
|
<br><a class=none name="ppcre-syntax-error"><b>ppcre-syntax-error</b></a>
|
|
|
|
<blockquote><br>
|
|
An error of type <code>PPCRE-SYNTAX-ERROR</code> is signaled if
|
|
CL-PPCRE's parser encounters an error when trying to parse a regex
|
|
string or to convert a parse tree into its internal representation.
|
|
This is a direct subtype of <a
|
|
href="#ppcre-error"><code>PPCRE-ERROR</code></a> with two additional
|
|
slots. These denote the regex string which HTML-PPCRE was parsing and
|
|
the position within the string where the error occured. If the error
|
|
happens while CL-PPCRE is converting a parse tree both of these slots
|
|
contain <code>NIL</code>. (See the next two entries on how to access
|
|
these slots.)
|
|
<p>
|
|
As many syntax errors can't be detected before the parser is at the
|
|
end of the stream, the row and column usually denote the last position
|
|
where the parser was happy and not the position where it gave up.
|
|
|
|
<pre>
|
|
* (handler-case
|
|
(cl-ppcre:scan "foo**x" "fooox")
|
|
(cl-ppcre:ppcre-syntax-error (condition)
|
|
(format t "Houston, we've got a problem with the string ~S:~%~
|
|
Looks like something went wrong at position ~A.~%~
|
|
The last message we received was \"~?\"."
|
|
(cl-ppcre:ppcre-syntax-error-string condition)
|
|
(cl-ppcre:ppcre-syntax-error-pos condition)
|
|
(simple-condition-format-control condition)
|
|
(simple-condition-format-arguments condition))
|
|
(values)))
|
|
Houston, we've got a problem with the string "foo**x":
|
|
Looks like something went wrong at position 4.
|
|
The last message we received was "Quantifier '*' not allowed".
|
|
</pre>
|
|
</blockquote>
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="ppcre-syntax-error-string"><b>ppcre-syntax-error-string</b></a> <i>condition</i> => <i>string</i></a>
|
|
|
|
<blockquote><br>
|
|
If <code><i>condition</i></code> is a condition of type <a
|
|
href="#ppcre-syntax-error"><code>PPCRE-SYNTAX-ERROR</code></a> this
|
|
function will return the string the parser was parsing when the error was
|
|
encountered (or <code>NIL</code> if the error happened while trying to
|
|
convert a parse tree). This might be particularly useful when <a
|
|
href="#*allow-quoting*"><code>*ALLOW-QUOTING*</code></a> is
|
|
<em>true</em> because in this case the offending string might not be the one you gave to the <a
|
|
href="#create-scanner1"><code>CREATE-SCANNER</code></a> function.
|
|
</blockquote>
|
|
|
|
<p><br>[Function]
|
|
<br><a class=none name="ppcre-syntax-error-pos"><b>ppcre-syntax-error-pos</b></a> <i>condition</i> => <i>number</i></a>
|
|
|
|
<blockquote><br>
|
|
If <code><i>condition</i></code> is a condition of type <a
|
|
href="#ppcre-syntax-error"><code>PPCRE-SYNTAX-ERROR</code></a> this
|
|
function will return the position within the string where the error
|
|
occured (or <code>NIL</code> if the error happened while trying to
|
|
convert a parse tree).
|
|
</blockquote>
|
|
|
|
|
|
<br> <br><h3><a name="install" class=none>Download and installation</a></h3>
|
|
|
|
CL-PPCRE together with this documentation can be downloaded from <a
|
|
href="http://weitz.de/files/cl-ppcre.tgz">http://weitz.de/files/cl-ppcre.tgz</a>. The
|
|
current version is 0.7.4 - older versions are
|
|
available for download through URLs like
|
|
<code>http://weitz.de/files/cl-ppcre-<version>.tgz</code>. A <a
|
|
href="CHANGELOG">CHANGELOG</a> is available.
|
|
<p>
|
|
If you're on <a href="http://www.debian.org/">Debian</a> you should
|
|
probably use the <a
|
|
href="http://packages.debian.org/cgi-bin/search_packages.pl?keywords=cl-ppcre&searchon=names&version=all&release=all">cl-ppcre
|
|
Debian package</a> which is available thanks to <a href="http://b9.com/">Kevin
|
|
Rosenberg</a>. There's also a port
|
|
for <a href="http://www.cliki.net/gentoo">Gentoo Linux</a> thanks to Matthew Kennedy and a <a href="http://www.freebsd.org/cgi/url.cgi?ports/textproc/cl-ppcre/pkg-descr">FreeBSD port</a> thanks to Henrik Motakef.
|
|
Installation via <a
|
|
href="http://www.cliki.net/asdf-install">asdf-install</a> should as well
|
|
be possible.
|
|
<p>
|
|
CL-PPCRE comes with simple system definitions for <a
|
|
href="http://www.cliki.net/mk-defsystem">MK:DEFSYSTEM</a> and <a
|
|
href="http://www.cliki.net/asdf">asdf</a> so you can either adapt it
|
|
to your needs or just unpack the archive and from within the CL-PPCRE
|
|
directory start your Lisp image and evaluate the form
|
|
<code>(mk:compile-system "cl-ppcre")</code> (or the
|
|
equivalent one for asdf) which should compile and load the whole
|
|
system.
|
|
<p>
|
|
If for some reason you don't want to use MK:DEFSYSTEM or asdf you
|
|
can just <code>LOAD</code> the file <code>load.lisp</code> or you
|
|
can also get away with something like this:
|
|
|
|
<pre>
|
|
(loop for name in '("packages" "specials" "util" "errors" "lexer"
|
|
"parser" "regex-class" "convert" "optimize"
|
|
"closures" "repetition-closures" "scanner" "api")
|
|
do (compile-file (make-pathname :name name
|
|
:type "lisp"))
|
|
(load name))
|
|
</pre>
|
|
|
|
Note that on CL implementations which use the Python compiler
|
|
(i.e. CMUCL, SBCL, SCL) you can concatenate the compiled object files
|
|
to create one single object file which you can load afterwards:
|
|
|
|
<pre>
|
|
cat {packages,specials,util,errors,lexer,parser,regex-class,convert,optimize,closures,repetition-closures,scanner,api}.x86f > cl-ppcre.x86f
|
|
</pre>
|
|
|
|
(Replace ".<code>x86f</code>" with the correct suffix for
|
|
your platform.)
|
|
|
|
|
|
<br> <br><h3><a name="mail" class=none>Support and mailing lists</a></h3>
|
|
|
|
For questions, bug reports, feature requests, improvements, or patches
|
|
please use the <a
|
|
href="http://common-lisp.net/mailman/listinfo/cl-ppcre-devel">cl-ppcre-devel
|
|
mailing list</a>. If you want to be notified about future releases
|
|
subscribe to the <a
|
|
href="http://common-lisp.net/mailman/listinfo/cl-ppcre-announce">cl-ppcre-announce
|
|
mailing list</a>. These mailing lists were made available thanks to
|
|
the services of <a href="http://common-lisp.net/">common-lisp.net</a>.
|
|
|
|
<br> <br><h3><a name="test" class=none>Testing CL-PPCRE</a></h3>
|
|
|
|
CL-PPCRE comes with a comprehensive test suite most of which is stolen
|
|
from the <a href="http://www.pcre.org/">PCRE</a> library. You can use
|
|
it like this:
|
|
|
|
<pre>
|
|
* (mk:compile-system "cl-ppcre-test")
|
|
<font color=orange>; Loading #p"/home/edi/cl-ppcre/cl-ppcre.system".
|
|
; Loading #p"/home/edi/cl-ppcre/packages.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/specials.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/util.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/errors.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/lexer.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/parser.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/regex-class.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/convert.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/optimize.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/closures.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/repetition-closures.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/scanner.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/api.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/ppcre-tests.x86f".</font>
|
|
NIL
|
|
|
|
* (cl-ppcre-test:test)
|
|
|
|
<font color=orange>;; ....
|
|
;; (a list of <a href="#perl">incompatibilities with Perl</a>)</font color=orange>
|
|
</pre>
|
|
|
|
(If you're not using MK:DEFSYSTEM or asdf it suffices to build
|
|
CL-PPCRE and then compile and load the file
|
|
<code>ppcre-tests.lisp</code>.)
|
|
<p>
|
|
With LispWorks, SCL, and SBCL (starting from version 0.8.4.8) you can also call
|
|
<code>CL-PPCRE-TEST:TEST</code> with a keyword argument argument
|
|
<code>THREADED</code> which - in addition to the usual tests - will
|
|
also check whether the scanners created by CL-PPCRE are thread-safe.
|
|
<p>
|
|
Note that the file <code>testdata</code> provided with CL-PPCRE
|
|
was created on a Linux system with Perl 5.8.0. You can (and you
|
|
<em>should</em> if you're on Mac OS or Windows) create your own
|
|
<code>testdata</code> with the Perl script
|
|
<code>perltest.pl</code>:
|
|
|
|
<pre>
|
|
edi@bird:~/cl-ppcre > perl perltest.pl < testinput > testdata
|
|
</pre>
|
|
|
|
Of course you can also create your own tests - the format accepted by
|
|
<code>perltest.pl</code> should be rather clear from looking at the
|
|
file <code>testinput</code>. Note that the target strings are wrapped
|
|
in double quotes and then fed to Perl's <code>eval</code> so you can
|
|
use ugly Perl constructs like, say, <code>a@{['b' x 10]}c</code> which
|
|
will result in the target string
|
|
<code>"abbbbbbbbbbc"</code>.
|
|
|
|
<br> <br><h3><a name="perl" class=none>Compatibility with Perl</a></h3>
|
|
|
|
Depending on your Perl version you might encounter a couple of small
|
|
incompatibilities with Perl most of which aren't due to CL-PPCRE:
|
|
|
|
<h4><a name="empty" class=none>Empty strings instead of <code>undef</code> in <code>$1</code>, <code>$2</code>, etc.</a></h4>
|
|
|
|
(Cf. case #629 of <a href="#test"><code>testdata</code></a>.)
|
|
This is <a
|
|
href="http://groups.google.com/groups?threadm=87u1kw8hfr.fsf%40dyn164.dbdmedia.de">a
|
|
bug</a> in Perl 5.6.1 and earlier which has been fixed in 5.8.0.
|
|
|
|
<h4><a name="scope" class=none>Strange scoping of embedded modifiers</a></h4>
|
|
|
|
(Cf. case #430 of <a href="#test"><code>testdata</code></a>.)
|
|
This is <a
|
|
href="http://groups.google.com/groups?threadm=871y80dpqh.fsf%40bird.agharta.de">a
|
|
bug</a> in Perl 5.6.1 and earlier which has been fixed in 5.8.0.
|
|
|
|
<h4><a name="inconsistent" class=none>Inconsistent capturing of <code>$1</code>, <code>$2</code>, etc.</a></h4>
|
|
|
|
(Cf. case #662 of <a href="#test"><code>testdata</code></a>.)
|
|
This is <a
|
|
href="http://bugs6.perl.org/rt2/Ticket/Display.html?id=18708">a
|
|
bug</a> in Perl which hasn't been fixed yet.
|
|
|
|
<h4><a name="lookaround" class=none>Captured groups not available outside of look-aheads and look-behinds</a></h4>
|
|
|
|
(Cf. case #1439 of <a href="#test"><code>testdata</code></a>.)
|
|
Well, OK, this ain't a Perl bug. I just can't quite understand why
|
|
captured groups should only be seen within the scope of a look-ahead
|
|
or look-behind. For the moment, CL-PPCRE and Perl agree to
|
|
disagree... :)
|
|
|
|
<h4><a name="order" class=none>Alternations don't always work from left to right</a></h4>
|
|
|
|
(Cf. case #790 of <a href="#test"><code>testdata</code></a>.) I
|
|
also think this a Perl bug but I currently have lost the drive to
|
|
report it.
|
|
|
|
<h4><a name="mac" class=none><code>"\r"</code> doesn't work with MCL</a></h4>
|
|
|
|
(Cf. case #9 of <a href="#test"><code>testdata</code></a>.) For
|
|
some strange reason that I don't understand MCL translates
|
|
<code>#\Return</code> to <code>(CODE-CHAR 10)</code> while MacPerl
|
|
translates <code>"\r"</code> to <code>(CODE-CHAR
|
|
13)</code>. Hmmm...
|
|
|
|
<h4><a name="alpha" class=none>What about <code>"\w"</code>?</a></h4>
|
|
|
|
CL-PPCRE uses <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/f_alphan.htm"><code>ALPHANUMERICP</code></a>
|
|
to decide whether a character matches Perl's
|
|
<code>"\w"</code>, so depending on your CL implementation
|
|
you might encounter differences between Perl and CL-PPCRE when
|
|
matching non-ASCII characters.
|
|
|
|
<br> <br><h3><a name="performance" class=none>Performance</a></h3>
|
|
|
|
<h4><a name="bench" class=none>Benchmarking</a></h4>
|
|
|
|
The <a href="">CL-PPCRE test suite</a> can also be used for
|
|
benchmarking purposes: If you call <code>perltest.pl</code> with a
|
|
command line argument it will be interpreted as the number of seconds
|
|
each test should run. Perl will time its tests accordingly and create
|
|
output which, when fed to <code>CL-PPCRE-TEST:TEST</code>, will result
|
|
in a benchmark. Here's an example:
|
|
|
|
<pre>
|
|
edi@bird:~/cl-ppcre > echo "/((a{0,5}){0,5})*[c]/
|
|
aaaaaaaaaaaac
|
|
|
|
/((a{0,5})*)*[c]/
|
|
aaaaaaaaaaaac" | perl perltest.pl .5 > timedata
|
|
1
|
|
2
|
|
|
|
edi@bird:~/cl-ppcre > cmucl -quiet
|
|
<font color=orange>; Loading #p"/home/edi/.cmucl-init".</font>
|
|
|
|
* (mk:compile-system "cl-ppcre-test")
|
|
<font color=orange>; Loading #p"/home/edi/cl-ppcre/cl-ppcre.system".
|
|
; Loading #p"/home/edi/cl-ppcre/packages.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/specials.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/util.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/errors.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/lexer.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/parser.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/regex-class.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/convert.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/optimize.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/closures.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/repetition-closures.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/scanner.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/api.x86f".
|
|
; Loading #p"/home/edi/cl-ppcre/ppcre-tests.x86f".</font>
|
|
NIL
|
|
|
|
* (cl-ppcre-test:test :file-name "/home/edi/cl-ppcre/timedata")
|
|
1: 0.5559 (1000000 repetitions, Perl: 4.5330 seconds, CL-PPCRE: 2.5200 seconds)
|
|
2: 0.4573 (1000000 repetitions, Perl: 4.5922 seconds, CL-PPCRE: 2.1000 seconds)
|
|
NIL
|
|
</pre>
|
|
|
|
We gave two test cases to <code>perltest.pl</code> and asked it to repeat those tests often enough so that it takes at least 0.5 seconds to run each of them. In both cases, CMUCL was about twice as fast as Perl.
|
|
<p>
|
|
Here are some more benchmarks (done with Perl 5.6.1 and CMUCL 18d+):
|
|
<p>
|
|
|
|
<table border=1>
|
|
<tr><td><b>Test case</b></td><td><b>Repetitions</b></td><td><b>Perl (sec)</b></td><td><b>CL-PPCRE (sec)</b></td><td><b>Ratio CL-PPCRE/Perl</b></td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /(.)*/s</code></td><td align=right>100000</td><td align=right>0.1394</td><td align=right>0.0700</td><td align=right>0.5022</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /(.)*/s</code></td><td align=right>100000</td><td align=right>0.1628</td><td align=right>0.0600</td><td align=right>0.3685</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /(.)*/s</code></td><td align=right>100000</td><td align=right>0.5071</td><td align=right>0.0600</td><td align=right>0.1183</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /(.)*/s</code></td><td align=right>10000</td><td align=right>0.3902</td><td align=right>0.0000</td><td align=right>0.0000</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /.*/</code></td><td align=right>100000</td><td align=right>0.1520</td><td align=right>0.0800</td><td align=right>0.5262</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /.*/</code></td><td align=right>100000</td><td align=right>0.3786</td><td align=right>0.5400</td><td align=right>1.4263</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /.*/</code></td><td align=right>10000</td><td align=right>0.2709</td><td align=right>0.5100</td><td align=right>1.8826</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /.*/</code></td><td align=right>1000</td><td align=right>0.2734</td><td align=right>0.5100</td><td align=right>1.8656</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /.*/s</code></td><td align=right>100000</td><td align=right>0.1320</td><td align=right>0.0300</td><td align=right>0.2274</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /.*/s</code></td><td align=right>100000</td><td align=right>0.1634</td><td align=right>0.0300</td><td align=right>0.1836</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /.*/s</code></td><td align=right>100000</td><td align=right>0.5304</td><td align=right>0.0300</td><td align=right>0.0566</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /.*/s</code></td><td align=right>10000</td><td align=right>0.3966</td><td align=right>0.0000</td><td align=right>0.0000</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /x*/</code></td><td align=right>100000</td><td align=right>0.1507</td><td align=right>0.0900</td><td align=right>0.5970</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /x*/</code></td><td align=right>100000</td><td align=right>0.3782</td><td align=right>0.6300</td><td align=right>1.6658</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /x*/</code></td><td align=right>10000</td><td align=right>0.2730</td><td align=right>0.6000</td><td align=right>2.1981</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /x*/</code></td><td align=right>1000</td><td align=right>0.2708</td><td align=right>0.5900</td><td align=right>2.1790</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /[xy]*/</code></td><td align=right>100000</td><td align=right>0.2637</td><td align=right>0.1500</td><td align=right>0.5688</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /[xy]*/</code></td><td align=right>10000</td><td align=right>0.1449</td><td align=right>0.1200</td><td align=right>0.8282</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /[xy]*/</code></td><td align=right>1000</td><td align=right>0.1344</td><td align=right>0.1100</td><td align=right>0.8185</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /[xy]*/</code></td><td align=right>100</td><td align=right>0.1355</td><td align=right>0.1200</td><td align=right>0.8857</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /(.)*/</code></td><td align=right>100000</td><td align=right>0.1523</td><td align=right>0.1100</td><td align=right>0.7220</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /(.)*/</code></td><td align=right>100000</td><td align=right>0.3735</td><td align=right>0.5700</td><td align=right>1.5262</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /(.)*/</code></td><td align=right>10000</td><td align=right>0.2735</td><td align=right>0.5100</td><td align=right>1.8647</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /(.)*/</code></td><td align=right>1000</td><td align=right>0.2598</td><td align=right>0.5000</td><td align=right>1.9242</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /(x)*/</code></td><td align=right>100000</td><td align=right>0.1565</td><td align=right>0.1300</td><td align=right>0.8307</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /(x)*/</code></td><td align=right>100000</td><td align=right>0.3783</td><td align=right>0.6600</td><td align=right>1.7446</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /(x)*/</code></td><td align=right>10000</td><td align=right>0.2720</td><td align=right>0.6000</td><td align=right>2.2055</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /(x)*/</code></td><td align=right>1000</td><td align=right>0.2725</td><td align=right>0.6000</td><td align=right>2.2020</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /(y|x)*/</code></td><td align=right>10000</td><td align=right>0.2411</td><td align=right>0.1000</td><td align=right>0.4147</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /(y|x)*/</code></td><td align=right>1000</td><td align=right>0.2313</td><td align=right>0.0900</td><td align=right>0.3891</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /(y|x)*/</code></td><td align=right>100</td><td align=right>0.2336</td><td align=right>0.0900</td><td align=right>0.3852</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /(y|x)*/</code></td><td align=right>10</td><td align=right>0.4165</td><td align=right>0.0900</td><td align=right>0.2161</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /([xy])*/</code></td><td align=right>100000</td><td align=right>0.2678</td><td align=right>0.1800</td><td align=right>0.6721</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /([xy])*/</code></td><td align=right>10000</td><td align=right>0.1459</td><td align=right>0.1200</td><td align=right>0.8227</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /([xy])*/</code></td><td align=right>1000</td><td align=right>0.1372</td><td align=right>0.1100</td><td align=right>0.8017</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /([xy])*/</code></td><td align=right>100</td><td align=right>0.1358</td><td align=right>0.1100</td><td align=right>0.8098</td></tr>
|
|
<tr><td><code>"@{['x' x 100]}" =~ /((x){2})*/</code></td><td align=right>10000</td><td align=right>0.1073</td><td align=right>0.0400</td><td align=right>0.3727</td></tr>
|
|
<tr><td><code>"@{['x' x 1000]}" =~ /((x){2})*/</code></td><td align=right>10000</td><td align=right>0.9146</td><td align=right>0.2400</td><td align=right>0.2624</td></tr>
|
|
<tr><td><code>"@{['x' x 10000]}" =~ /((x){2})*/</code></td><td align=right>1000</td><td align=right>0.9020</td><td align=right>0.2300</td><td align=right>0.2550</td></tr>
|
|
<tr><td><code>"@{['x' x 100000]}" =~ /((x){2})*/</code></td><td align=right>100</td><td align=right>0.8983</td><td align=right>0.2300</td><td align=right>0.2560</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/</code></td><td align=right>100000</td><td align=right>0.2829</td><td align=right>0.2300</td><td align=right>0.8129</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/</code></td><td align=right>10000</td><td align=right>0.1859</td><td align=right>0.1700</td><td align=right>0.9143</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/</code></td><td align=right>1000</td><td align=right>0.1420</td><td align=right>0.1700</td><td align=right>1.1968</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /[a-z]*FOOBARBAZ/</code></td><td align=right>1000000</td><td align=right>0.9196</td><td align=right>0.4600</td><td align=right>0.5002</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /[a-z]*FOOBARBAZ/</code></td><td align=right>100000</td><td align=right>0.2166</td><td align=right>0.2500</td><td align=right>1.1542</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /[a-z]*FOOBARBAZ/</code></td><td align=right>10000</td><td align=right>0.1465</td><td align=right>0.2300</td><td align=right>1.5696</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/</code></td><td align=right>100000</td><td align=right>0.2917</td><td align=right>0.2600</td><td align=right>0.8915</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/</code></td><td align=right>10000</td><td align=right>0.1811</td><td align=right>0.1800</td><td align=right>0.9942</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/</code></td><td align=right>1000</td><td align=right>0.1424</td><td align=right>0.1600</td><td align=right>1.1233</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /([a-z])*FOOBARBAZ/</code></td><td align=right>1000000</td><td align=right>0.9154</td><td align=right>0.7400</td><td align=right>0.8083</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /([a-z])*FOOBARBAZ/</code></td><td align=right>100000</td><td align=right>0.2170</td><td align=right>0.2800</td><td align=right>1.2901</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /([a-z])*FOOBARBAZ/</code></td><td align=right>10000</td><td align=right>0.1497</td><td align=right>0.2300</td><td align=right>1.5360</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/</code></td><td align=right>10000</td><td align=right>0.4359</td><td align=right>0.1500</td><td align=right>0.3441</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/</code></td><td align=right>1000</td><td align=right>0.5456</td><td align=right>0.1500</td><td align=right>0.2749</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/</code></td><td align=right>10</td><td align=right>0.2039</td><td align=right>0.0600</td><td align=right>0.2943</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/</code></td><td align=right>1000000</td><td align=right>0.9311</td><td align=right>0.7400</td><td align=right>0.7947</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/</code></td><td align=right>100000</td><td align=right>0.2162</td><td align=right>0.2700</td><td align=right>1.2489</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/</code></td><td align=right>10000</td><td align=right>0.1488</td><td align=right>0.2300</td><td align=right>1.5455</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /[a-z]*FOOBARBAZ/i</code></td><td align=right>1000</td><td align=right>0.1555</td><td align=right>0.0000</td><td align=right>0.0000</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /[a-z]*FOOBARBAZ/i</code></td><td align=right>10</td><td align=right>0.1441</td><td align=right>0.0000</td><td align=right>0.0000</td></tr>
|
|
<tr><td><code>"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /[a-z]*FOOBARBAZ/i</code></td><td align=right>10</td><td align=right>13.7150</td><td align=right>0.0100</td><td align=right>0.0007</td></tr>
|
|
|
|
</table>
|
|
|
|
<p>
|
|
As you might have noticed, Perl shines if it can reduce significant
|
|
parts of the matching process to cases where it can advance through
|
|
the target string one character at a time. This leads to C code where
|
|
you can very efficiently test and increment a pointer into a string in
|
|
a tight loop and can hardly be beaten with CL. In almost all other
|
|
cases, the CMUCL/CL-PPCRE combination is usually faster than Perl -
|
|
sometimes a lot faster.
|
|
<p>
|
|
As most of the examples above were chosen to make Perl look good
|
|
here's <a href="benchmarks.2002-12-22.txt">another benchmark</a> - the
|
|
result of running <a href="#test"><code>perltest.pl</code></a> against the
|
|
full <a href="#test"><code>testdata</code></a> file with a time
|
|
limit of 0.1 seconds, CL-PPCRE 0.1.2 on CMUCL 18e-pre
|
|
vs. Perl 5.6.1. CL-PPCRE is faster than Perl in 1511 of 1545
|
|
cases - in 1045 cases it's more than twice as fast.
|
|
<p>
|
|
Note that Perl as well as CL-PPCRE keep the rightmost matches in
|
|
registers - keep that in mind if you benchmark against other regex
|
|
implementations. Also note that <code>CL-PPCRE-TEST:TEST</code>
|
|
automatically skips test cases where Perl and CL-PPCRE don't agree.
|
|
|
|
<h4><a name="other" class=none>Other performance issues</a></h4>
|
|
|
|
While the scanners created by CL-PPCRE are pretty fast, the process
|
|
which creates scanners from Perl regex strings and parse trees isn't
|
|
that speedy and conses a lot. It is recommended that you store and
|
|
re-use scanners if possible. The <code>DO-</code>macros will do this
|
|
for you automatically.
|
|
<p>
|
|
However, beginning with version 0.5.2, CL-PPCRE uses a <a
|
|
name="compiler-macro"
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/26_glo_c.htm#compiler_macro">compiler
|
|
macro</a> and <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/s_ld_tim.htm"><code>LOAD-TIME-VALUE</code></a>
|
|
to make sure that the scanner is only built once if the first argument
|
|
to <a href="#scan"><code>SCAN</code></a>, <a href="#scan-to-strings"><code>SCAN-TO-STRINGS</code></a>, <a href="#split"><code>SPLIT</code></a>, or
|
|
<a href="#regex-replace"><code>REGEX-REPLACE</code></a> is a <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/26_glo_c.htm#constant_form">constant
|
|
form</a>. (But see the notes for <a
|
|
href="#regex-char-code-limit"><code>*REGEX-CHAR-CODE-LIMIT*</code></a> and
|
|
<a href="#use-bmh-matchers"><code>*USE-BMH-MATCHERS*</code></a>.)
|
|
<p>
|
|
Here's an example of its effect
|
|
|
|
<pre>
|
|
* (trace cl-ppcre::convert)
|
|
(CL-PPCRE::CONVERT)
|
|
* (defun foo (string) (cl-ppcre:scan "(?s).*" string))
|
|
FOO
|
|
* (time (foo "The quick brown fox"))
|
|
Compiling LAMBDA NIL:
|
|
Compiling Top-Level Form:
|
|
|
|
0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
|
|
0: CL-PPCRE::CONVERT returned
|
|
#<CL-PPCRE::SEQ {48B033C5}>
|
|
0
|
|
#<CL-PPCRE::EVERYTHING {48B031D5}>
|
|
Evaluation took:
|
|
0.0 seconds of real time
|
|
0.00293 seconds of user run time
|
|
9.77e-4 seconds of system run time
|
|
0 page faults and
|
|
11,408 bytes consed.
|
|
0
|
|
19
|
|
#()
|
|
#()
|
|
* (time (foo "The quick brown fox"))
|
|
Compiling LAMBDA NIL:
|
|
Compiling Top-Level Form:
|
|
|
|
0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
|
|
0: CL-PPCRE::CONVERT returned
|
|
#<CL-PPCRE::SEQ {48B14C4D}>
|
|
0
|
|
#<CL-PPCRE::EVERYTHING {48B14B65}>
|
|
Evaluation took:
|
|
0.0 seconds of real time
|
|
0.00293 seconds of user run time
|
|
0.0 seconds of system run time
|
|
0 page faults and
|
|
10,960 bytes consed.
|
|
0
|
|
19
|
|
#()
|
|
#()
|
|
* (compile 'foo)
|
|
0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
|
|
0: CL-PPCRE::CONVERT returned
|
|
#<CL-PPCRE::SEQ {48B1FEC5}>
|
|
0
|
|
#<CL-PPCRE::EVERYTHING {48B1FDDD}>
|
|
Compiling LAMBDA (STRING):
|
|
Compiling Top-Level Form:
|
|
FOO
|
|
NIL
|
|
NIL
|
|
* (time (foo "The quick brown fox"))
|
|
Compiling LAMBDA NIL:
|
|
Compiling Top-Level Form:
|
|
|
|
Evaluation took:
|
|
0.0 seconds of real time
|
|
0.0 seconds of user run time
|
|
0.0 seconds of system run time
|
|
0 page faults and
|
|
0 bytes consed.
|
|
0
|
|
19
|
|
#()
|
|
#()
|
|
* (time (foo "The quick brown fox"))
|
|
Compiling LAMBDA NIL:
|
|
Compiling Top-Level Form:
|
|
|
|
Evaluation took:
|
|
0.0 seconds of real time
|
|
0.0 seconds of user run time
|
|
0.0 seconds of system run time
|
|
0 page faults and
|
|
0 bytes consed.
|
|
0
|
|
19
|
|
#()
|
|
#()
|
|
*
|
|
</pre>
|
|
|
|
<p>
|
|
Of course, the usual rules for creating efficient regular expressions
|
|
apply to CL-PPCRE as well although it can optimize a couple of cases
|
|
itself. The most important rule is probably that you shouldn't use
|
|
capturing groups if you don't need the captured information, i.e. use
|
|
<code>"(?:a|b)*"</code> instead of
|
|
<code>"(a|b)*"</code> if you don't need to refer to the
|
|
register. (In fact, in this particular case CL-PPCRE will be able to
|
|
optimize away the register group, but it won't if you replace
|
|
<code>"a|b"</code> with, say,
|
|
<code>"a|bc"</code>.)
|
|
<p>
|
|
Another point worth mentioning is that you definitely should use
|
|
single-line mode if you have long strings without
|
|
<code>#\Newline</code> (or where you don't care about the line breaks)
|
|
and plan to use regular expressions like
|
|
<code>".*"</code>. See the <a href="#bench">benchmarks</a>
|
|
for comparisons between single-line mode and normal mode with such
|
|
target strings.
|
|
<p>
|
|
Another thing to consider is that, for performance reasons, CL-PPCRE
|
|
assumes that most of the target strings you're trying to match are <a
|
|
href="http://www.lispworks.com/reference/HyperSpec/Body/t_smp_st.htm">simple
|
|
strings</a> and coerces non-simple strings to simple strings before
|
|
scanning them. If you plan on working with non-simple strings mostly
|
|
you might consider modifying the CL-PPCRE source code. This is easy:
|
|
Change all occurences of <code>SCHAR</code> to <code>CHAR</code> and
|
|
redefine the macro in <code>util.lisp</code> where the coercion takes
|
|
place - that's all.
|
|
|
|
<br> <br><h3><a name="bugs" class=none>Bugs and problems</a></h3>
|
|
|
|
<h4><a name="stack" class=none>Stack overflow</a></h4>
|
|
|
|
CL-PPCRE can optimize away a lot of unnecessary backtracking but
|
|
sometimes this simply isn't possible. With complicated regular
|
|
expressions and long strings this might lead to stack overflows
|
|
depending on your machine and your CL implementation.
|
|
<p>
|
|
Here's one example with CLISP:
|
|
|
|
<pre>
|
|
[1]> (defun target (n) (concatenate 'string (make-string n :initial-element #\a) "b"))
|
|
TARGET
|
|
|
|
[2]> (cl-ppcre:scan "a*" (target 1000))
|
|
0 ;
|
|
1000 ;
|
|
#() ;
|
|
#()
|
|
|
|
[3]> (cl-ppcre:scan "(?:a|b)*" (target 1000))
|
|
0 ;
|
|
1001 ;
|
|
#() ;
|
|
#()
|
|
|
|
[4]> (cl-ppcre:scan "(a|b)*" (target 1000))
|
|
0 ;
|
|
1001 ;
|
|
#(1000) ;
|
|
#(1001)
|
|
|
|
[5]> (cl-ppcre:scan "(a|b)*" (target 10000))
|
|
0 ;
|
|
10001 ;
|
|
#(10000) ;
|
|
#(10001)
|
|
|
|
[6]> (cl-ppcre:scan "(a|b)*" (target 100000))
|
|
0 ;
|
|
100001 ;
|
|
#(100000) ;
|
|
#(100001)
|
|
|
|
[7]> (cl-ppcre:scan "(a|b)*" (target 1000000))
|
|
0 ;
|
|
1000001 ;
|
|
#(1000000) ;
|
|
#(1000001)
|
|
|
|
<font color=orange>;; No problem until now - but...</font>
|
|
|
|
[8]> (cl-ppcre:scan "(a|)*" (target 100000))
|
|
*** - Lisp stack overflow. RESET
|
|
|
|
[9]> (cl-ppcre:scan "(a|)*" (target 3200))
|
|
*** - Lisp stack overflow. RESET
|
|
</pre>
|
|
|
|
<p>
|
|
With CMUCL the situation is better and worse at the same time. It will
|
|
take a lot longer until CMUCL gives up but if it gives up the whole
|
|
Lisp image will silently die (at least on my machine):
|
|
|
|
<pre>
|
|
* (defun target (n) (concatenate 'string (make-string n :initial-element #\a) "b"))
|
|
TARGET
|
|
|
|
* (cl-ppcre:scan "(a|)*" (target 3200))
|
|
0
|
|
3200
|
|
#(3200)
|
|
#(3200)
|
|
|
|
* (cl-ppcre:scan "(a|)*" (target 10000))
|
|
0
|
|
10000
|
|
#(10000)
|
|
#(10000)
|
|
|
|
* (cl-ppcre:scan "(a|)*" (target 100000))
|
|
0
|
|
100000
|
|
#(100000)
|
|
#(100000)
|
|
|
|
* (cl-ppcre:scan "(a|)*" (target 1000000))
|
|
0
|
|
1000000
|
|
#(1000000)
|
|
#(1000000)
|
|
|
|
<font color=orange>;; No problem until now - but...</font>
|
|
|
|
* (cl-ppcre:scan "(a|)*" (target 10000000))
|
|
edi@bird:~ >
|
|
</pre>
|
|
|
|
This behaviour can be changed with <em>very</em> conservative optimization settings but that'll make CL-PPCRE crawl compared to Perl.
|
|
|
|
<p>
|
|
You might want to compare this to the way Perl handles the same situation. It might lie to you:
|
|
|
|
<pre>
|
|
edi@bird:~ > perl -le '$_="a" x 32766 . "b"; /(a|)*/; print $1'
|
|
|
|
edi@bird:~ > perl -le '$_="a" x 32767 . "b"; /(a|)*/; print $1'
|
|
a
|
|
</pre>
|
|
|
|
Or it might warn you before it's lying to you:
|
|
<pre>
|
|
edi@bird:~ > perl -lwe '$_="a" x 32767 . "b"; /(a|)*/; print $1'
|
|
Complex regular subexpression recursion limit (32766) exceeded at -e line 1.
|
|
a
|
|
</pre>
|
|
|
|
Or it might simply die:
|
|
<pre>
|
|
edi@bird:~ > /opt/perl-5.8/bin/perl -lwe '$_="a" x 32767 . "b"; /(a|)*/; print $1'
|
|
Segmentation fault
|
|
</pre>
|
|
|
|
Your mileage may vary, of course...
|
|
|
|
<h4><a name="quote" class=none><code>"\Q"</code> doesn't work, or does it?</a></h4>
|
|
|
|
In Perl the following code works as expected, i.e. it prints <code>1</code>.
|
|
<pre>
|
|
#!/usr/bin/perl -l
|
|
|
|
$a = '\E*';
|
|
print 1
|
|
if '\E*\E*' =~ /(?:\Q$a\E){2}/;
|
|
</pre>
|
|
|
|
If you try to do something similar in CL-PPCRE you get an error:
|
|
|
|
<pre>
|
|
* (let ((cl-ppcre:*allow-quoting* t)
|
|
(a "\\E*"))
|
|
(cl-ppcre:scan (concatenate 'string "(?:\\Q" a "\\E){2}") "\\E*\\E*"))
|
|
Quantifier '*' not allowed at position 3 in string "(?:*\\E){2}"
|
|
</pre>
|
|
|
|
The error message might give you a hint as to why this happens:
|
|
Because <a href="#*allow-quoting*"><code>*ALLOW-QUOTING*</code></a>
|
|
was <em>true</em> the concatenated string was pre-processed before it
|
|
was fed to CL-PPCRE's parser - the result of this pre-processing is
|
|
<code>"(?:*\\E){2}"</code> because the
|
|
<code>"\\E"</code> in the string <code>A</code> was taken to
|
|
be the end of the quoted section started by
|
|
<code>"\\Q"</code>. This cannot happen in Perl due to its
|
|
complicated interpolation rules - see <code>man perlop</code> for
|
|
the scary details. It <em>can</em> happen in CL-PPCRE, though.
|
|
Bummer!
|
|
<p>
|
|
What gives? <code>"\\Q...\\E"</code> in CL-PPCRE should only
|
|
be used in literal strings. If you want to quote arbitrary strings
|
|
try <a href="http://weitz.de/cl-interpol/">CL-INTERPOL</a> or use <a
|
|
href="#quote-meta-chars"><code>QUOTE-META-CHARS</code></a>:
|
|
<pre>
|
|
* (let ((a "\\E*"))
|
|
(cl-ppcre:scan (concatenate 'string
|
|
"(?:" (cl-ppcre:quote-meta-chars a) "){2}")
|
|
"\\E*\\E*"))
|
|
0
|
|
6
|
|
#()
|
|
#()
|
|
</pre>
|
|
Or, even better and Lisp-ier, use the <a href="#create-scanner2">S-expression syntax</a> instead - no need for quoting in this case:
|
|
<pre>
|
|
* (let ((a "\\E*"))
|
|
(cl-ppcre:scan `(:greedy-repetition 2 2 ,a)
|
|
"\\E*\\E*"))
|
|
0
|
|
6
|
|
#()
|
|
#()
|
|
</pre>
|
|
|
|
<h4><a name="backslash" class=none>Backslashes may confuse you...</a></h4>
|
|
|
|
<pre>
|
|
* (let ((a "y\\y"))
|
|
(cl-ppcre:scan a a))
|
|
NIL
|
|
</pre>
|
|
|
|
You didn't expect this to yield <code>NIL</code>, did you? Shouldn't something like <code>(CL-PPCRE:SCAN A A)</code> always return a true value? No, because the first and the second argument to <code>SCAN</code> are handled differently: The first argument is fed to CL-PPCRE's parser and is treated like a Perl regular expression. In particular, the parser "sees" <code>\y</code> and converts it to <code>y</code> because <code>\y</code> has no special meaning in regular expressions. So, the regular expression is the constant string <code>"yy"</code>. But the second argument isn't converted - it is left as is, i.e. it's equivalent to Perl's <code>'y\y'</code>. In other words, this example would be equivalent to the Perl code
|
|
|
|
<pre>
|
|
'y\y' =~ /y\y/;
|
|
</pre>
|
|
|
|
or to
|
|
|
|
<pre>
|
|
$a = 'y\y';
|
|
$a =~ /$a/;
|
|
</pre>
|
|
|
|
which should explain why it doesn't match.
|
|
<p>
|
|
Still confused? You might want to try <a href="http://weitz.de/cl-interpol/">CL-INTERPOL</a>.
|
|
|
|
<br> <br><h3><a class=none name="remarks">Remarks</a></h3>
|
|
|
|
The sample output from CMUCL and CLISP has been slightly edited to
|
|
increase readability.
|
|
<p>
|
|
All test cases and benchmarks in this document where performed on an
|
|
IBM Thinkpad T23 laptop (Pentium III 1.2 GHz,
|
|
768 MB RAM) running <a href="http://www.gentoo.org/">Gentoo
|
|
Linux</a> 1.1a.
|
|
|
|
<br> <br><h3><a class=none name="ack">Acknowledgements</a></h3>
|
|
|
|
Although I didn't use their code I was heavily inspired by looking at
|
|
the Scheme/CL regex implementations of <a
|
|
href="http://www.ccs.neu.edu/home/dorai/pregexp/pregexp.html">Dorai
|
|
Sitaram</a> and <a
|
|
href="http://www.geocities.com/mparker762/clawk#regex">Michael
|
|
Parker</a>. Also, the nice folks from CMUCL's <a
|
|
href="http://www.cons.org/cmucl/support.html">mailing list</a> as well
|
|
as the output of Perl's <code>use re "debug"</code> pragma
|
|
have been very helpful in optimizing the scanners created by CL-PPCRE.
|
|
|
|
<p>
|
|
The asdf system definitions were kindly provided by Marco
|
|
Baringer. Hannu Koivisto provided patches to make the
|
|
<code>.system</code> files more usable. Thanks to Kevin Rosenberg and
|
|
Douglas Crosher for pointing out how to be friendly to case-sensitive
|
|
ACL images. Thanks to Karsten Poeck and JP Massar for their help in
|
|
making CL-PPCRE work with Corman Lisp. JP Massar and Kent M. Pitman
|
|
also helped to improve/fix the test suite and the compiler macro.
|
|
|
|
<p>
|
|
Thanks to the guys at "Café Olé" in Hamburg
|
|
where I wrote most of the code and thanks to my wife for lending me
|
|
her PowerBook to test CL-PPCRE with MCL and OpenMCL.
|
|
|
|
<p>
|
|
$Header: /home/manuel/bknr-cvs/cvs/thirdparty/cl-ppcre/doc/index.html,v 1.1 2004/06/23 08:27:10 hans Exp $
|
|
<p><a href="http://weitz.de/index.html">BACK TO MY HOMEPAGE</a>
|
|
|
|
</body>
|
|
</html> |