Switch to 16bit implementation for Lispworks8 64bit

This avoids an optimization problem in keccak-state-merge-input and is
also slightly faster due to less consing in this variant.

.gitattributes

@@ -1,2 +1,2 @@
-/*.lisp ident
-/*.asd ident
+*.lisp ident
+*.asd ident

.travis.yml

@@ -0,0 +1,27 @@
+language: lisp
+sudo: required
+
+env:
+  matrix:
+    - LISP=abcl
+    - LISP=allegro
+    - LISP=sbcl
+    - LISP=sbcl32
+    - LISP=ccl
+    - LISP=ccl32
+    - LISP=clisp
+    - LISP=clisp32
+    - LISP=ecl
+
+install:
+  - curl -L https://github.com/luismbo/cl-travis/raw/master/install.sh | sh
+  - git clone https://github.com/gvanas/KeccakCodePackage.git
+
+script:
+  - cl -e '(asdf:load-system "sha3")
+           (load (compile-file "keccak-reference.lisp"))
+           (unless (keccak:test-sha3-msgkat "KeccakCodePackage/TestVectors/"
+                     (lambda (cl-user::total-bits cl-user::bit-rate cl-user::output-bits cl-user::message)
+                       (declare (ignore cl-user::total-bits cl-user::bit-rate))
+                       (sha3:sha3-digest-vector cl-user::message :output-bit-length cl-user::output-bits)))
+             (uiop:quit 1))'

COPYING

@@ -1,4 +1,4 @@
-  Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai
+  Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai
 
   Permission is hereby granted, free of charge, to any person obtaining
   a copy of this software and associated documentation files (the

NEWS

@@ -1,3 +1,40 @@
+Release 1.1.2
+=============
+
+ * Switch to 16bit implementation for 64bit LispWorks 8, to avoid
+   compiler optimization problems in keccak-state-merge-input, and
+   also slightly higher performance due to non-consing.
+
+Release 1.1.1
+=============
+
+ * Fix a bug in pad-message-to-width on ACL, which would lead to
+   completely erroneous results being calculated for all inputs
+   due to differences in the result of adjust-array between ACL
+   and all other currently supported implementations.
+
+Release 1.1.0
+=============
+
+ * Change sha3 functions to match the final FIPS 202 SHA3 standard,
+   by appending a 01 suffix to messages prior to digest calculation.
+   The old Keccak submission behavior can be retained by passing in
+   the new raw-keccak-p keyword argument with true to the relevant
+   functions.  Thanks to David McClain for prompting this change.
+
+ * Fixes a bug in the bug fix of release 1.0.2 for the sha3-update
+   handling of not completely filled buffers reported by David McClain,
+   which could lead to out-of-bounds accesses in calls to the
+   pad-message-to-width function.
+
+Release 1.0.2
+=============
+
+ * Fixes a bug reported by Orivej Desh where two or more calls to
+   sha3-update which didn't fill the buffer could lead to the second
+   and later updates being ignored, thereby creating wrong message
+   digests.
+
 Release 1.0.1
 =============
 

README

@@ -1,114 +0,0 @@
-This library is an implementation of the Secure Hash Algorithm 3
-(SHA-3), also known as Keccak.  The implementation is constrained to
-messages with an integral number of octets, i.e. sub-byte length
-messages are not supported.
-
-The code should be portable across nearly all ANSI compliant CL
-implementations with specialized versions tuned for implementations
-that offer unboxed 64bit arithmetic, unboxed 32bit arithmetic and for
-implementations with efficient fixnum arithmetic (requiring fixnums
-that can represent (unsigned-byte 16)).  Especially the 64 and 32bit
-implementations have been mostly optimized for SBCL and CMU CL.  For
-those implementations, digests with a 1024 bit-rate (and 288 bit
-digest output) can be generated in between 30 (64bit SBCL) to around
-100 (32bit CMU CL) cycles/byte on an i7-640M; whereas optimized
-C/assembler implementations reach around 12 to 50 cycles/byte on 64/32
-bit Intel hardware.  The reason for the discrepancy probably lies in
-missing peephole and dependency optimizations in the SBCL/CMU CL
-compiler backend.
-
-The mid-level interfaces to the digest routines are the functions
-
-- sha3:sha3-init &key output-bit-length bit-rate
-  
-  Create and return a new SHA-3 state.  If `output-bit-length' is
-  specified then the state will run at the bit rate specified for the
-  given output bit length.  If `output-bit-length' is unspecified,
-  `bit-rate' can be specified to select a suitable bit rate.  If both
-  are left unspecified then a default bit rate of 1024 bits is
-  selected, which is suitable for arbitrary output bit lengths of up
-  to 288 bits.
-
-- sha3:sha3-copy state
-  
-  Return an independent copy of the SHA-3 state `state'.
-
-- sha3:sha3-state-p state
-  
-  Test whether a given object is a SHA-3 state, i.e. is an instance of
-  the class sha3:sha3-state.
-
-- sha3:sha3-update state vector &key (start 0) (end (length vector))
-  
-  Update the given SHA-3 state `state' from `vector', which must be a
-  simple-array with element-type (unsigned-byte 8), bounded by `start'
-  and `end', which must be numeric bounding-indices.
-
-- sha3:sha3-final state &key output-bit-length
-  
-  If the given SHA-3 state `state' has not already been finalized,
-  finalize it by processing any remaining input in its buffer, with
-  suitable padding as specified by the SHA-3 standard.  Returns the
-  message digest as a simple-array of (unsigned-byte 8).  The length
-  of the returned digest is determined either by the output bit length
-  or bit rate specified on state creation, or for the special case of
-  default parameters being used, by the optional keyword argument
-  `output-bit-length'.  If the state has previously been finalized,
-  the function will return the digest again.
-
-For convenience the following high-level functions produce digests in
-one step from 1d simple-arrays and streams with element-type
-(unsigned-byte 8), as well as files:
-
-- sha3:sha3-digest-vector vector &key (start 0) end (output-bit-length 512)
-  
-  Calculate an SHA-3 message-digest of data in `vector', which should
-  be a 1d simple-array with element type (unsigned-byte 8), bounded by
-  `start' and `end'.  The bit length of the message digest produced is
-  controlled by `output-bit-length', which can take on the values 224,
-  256, 288, 384 and 512, which is the default value.
-
-- sha3:sha3-digest-stream stream &key (output-bit-length 512)
-  
-  Calculate an SHA-3 message-digest of data read from `stream', which
-  should be a stream with element type (unsigned-byte 8).  The bit
-  length of the message digest produced is controlled by
-  `output-bit-length', which can take on the values 224, 256, 288, 384
-  and 512, which is the default value.
-
-- sha3:sha3-digest-file pathname &key (output-bit-length 512)
-  
-  Calculate an SHA-3 message-digest of the file specified by
-  `pathname'.  The bit length of the message digest produced is
-  controlled by `output-bit-length', which can take on the values 224,
-  256, 288, 384 and 512, which is the default value.
-
-Note that in order to generate a message digest of a string it will
-have to be converted to a simple-array with element-type
-(unsigned-byte 8) in the proper output-encoding.  This will have to
-rely on implementation-specific functions and is not part of the SHA3
-library.
-
-The file keccak-reference.lisp contains a slow simple reference
-implementation, and testdriver code, which allows testing of the tuned
-implementations against this reference and against test data available
-from the Keccak Site at: http://keccak.noekeon.org/KeccakKAT-3.zip
-
-The testcases from the Keccak test data can be run with the following
-form:
-
-(keccak:test-keccak-msgkat 
- "/Path/To/MsgKatDirectory"
- (lambda (total-bits bit-rate output-bits message)
-   (declare (ignore total-bits bit-rate))
-   (sha3:sha3-digest-vector message :output-bit-length output-bits)))
-
-This SHA-3 implementation is licensed under the MIT-style license
-contained in the file COPYING and the header of each source file.
-Many thanks go to the Keccak Team (Guido Bertoni, Joan Daemen, Michaël
-Peeters and Gilles Van Assche, cf. http://keccak.noekeon.org) for
-their algorithm and excellent documentation and reference
-implementations.
-
-Please direct any feedback to pmai@pmsf.de.  A git repository of this
-library is available under git://github.com/pmai/sha3.git

README.md

@@ -0,0 +1,168 @@
+[![Build Status](https://travis-ci.org/pmai/sha3.svg?branch=master)](https://travis-ci.org/pmai/sha3)
+
+This library is an implementation of the Secure Hash Algorithm 3
+([SHA-3][]), also known as [Keccak][].  The implementation is
+constrained to messages with an integral number of octets, i.e.
+sub-byte length messages are not supported.
+
+**NOTE** that prior to release 1.0.2 this package had a bug in the
+generation of message digests where multiple calls to sha3-update
+with partial buffers could lead to input data being ignored and
+therefore erroneous message digests being generated.  Uses with
+only one call to sha3-update and the high-level routines were not
+affected by this bug.
+
+**NOTE** that prior to release 1.1.0 this package computed digests
+based on the Keccak submission to the SHA-3 contest and did not
+yet take into account the added suffix that the FIPS 202 SHA-3
+final standard adds to messages prior to calculating the digest,
+since this was not part of the Keccak submission.  Starting with
+1.1.0 the functions in the sha3 package do by default calculate
+disgests that match the FIPS 202 standard, and will calculate
+the old pre-standard digests only if the new optional keyword
+argument `:raw-keccak-p` is passed with a true value.
+
+The code should be portable across nearly all ANSI compliant CL
+implementations with specialized versions tuned for implementations
+that offer unboxed 64bit arithmetic, unboxed 32bit arithmetic and for
+implementations with efficient fixnum arithmetic (requiring fixnums
+that can represent (unsigned-byte 16)).  Especially the 64 and 32bit
+implementations have been mostly optimized for SBCL and CMU CL.  For
+those implementations, digests with a 1024 bit-rate (and 288 bit
+digest output) can be generated in between 30 (64bit SBCL) to around
+100 (32bit CMU CL) cycles/byte on an i7-640M; whereas optimized
+C/assembler implementations reach around 12 to 50 cycles/byte on 64/32
+bit Intel hardware.  The reason for the discrepancy probably lies in
+missing peephole and dependency optimizations in the SBCL/CMU CL
+compiler backend.
+
+The mid-level interfaces to the digest routines are the functions
+
+- `sha3:sha3-init &key output-bit-length bit-rate`
+  
+  Create and return a new SHA-3 state.  If `output-bit-length` is
+  specified then the state will run at the bit rate specified for the
+  given output bit length.  If `output-bit-length` is unspecified,
+  `bit-rate` can be specified to select a suitable bit rate.  If both
+  are left unspecified then a default bit rate of 1024 bits is
+  selected, which is suitable for arbitrary output bit lengths of up
+  to 288 bits.
+
+- `sha3:sha3-copy state`
+  
+  Return an independent copy of the SHA-3 state `state`.
+
+- `sha3:sha3-state-p state`
+  
+  Test whether a given object is a SHA-3 state, i.e. is an instance of
+  the class `sha3:sha3-state`.
+
+- `sha3:sha3-update state vector &key (start 0) (end (length vector))`
+  
+  Update the given SHA-3 state `state` from `vector`, which must be a
+  simple-array with element-type (unsigned-byte 8), bounded by `start`
+  and `end`, which must be numeric bounding-indices.
+
+- `sha3:sha3-final state &key output-bit-length raw-keccak-p`
+  
+  If the given SHA-3 state `state` has not already been finalized,
+  finalize it by processing any remaining input in its buffer, with
+  the specified suffix of 01 and suitable padding as specified by the
+  SHA-3 standard (the specified SHA-3 suffix can be elided with the
+  optional keyword argument `raw-keccak-p` to generate digests as the
+  initial Keccak submission would have generated).  Returns the
+  message digest as a `simple-array` of `(unsigned-byte 8)`.  The
+  length of the returned digest is determined either by the output bit
+  length or bit rate specified on state creation, or for the special
+  case of default parameters being used, by the optional keyword
+  argument `output-bit-length`.  If the state has previously been
+  finalized, the function will return the digest again.
+
+For convenience the following high-level functions produce digests in
+one step from 1d simple-arrays and streams with element-type
+`(unsigned-byte 8)`, as well as files:
+
+- `sha3:sha3-digest-vector vector &key (start 0) end (output-bit-length 512) raw-keccak-p`
+  
+  Calculate an SHA-3 message-digest of data in `vector`, which should
+  be a 1d simple-array with element type `(unsigned-byte 8)`, bounded
+  by `start` and `end`.  The bit length of the message digest produced
+  is controlled by `output-bit-length`, which can take on the values
+  224, 256, 288, 384 and 512, which is the default value.  Using the
+  optional `raw-keccak-p` keyword argument the SHA-3 mandated 01 suffix
+  that is appended to the actual message prior to padding can be elided
+  to yield message digests that match the original Keccak submission
+  instead of the actual SHA-3 standard.  Use this option only for
+  compatibility with historical implementations.
+
+- `sha3:sha3-digest-stream stream &key (output-bit-length 512) raw-keccak-p`
+  
+  Calculate an SHA-3 message-digest of data read from `stream`, which
+  should be a `stream` with element type `(unsigned-byte 8)`. The bit
+  length of the message digest produced is controlled by
+  `output-bit-length`, which can take on the values 224, 256, 288, 384
+  and 512, which is the default value. Using the optional `raw-keccak-p`
+  keyword argument the SHA-3 mandated 01 suffix that is appended to the
+  actual message prior to padding can be elided to yield message digests
+  that match the original Keccak submission instead of the actual SHA-3
+  standard.  Use this option only for compatibility with historical
+  implementations.  
+
+- `sha3:sha3-digest-file pathname &key (output-bit-length 512) raw-keccak-p`
+  
+  Calculate an SHA-3 message-digest of the file specified by
+  `pathname`.  The bit length of the message digest produced is
+  controlled by `output-bit-length`, which can take on the values 224,
+  256, 288, 384 and 512, which is the default value. Using the optional
+  `raw-keccak-p` keyword argument the SHA-3 mandated 01 suffix that is
+  appended to the actual message prior to padding can be elided to yield
+  message digests that match the original Keccak submission instead of
+  the actual SHA-3 standard.  Use this option only for compatibility
+  with historical implementations.
+
+Note that in order to generate a message digest of a string it will
+have to be converted to a `simple-array` with element-type
+`(unsigned-byte 8)` in the proper output-encoding.  This will have to
+rely on implementation-specific functions and is not part of the SHA3
+library.
+
+The file `keccak-reference.lisp` contains a slow simple reference
+implementation, and testdriver code, which allows testing of the tuned
+implementations against this reference and against test data available
+from the Keccak Site at: http://keccak.noekeon.org/KeccakKAT-3.zip
+
+The testcases from the Keccak test data can be run with the following
+form:
+
+```lisp
+(keccak:test-keccak-msgkat 
+ "/Path/To/MsgKatDirectory"
+ (lambda (total-bits bit-rate output-bits message)
+   (declare (ignore total-bits bit-rate))
+   (sha3:sha3-digest-vector message :output-bit-length output-bits :raw-keccak-p t)))
+```
+
+The adapted SHA-3 testcases from the Keccak Code Package test vectors
+available under https://github.com/gvanas/KeccakCodePackage/tree/master/TestVectors
+can be run with the following form:
+
+```lisp
+(keccak:test-sha3-msgkat 
+ "/Path/To/MsgKatDirectory"
+ (lambda (total-bits bit-rate output-bits message)
+   (declare (ignore total-bits bit-rate))
+   (sha3:sha3-digest-vector message :output-bit-length output-bits)))
+```
+
+This SHA-3 implementation is licensed under the MIT-style license
+contained in the file COPYING and the header of each source file.
+Many thanks go to the Keccak Team (Guido Bertoni, Joan Daemen, Michaël
+Peeters and Gilles Van Assche, cf. http://keccak.team) for
+their algorithm and excellent documentation and reference
+implementations.
+
+Please direct any feedback to pmai@pmsf.de.  A git repository of this
+library is available under git://github.com/pmai/sha3.git
+
+[SHA-3]: https://en.wikipedia.org/wiki/SHA-3
+[Keccak]: https://keccak.team/

common.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -181,16 +181,22 @@ Only supports atoms and function forms, no special forms."
 ;;; Message Padding for last block
 ;;;
 
-(defun pad-message-to-width (message bit-width)
+(defun pad-message-to-width (message bit-width add-fips-202-suffix-p)
   "Destructively pad the given message to the given bit-width according to 
-Keccak padding rules and return the padded message."
-  (let ((message-byte-length (length message))
-        (width-bytes (truncate bit-width 8)))
-    (setq message (adjust-array message (list width-bytes)))
-    (setf (aref message message-byte-length) #x01)
+the Keccak 10*1 padding rules, optionally appending the FIPS 202/SHA-3
+mandated 01 suffix first, and return the padded message."
+  (let* ((message-byte-length (length message))
+         (width-bytes (truncate bit-width 8))
+         (padded (make-array width-bytes
+                             :element-type '(unsigned-byte 8))))
+    (replace padded message)
+    ;; FIPS 202 SHA-3 mandates the appending of a 01 suffix prior to the
+    ;; final Keccak padding so that the first byte following the message
+    ;; will be #b00000101 instead of #b00000001 for raw Keccak.
+    (setf (aref padded message-byte-length) (if add-fips-202-suffix-p #x06 #x01))
     (loop for index from (1+ message-byte-length) below width-bytes
-          do (setf (aref message index) #x00)
+          do (setf (aref padded index) #x00)
           finally
-       (setf (aref message (1- width-bytes))
-             (logior #x80 (aref message (1- width-bytes))))))
-  message)
+       (setf (aref padded (1- width-bytes))
+             (logior #x80 (aref padded (1- width-bytes)))))
+    padded))

keccak-16bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-32bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-64bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-reference.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -36,7 +36,8 @@
            #:test-with-testsuite
            #:read-testsuite-from-file
            #:test-with-testsuite-from-file
-           #:test-keccak-msgkat))
+           #:test-keccak-msgkat
+           #:test-sha3-msgkat))
 
 (cl:in-package #:keccak-reference)
 
@@ -380,3 +381,22 @@
        (setq result nil))
         finally
      (return result)))
+
+(defun test-sha3-msgkat (directory &optional function)
+  (loop with result = t
+        for (filename total-bits bit-rate output-bits) in
+        '(("ShortMsgKAT_SHA3-224.txt" 1600 1152 224)
+          ("ShortMsgKAT_SHA3-256.txt" 1600 1088 256)
+          ("ShortMsgKAT_SHA3-384.txt" 1600  832 384)
+          ("ShortMsgKAT_SHA3-512.txt" 1600  576 512))
+        do
+     (unless
+         (test-with-testsuite-from-file
+          (merge-pathnames filename directory)
+          (if (null function)
+              (lambda (message) (keccak total-bits bit-rate output-bits message))
+              (lambda (message)
+                (funcall function total-bits bit-rate output-bits message))))
+       (setq result nil))
+        finally
+     (return result)))

pkgdef.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

sha3.asd

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -26,7 +26,7 @@
 ;;;; other dealings in this Software without prior written authorization
 ;;;; from the author.
 ;;;; 
-;;;; $Id: 2cbc5857639b13160ef580514a78749c02b70c0b $
+;;;; $Id$
 
 (cl:in-package #:cl-user)
 
@@ -42,7 +42,7 @@
   :author "Pierre R. Mai <pmai@pmsf.de>"
   :maintainer "Pierre R. Mai <pmai@pmsf.de>"
   :licence "MIT/X11"
-  :version "1.0.0"
+  :version "1.1.2"
   #+sbcl :depends-on #+sbcl ("sb-rotate-byte")
   :components ((:file "pkgdef")
                (:file "common" :depends-on ("pkgdef"))
@@ -51,10 +51,10 @@
                #+(or (and :sbcl (not (or :x86-64 :alpha))) 
                      :cmucl
                      (and :ccl :64-bit-target)
-                     (and :lispworks :lispworks-64bit))
+                     (and :lispworks :lispworks-64bit (not :lispworks8)))
                (:file "keccak-32bit" :depends-on ("pkgdef" "common"))
                #-(or :sbcl :cmucl (and :ccl :64-bit-target) 
-                     (and :lispworks :lispworks-64bit))
+                     (and :lispworks :lispworks-64bit (not :lispworks8)))
                (:file "keccak-16bit" :depends-on ("pkgdef" "common"))
                (:file "sha3" 
                       :depends-on ("pkgdef" 
@@ -64,8 +64,10 @@
                                    #+(or (and :sbcl (not (or :x86-64 :alpha)))
                                          :cmucl
                                          (and :ccl :64-bit-target)
-                                         (and :lispworks :lispworks-64bit))
+                                         (and :lispworks :lispworks-64bit
+					      (not :lispworks8)))
                                    "keccak-32bit"
                                    #-(or :sbcl :cmucl (and :ccl :64-bit-target)
-                                         (and :lispworks :lispworks-64bit))
+                                         (and :lispworks :lispworks-64bit
+					      (not :lispworks8)))
                                    "keccak-16bit"))))

sha3.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -110,16 +110,21 @@ and `end', which must be numeric bounding-indices."
              #.*optimize-declaration*)
     ;; Handle potential remaining bytes
     (unless (zerop buffer-index)
-      (let ((remainder (- (length buffer) buffer-index)))
-        (declare (type fixnum remainder))
+      (let ((remainder (- rate-bytes buffer-index))
+            (length (- end start)))
+        (declare (type fixnum remainder length))
         (replace buffer vector :start1 buffer-index :start2 start :end2 end)
-        (when (>= (- end start) remainder)
-          (keccak-state-merge-input keccak-state bit-rate buffer 0)
-          (keccak-f keccak-state))
+        ;; Return if still unfilled buffer
+        (when (< length remainder)
+          (incf (sha3-state-buffer-index state) length)
+          (return-from sha3-update))
+        ;; Else handle now complete buffer
+        (keccak-state-merge-input keccak-state bit-rate buffer 0)
+        (keccak-f keccak-state)
         (setf (sha3-state-buffer-index state) 0
-              start (min (+ start remainder) end))))
+              start (+ start remainder))))
     ;; Now handle full blocks, stuff any remainder into buffer
-    (loop for block-offset of-type fixnum from start to end by rate-bytes
+    (loop for block-offset of-type fixnum from start below end by rate-bytes
           do
        (cond
          ((<= (+ block-offset rate-bytes) end)
@@ -129,10 +134,13 @@ and `end', which must be numeric bounding-indices."
           (replace buffer vector :start1 0 :start2 block-offset)
           (setf (sha3-state-buffer-index state) (- end block-offset)))))))
 
-(defun sha3-final (state &key (output-bit-length nil output-bit-length-p))
+(defun sha3-final (state &key (output-bit-length nil output-bit-length-p) raw-keccak-p)
   "If the given SHA-3 state `state' has not already been finalized,
 finalize it by processing any remaining input in its buffer, with
-suitable padding as specified by the SHA-3 standard.  Returns the
+the specified suffix of 01 and suitable padding as specified by the
+SHA-3 standard (the specified SHA-3 suffix can be elided with the
+optional keyword argument `raw-keccak-p' to generate digests as the
+initial Keccak submission would have generated).   Returns the
 message digest as a simple-array of (unsigned-byte 8).  The length
 of the returned digest is determined either by the output bit length
 or bit rate specified on state creation, or for the special case of
@@ -172,7 +180,8 @@ the function will return the digest again."
        (keccak-state-merge-input keccak-state bit-rate 
                                  (pad-message-to-width
                                   (subseq buffer 0 buffer-index)
-                                  bit-rate)
+                                  bit-rate
+                                  (not raw-keccak-p))
                                  0)
        (keccak-f keccak-state)
        (setf (sha3-state-buffer-index state) 0
@@ -184,12 +193,18 @@ the function will return the digest again."
 ;;;
 
 (defun sha3-digest-vector (vector &key (start 0) end 
-                           (output-bit-length 512))
+                           (output-bit-length 512)
+                           raw-keccak-p)
   "Calculate an SHA-3 message-digest of data in `vector', which should
 be a 1d simple-array with element type (unsigned-byte 8), bounded by
 `start' and `end'.  The bit length of the message digest produced is
 controlled by `output-bit-length', which can take on the values 224,
-256, 288, 384 and 512, which is the default value."
+256, 288, 384 and 512, which is the default value.  Using the optional
+`raw-keccak-p' keyword argument the SHA-3 mandated 01 suffix that is
+appended to the actual message prior to padding can be elided to yield
+message digests that match the original Keccak submission instead of
+the actual SHA-3 standard.  Use this option only for compatibility
+with historical implementations."
   (declare (optimize (speed 3) (safety 3) (space 0) (debug 1))
            (type (simple-array (unsigned-byte 8) (*)) vector)
            (type fixnum start)
@@ -202,7 +217,7 @@ controlled by `output-bit-length', which can take on the values 224,
       (let ((real-end (or end (length vector))))
         (declare (type fixnum real-end))
         (sha3-update state vector :start start :end real-end))
-      (sha3-final state))))
+      (sha3-final state :raw-keccak-p raw-keccak-p))))
 
 (eval-when (:compile-toplevel :load-toplevel :execute)
   (defconstant +buffer-size+ (* 128 1024)
@@ -211,12 +226,17 @@ controlled by `output-bit-length', which can take on the values 224,
 
 (deftype buffer-index () `(integer 0 ,+buffer-size+))
 
-(defun sha3-digest-stream (stream &key (output-bit-length 512))
+(defun sha3-digest-stream (stream &key (output-bit-length 512) raw-keccak-p)
   "Calculate an SHA-3 message-digest of data read from `stream', which
 should be a stream with element type (unsigned-byte 8).  The bit
 length of the message digest produced is controlled by
 `output-bit-length', which can take on the values 224, 256, 288, 384
-and 512, which is the default value."
+and 512, which is the default value.  Using the optional `raw-keccak-p'
+keyword argument the SHA-3 mandated 01 suffix that is appended to the
+actual message prior to padding can be elided to yield message digests
+that match the original Keccak submission instead of the actual SHA-3
+standard.  Use this option only for compatibility with historical
+implementations."
   (declare (optimize (speed 3) (safety 3) (space 0) (debug 1))
            (type stream stream)
            (type (integer 0 1600) output-bit-length))
@@ -233,16 +253,21 @@ and 512, which is the default value."
             do (sha3-update state buffer :end bytes)
             until (< bytes +buffer-size+)
             finally
-         (return (sha3-final state))))))
+         (return (sha3-final state :raw-keccak-p raw-keccak-p))))))
 
-(defun sha3-digest-file (pathname &key (output-bit-length 512))
+(defun sha3-digest-file (pathname &key (output-bit-length 512) raw-keccak-p)
   "Calculate an SHA-3 message-digest of the file specified by
 `pathname'.  The bit length of the message digest produced is
 controlled by `output-bit-length', which can take on the values 224,
-256, 288, 384 and 512, which is the default value."
+256, 288, 384 and 512, which is the default value.  Using the optional
+`raw-keccak-p' keyword argument the SHA-3 mandated 01 suffix that is
+appended to the actual message prior to padding can be elided to yield
+message digests that match the original Keccak submission instead of
+the actual SHA-3 standard.  Use this option only for compatibility
+with historical implementations."
   (declare (optimize (speed 3) (safety 3) (space 0) (debug 1))
            (type (integer 0 1600) output-bit-length))
   (locally
       (declare (optimize (safety 1) (debug 0)))
     (with-open-file (stream pathname :element-type '(unsigned-byte 8))
-      (sha3-digest-stream stream :output-bit-length output-bit-length))))
+      (sha3-digest-stream stream :output-bit-length output-bit-length :raw-keccak-p raw-keccak-p))))