Added testing with SHA3 test vectors from Keccak Code Package.

The final FIPS 202 SHA-3 standard mandates the prepending of a
01 suffix to the message prior to padding, which the original
Keccak submission did not specify.  This change adjusts all
sha3 wrappers to behave standard conforming, and adds an optional
keyword argument raw-keccak-p to specify the original treatment.
Fixes #2.

.gitattributes

@@ -1,2 +1,2 @@
-/*.lisp ident
-/*.asd ident
+*.lisp ident
+*.asd ident

.gitignore

@@ -2,6 +2,7 @@
 *.fas
 *.fasl
 *.ofasl
+*.64ofasl
 *.nfasl
 *.xfasl
 *.dx32fsl

COPYING

@@ -1,4 +1,4 @@
-  Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai
+  Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai
 
   Permission is hereby granted, free of charge, to any person obtaining
   a copy of this software and associated documentation files (the

NEWS

@@ -1,3 +1,30 @@
+Release 1.1.0
+=============
+
+ * Change sha3 functions to match the final FIPS 202 SHA3 standard,
+   by appending a 01 suffix to messages prior to digest calculation.
+   The old Keccak submission behavior can be retained by passing in
+   the new raw-keccak-p keyword argument with true to the relevant
+   functions.  Thanks to David McClain for prompting this change.
+
+ * Fixes a bug in the bug fix of release 1.0.2 for the sha3-update
+   handling of not completely filled buffers reported by David McClain,
+   which could lead to out-of-bounds accesses in calls to the
+   pad-message-to-width function.
+
+Release 1.0.2
+=============
+
+ * Fixes a bug reported by Orivej Desh where two or more calls to
+   sha3-update which didn't fill the buffer could lead to the second
+   and later updates being ignored, thereby creating wrong message
+   digests.
+
+Release 1.0.1
+=============
+
+ * Change to 32bit implementation for 64bit LispWorks.
+
 Release 1.0.0
 =============
 

README

@@ -3,6 +3,23 @@ This library is an implementation of the Secure Hash Algorithm 3
 messages with an integral number of octets, i.e. sub-byte length
 messages are not supported.
 
+NOTE that prior to release 1.0.2 this package had a bug in the
+generation of message digests where multiple calls to sha3-update
+with partial buffers could lead to input data being ignored and
+therefore erroneous message digests being generated.  Uses with
+only one call to sha3-update and the high-level routines were not
+affected by this bug.
+
+NOTE that prior to release 1.1.0 this package computed digests
+based on the Keccak submission to the SHA-3 contest and did not
+yet take into account the added suffix that the FIPS 202 SHA-3
+final standard adds to messages prior to calculating the digest,
+since this was not part of the Keccak submission.  Starting with
+1.1.0 the functions in the sha3 package do by default calculate
+disgests that match the FIPS 202 standard, and will calculate
+the old pre-standard digests only if the new optional keyword
+argument :raw-keccak-p is passed with a true value.
+
 The code should be portable across nearly all ANSI compliant CL
 implementations with specialized versions tuned for implementations
 that offer unboxed 64bit arithmetic, unboxed 32bit arithmetic and for
@@ -44,11 +61,14 @@ The mid-level interfaces to the digest routines are the functions
   simple-array with element-type (unsigned-byte 8), bounded by `start'
   and `end', which must be numeric bounding-indices.
 
-- sha3:sha3-final state &key output-bit-length
+- sha3:sha3-final state &key output-bit-length raw-keccak-p
   
   If the given SHA-3 state `state' has not already been finalized,
   finalize it by processing any remaining input in its buffer, with
-  suitable padding as specified by the SHA-3 standard.  Returns the
+  the specified suffix of 01 and suitable padding as specified by the
+  SHA-3 standard (the specified SHA-3 suffix can be elided with the
+  optional keyword argument `raw-keccak-p' to generate digests as the
+  initial Keccak submission would have generated).  Returns the
   message digest as a simple-array of (unsigned-byte 8).  The length
   of the returned digest is determined either by the output bit length
   or bit rate specified on state creation, or for the special case of
@@ -60,28 +80,43 @@ For convenience the following high-level functions produce digests in
 one step from 1d simple-arrays and streams with element-type
 (unsigned-byte 8), as well as files:
 
-- sha3:sha3-digest-vector vector &key (start 0) end (output-bit-length 512)
+- sha3:sha3-digest-vector vector &key (start 0) end (output-bit-length 512) raw-keccak-p
   
   Calculate an SHA-3 message-digest of data in `vector', which should
   be a 1d simple-array with element type (unsigned-byte 8), bounded by
   `start' and `end'.  The bit length of the message digest produced is
   controlled by `output-bit-length', which can take on the values 224,
-  256, 288, 384 and 512, which is the default value.
+  256, 288, 384 and 512, which is the default value.  Using the optional
+  `raw-keccak-p' keyword argument the SHA-3 mandated 01 suffix that is
+  appended to the actual message prior to padding can be elided to yield
+  message digests that match the original Keccak submission instead of
+  the actual SHA-3 standard.  Use this option only for compatibility
+  with historical implementations.
 
-- sha3:sha3-digest-stream stream &key (output-bit-length 512)
+- sha3:sha3-digest-stream stream &key (output-bit-length 512) raw-keccak-p
   
   Calculate an SHA-3 message-digest of data read from `stream', which
   should be a stream with element type (unsigned-byte 8).  The bit
   length of the message digest produced is controlled by
   `output-bit-length', which can take on the values 224, 256, 288, 384
-  and 512, which is the default value.
+  and 512, which is the default value.  Using the optional `raw-keccak-p'
+  keyword argument the SHA-3 mandated 01 suffix that is appended to the
+  actual message prior to padding can be elided to yield message digests
+  that match the original Keccak submission instead of the actual SHA-3
+  standard.  Use this option only for compatibility with historical
+  implementations.  
 
-- sha3:sha3-digest-file pathname &key (output-bit-length 512)
+- sha3:sha3-digest-file pathname &key (output-bit-length 512) raw-keccak-p
   
   Calculate an SHA-3 message-digest of the file specified by
   `pathname'.  The bit length of the message digest produced is
   controlled by `output-bit-length', which can take on the values 224,
-  256, 288, 384 and 512, which is the default value.
+  256, 288, 384 and 512, which is the default value.  Using the optional
+  `raw-keccak-p' keyword argument the SHA-3 mandated 01 suffix that is
+  appended to the actual message prior to padding can be elided to yield
+  message digests that match the original Keccak submission instead of
+  the actual SHA-3 standard.  Use this option only for compatibility
+  with historical implementations.
 
 Note that in order to generate a message digest of a string it will
 have to be converted to a simple-array with element-type
@@ -98,6 +133,16 @@ The testcases from the Keccak test data can be run with the following
 form:
 
 (keccak:test-keccak-msgkat 
+ "/Path/To/MsgKatDirectory"
+ (lambda (total-bits bit-rate output-bits message)
+   (declare (ignore total-bits bit-rate))
+   (sha3:sha3-digest-vector message :output-bit-length output-bits :raw-keccak-p t)))
+
+The adapted SHA-3 testcases from the Keccak Code Package test vectors
+available under https://github.com/gvanas/KeccakCodePackage/tree/master/TestVectors
+can be run with the following form:
+
+(keccak:test-sha3-msgkat 
  "/Path/To/MsgKatDirectory"
  (lambda (total-bits bit-rate output-bits message)
    (declare (ignore total-bits bit-rate))

common.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -181,13 +181,17 @@ Only supports atoms and function forms, no special forms."
 ;;; Message Padding for last block
 ;;;
 
-(defun pad-message-to-width (message bit-width)
+(defun pad-message-to-width (message bit-width add-fips-202-suffix-p)
   "Destructively pad the given message to the given bit-width according to 
-Keccak padding rules and return the padded message."
+the Keccak 10*1 padding rules, optionally appending the FIPS 202/SHA-3
+mandated 01 suffix first, and return the padded message."
   (let ((message-byte-length (length message))
         (width-bytes (truncate bit-width 8)))
     (setq message (adjust-array message (list width-bytes)))
-    (setf (aref message message-byte-length) #x01)
+    ;; FIPS 202 SHA-3 mandates the appending of a 01 suffix prior to the
+    ;; final Keccak padding so that the first byte following the message
+    ;; will be #b00000101 instead of #b00000001 for raw Keccak.
+    (setf (aref message message-byte-length) (if add-fips-202-suffix-p #x06 #x01))
     (loop for index from (1+ message-byte-length) below width-bytes
           do (setf (aref message index) #x00)
           finally

keccak-16bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-32bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-64bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-reference.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -36,7 +36,8 @@
            #:test-with-testsuite
            #:read-testsuite-from-file
            #:test-with-testsuite-from-file
-           #:test-keccak-msgkat))
+           #:test-keccak-msgkat
+           #:test-sha3-msgkat))
 
 (cl:in-package #:keccak-reference)
 
@@ -380,3 +381,22 @@
        (setq result nil))
         finally
      (return result)))
+
+(defun test-sha3-msgkat (directory &optional function)
+  (loop with result = t
+        for (filename total-bits bit-rate output-bits) in
+        '(("ShortMsgKAT_SHA3-224.txt" 1600 1152 224)
+          ("ShortMsgKAT_SHA3-256.txt" 1600 1088 256)
+          ("ShortMsgKAT_SHA3-384.txt" 1600  832 384)
+          ("ShortMsgKAT_SHA3-512.txt" 1600  576 512))
+        do
+     (unless
+         (test-with-testsuite-from-file
+          (merge-pathnames filename directory)
+          (if (null function)
+              (lambda (message) (keccak total-bits bit-rate output-bits message))
+              (lambda (message)
+                (funcall function total-bits bit-rate output-bits message))))
+       (setq result nil))
+        finally
+     (return result)))

pkgdef.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

sha3.asd

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -42,7 +42,7 @@
   :author "Pierre R. Mai <pmai@pmsf.de>"
   :maintainer "Pierre R. Mai <pmai@pmsf.de>"
   :licence "MIT/X11"
-  :version "1.0.0"
+  :version "1.1.0"
   #+sbcl :depends-on #+sbcl ("sb-rotate-byte")
   :components ((:file "pkgdef")
                (:file "common" :depends-on ("pkgdef"))
@@ -50,9 +50,11 @@
                (:file "keccak-64bit" :depends-on ("pkgdef" "common"))
                #+(or (and :sbcl (not (or :x86-64 :alpha))) 
                      :cmucl
-                     (and :ccl :64-bit-target))
+                     (and :ccl :64-bit-target)
+                     (and :lispworks :lispworks-64bit))
                (:file "keccak-32bit" :depends-on ("pkgdef" "common"))
-               #-(or :sbcl :cmucl (and :ccl :64-bit-target))
+               #-(or :sbcl :cmucl (and :ccl :64-bit-target) 
+                     (and :lispworks :lispworks-64bit))
                (:file "keccak-16bit" :depends-on ("pkgdef" "common"))
                (:file "sha3" 
                       :depends-on ("pkgdef" 
@@ -61,7 +63,9 @@
                                    "keccak-64bit"
                                    #+(or (and :sbcl (not (or :x86-64 :alpha)))
                                          :cmucl
-                                         (and :ccl :64-bit-target))
+                                         (and :ccl :64-bit-target)
+                                         (and :lispworks :lispworks-64bit))
                                    "keccak-32bit"
-                                   #-(or :sbcl :cmucl (and :ccl :64-bit-target))
+                                   #-(or :sbcl :cmucl (and :ccl :64-bit-target)
+                                         (and :lispworks :lispworks-64bit))
                                    "keccak-16bit"))))

sha3.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -110,16 +110,21 @@ and `end', which must be numeric bounding-indices."
              #.*optimize-declaration*)
     ;; Handle potential remaining bytes
     (unless (zerop buffer-index)
-      (let ((remainder (- (length buffer) buffer-index)))
-        (declare (type fixnum remainder))
+      (let ((remainder (- rate-bytes buffer-index))
+            (length (- end start)))
+        (declare (type fixnum remainder length))
         (replace buffer vector :start1 buffer-index :start2 start :end2 end)
-        (when (>= (- end start) remainder)
-          (keccak-state-merge-input keccak-state bit-rate buffer 0)
-          (keccak-f keccak-state))
+        ;; Return if still unfilled buffer
+        (when (< length remainder)
+          (incf (sha3-state-buffer-index state) length)
+          (return-from sha3-update))
+        ;; Else handle now complete buffer
+        (keccak-state-merge-input keccak-state bit-rate buffer 0)
+        (keccak-f keccak-state)
         (setf (sha3-state-buffer-index state) 0
-              start (min (+ start remainder) end))))
+              start (+ start remainder))))
     ;; Now handle full blocks, stuff any remainder into buffer
-    (loop for block-offset of-type fixnum from start to end by rate-bytes
+    (loop for block-offset of-type fixnum from start below end by rate-bytes
           do
        (cond
          ((<= (+ block-offset rate-bytes) end)
@@ -129,10 +134,13 @@ and `end', which must be numeric bounding-indices."
           (replace buffer vector :start1 0 :start2 block-offset)
           (setf (sha3-state-buffer-index state) (- end block-offset)))))))
 
-(defun sha3-final (state &key (output-bit-length nil output-bit-length-p))
+(defun sha3-final (state &key (output-bit-length nil output-bit-length-p) raw-keccak-p)
   "If the given SHA-3 state `state' has not already been finalized,
 finalize it by processing any remaining input in its buffer, with
-suitable padding as specified by the SHA-3 standard.  Returns the
+the specified suffix of 01 and suitable padding as specified by the
+SHA-3 standard (the specified SHA-3 suffix can be elided with the
+optional keyword argument `raw-keccak-p' to generate digests as the
+initial Keccak submission would have generated).   Returns the
 message digest as a simple-array of (unsigned-byte 8).  The length
 of the returned digest is determined either by the output bit length
 or bit rate specified on state creation, or for the special case of
@@ -172,7 +180,8 @@ the function will return the digest again."
        (keccak-state-merge-input keccak-state bit-rate 
                                  (pad-message-to-width
                                   (subseq buffer 0 buffer-index)
-                                  bit-rate)
+                                  bit-rate
+                                  (not raw-keccak-p))
                                  0)
        (keccak-f keccak-state)
        (setf (sha3-state-buffer-index state) 0
@@ -184,12 +193,18 @@ the function will return the digest again."
 ;;;
 
 (defun sha3-digest-vector (vector &key (start 0) end 
-                           (output-bit-length 512))
+                           (output-bit-length 512)
+                           raw-keccak-p)
   "Calculate an SHA-3 message-digest of data in `vector', which should
 be a 1d simple-array with element type (unsigned-byte 8), bounded by
 `start' and `end'.  The bit length of the message digest produced is
 controlled by `output-bit-length', which can take on the values 224,
-256, 288, 384 and 512, which is the default value."
+256, 288, 384 and 512, which is the default value.  Using the optional
+`raw-keccak-p' keyword argument the SHA-3 mandated 01 suffix that is
+appended to the actual message prior to padding can be elided to yield
+message digests that match the original Keccak submission instead of
+the actual SHA-3 standard.  Use this option only for compatibility
+with historical implementations."
   (declare (optimize (speed 3) (safety 3) (space 0) (debug 1))
            (type (simple-array (unsigned-byte 8) (*)) vector)
            (type fixnum start)
@@ -202,7 +217,7 @@ controlled by `output-bit-length', which can take on the values 224,
       (let ((real-end (or end (length vector))))
         (declare (type fixnum real-end))
         (sha3-update state vector :start start :end real-end))
-      (sha3-final state))))
+      (sha3-final state :raw-keccak-p raw-keccak-p))))
 
 (eval-when (:compile-toplevel :load-toplevel :execute)
   (defconstant +buffer-size+ (* 128 1024)
@@ -211,12 +226,17 @@ controlled by `output-bit-length', which can take on the values 224,
 
 (deftype buffer-index () `(integer 0 ,+buffer-size+))
 
-(defun sha3-digest-stream (stream &key (output-bit-length 512))
+(defun sha3-digest-stream (stream &key (output-bit-length 512) raw-keccak-p)
   "Calculate an SHA-3 message-digest of data read from `stream', which
 should be a stream with element type (unsigned-byte 8).  The bit
 length of the message digest produced is controlled by
 `output-bit-length', which can take on the values 224, 256, 288, 384
-and 512, which is the default value."
+and 512, which is the default value.  Using the optional `raw-keccak-p'
+keyword argument the SHA-3 mandated 01 suffix that is appended to the
+actual message prior to padding can be elided to yield message digests
+that match the original Keccak submission instead of the actual SHA-3
+standard.  Use this option only for compatibility with historical
+implementations."
   (declare (optimize (speed 3) (safety 3) (space 0) (debug 1))
            (type stream stream)
            (type (integer 0 1600) output-bit-length))
@@ -233,16 +253,21 @@ and 512, which is the default value."
             do (sha3-update state buffer :end bytes)
             until (< bytes +buffer-size+)
             finally
-         (return (sha3-final state))))))
+         (return (sha3-final state :raw-keccak-p raw-keccak-p))))))
 
-(defun sha3-digest-file (pathname &key (output-bit-length 512))
+(defun sha3-digest-file (pathname &key (output-bit-length 512) raw-keccak-p)
   "Calculate an SHA-3 message-digest of the file specified by
 `pathname'.  The bit length of the message digest produced is
 controlled by `output-bit-length', which can take on the values 224,
-256, 288, 384 and 512, which is the default value."
+256, 288, 384 and 512, which is the default value.  Using the optional
+`raw-keccak-p' keyword argument the SHA-3 mandated 01 suffix that is
+appended to the actual message prior to padding can be elided to yield
+message digests that match the original Keccak submission instead of
+the actual SHA-3 standard.  Use this option only for compatibility
+with historical implementations."
   (declare (optimize (speed 3) (safety 3) (space 0) (debug 1))
            (type (integer 0 1600) output-bit-length))
   (locally
       (declare (optimize (safety 1) (debug 0)))
     (with-open-file (stream pathname :element-type '(unsigned-byte 8))
-      (sha3-digest-stream stream :output-bit-length output-bit-length))))
+      (sha3-digest-stream stream :output-bit-length output-bit-length :raw-keccak-p raw-keccak-p))))