Switch to 16bit implementation for Lispworks8 64bit

This avoids an optimization problem in keccak-state-merge-input and is
also slightly faster due to less consing in this variant.

.travis.yml

@@ -0,0 +1,27 @@
+language: lisp
+sudo: required
+
+env:
+  matrix:
+    - LISP=abcl
+    - LISP=allegro
+    - LISP=sbcl
+    - LISP=sbcl32
+    - LISP=ccl
+    - LISP=ccl32
+    - LISP=clisp
+    - LISP=clisp32
+    - LISP=ecl
+
+install:
+  - curl -L https://github.com/luismbo/cl-travis/raw/master/install.sh | sh
+  - git clone https://github.com/gvanas/KeccakCodePackage.git
+
+script:
+  - cl -e '(asdf:load-system "sha3")
+           (load (compile-file "keccak-reference.lisp"))
+           (unless (keccak:test-sha3-msgkat "KeccakCodePackage/TestVectors/"
+                     (lambda (cl-user::total-bits cl-user::bit-rate cl-user::output-bits cl-user::message)
+                       (declare (ignore cl-user::total-bits cl-user::bit-rate))
+                       (sha3:sha3-digest-vector cl-user::message :output-bit-length cl-user::output-bits)))
+             (uiop:quit 1))'

COPYING

@@ -1,4 +1,4 @@
-  Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai
+  Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai
 
   Permission is hereby granted, free of charge, to any person obtaining
   a copy of this software and associated documentation files (the

NEWS

@@ -1,3 +1,18 @@
+Release 1.1.2
+=============
+
+ * Switch to 16bit implementation for 64bit LispWorks 8, to avoid
+   compiler optimization problems in keccak-state-merge-input, and
+   also slightly higher performance due to non-consing.
+
+Release 1.1.1
+=============
+
+ * Fix a bug in pad-message-to-width on ACL, which would lead to
+   completely erroneous results being calculated for all inputs
+   due to differences in the result of adjust-array between ACL
+   and all other currently supported implementations.
+
 Release 1.1.0
 =============
 

README.md

@@ -1,16 +1,18 @@
-This library is an implementation of the Secure Hash Algorithm 3
-(SHA-3), also known as Keccak.  The implementation is constrained to
-messages with an integral number of octets, i.e. sub-byte length
-messages are not supported.
+[![Build Status](https://travis-ci.org/pmai/sha3.svg?branch=master)](https://travis-ci.org/pmai/sha3)
 
-NOTE that prior to release 1.0.2 this package had a bug in the
+This library is an implementation of the Secure Hash Algorithm 3
+([SHA-3][]), also known as [Keccak][].  The implementation is
+constrained to messages with an integral number of octets, i.e.
+sub-byte length messages are not supported.
+
+**NOTE** that prior to release 1.0.2 this package had a bug in the
 generation of message digests where multiple calls to sha3-update
 with partial buffers could lead to input data being ignored and
 therefore erroneous message digests being generated.  Uses with
 only one call to sha3-update and the high-level routines were not
 affected by this bug.
 
-NOTE that prior to release 1.1.0 this package computed digests
+**NOTE** that prior to release 1.1.0 this package computed digests
 based on the Keccak submission to the SHA-3 contest and did not
 yet take into account the added suffix that the FIPS 202 SHA-3
 final standard adds to messages prior to calculating the digest,
@@ -18,7 +20,7 @@ since this was not part of the Keccak submission.  Starting with
 1.1.0 the functions in the sha3 package do by default calculate
 disgests that match the FIPS 202 standard, and will calculate
 the old pre-standard digests only if the new optional keyword
-argument :raw-keccak-p is passed with a true value.
+argument `:raw-keccak-p` is passed with a true value.
 
 The code should be portable across nearly all ANSI compliant CL
 implementations with specialized versions tuned for implementations
@@ -36,95 +38,95 @@ compiler backend.
 
 The mid-level interfaces to the digest routines are the functions
 
-- sha3:sha3-init &key output-bit-length bit-rate
+- `sha3:sha3-init &key output-bit-length bit-rate`
   
-  Create and return a new SHA-3 state.  If `output-bit-length' is
+  Create and return a new SHA-3 state.  If `output-bit-length` is
   specified then the state will run at the bit rate specified for the
-  given output bit length.  If `output-bit-length' is unspecified,
-  `bit-rate' can be specified to select a suitable bit rate.  If both
+  given output bit length.  If `output-bit-length` is unspecified,
+  `bit-rate` can be specified to select a suitable bit rate.  If both
   are left unspecified then a default bit rate of 1024 bits is
   selected, which is suitable for arbitrary output bit lengths of up
   to 288 bits.
 
-- sha3:sha3-copy state
+- `sha3:sha3-copy state`
   
-  Return an independent copy of the SHA-3 state `state'.
+  Return an independent copy of the SHA-3 state `state`.
 
-- sha3:sha3-state-p state
+- `sha3:sha3-state-p state`
   
   Test whether a given object is a SHA-3 state, i.e. is an instance of
-  the class sha3:sha3-state.
+  the class `sha3:sha3-state`.
 
-- sha3:sha3-update state vector &key (start 0) (end (length vector))
+- `sha3:sha3-update state vector &key (start 0) (end (length vector))`
   
-  Update the given SHA-3 state `state' from `vector', which must be a
-  simple-array with element-type (unsigned-byte 8), bounded by `start'
-  and `end', which must be numeric bounding-indices.
+  Update the given SHA-3 state `state` from `vector`, which must be a
+  simple-array with element-type (unsigned-byte 8), bounded by `start`
+  and `end`, which must be numeric bounding-indices.
 
-- sha3:sha3-final state &key output-bit-length raw-keccak-p
+- `sha3:sha3-final state &key output-bit-length raw-keccak-p`
   
-  If the given SHA-3 state `state' has not already been finalized,
+  If the given SHA-3 state `state` has not already been finalized,
   finalize it by processing any remaining input in its buffer, with
   the specified suffix of 01 and suitable padding as specified by the
   SHA-3 standard (the specified SHA-3 suffix can be elided with the
-  optional keyword argument `raw-keccak-p' to generate digests as the
+  optional keyword argument `raw-keccak-p` to generate digests as the
   initial Keccak submission would have generated).  Returns the
-  message digest as a simple-array of (unsigned-byte 8).  The length
-  of the returned digest is determined either by the output bit length
-  or bit rate specified on state creation, or for the special case of
-  default parameters being used, by the optional keyword argument
-  `output-bit-length'.  If the state has previously been finalized,
-  the function will return the digest again.
+  message digest as a `simple-array` of `(unsigned-byte 8)`.  The
+  length of the returned digest is determined either by the output bit
+  length or bit rate specified on state creation, or for the special
+  case of default parameters being used, by the optional keyword
+  argument `output-bit-length`.  If the state has previously been
+  finalized, the function will return the digest again.
 
 For convenience the following high-level functions produce digests in
 one step from 1d simple-arrays and streams with element-type
-(unsigned-byte 8), as well as files:
+`(unsigned-byte 8)`, as well as files:
 
-- sha3:sha3-digest-vector vector &key (start 0) end (output-bit-length 512) raw-keccak-p
+- `sha3:sha3-digest-vector vector &key (start 0) end (output-bit-length 512) raw-keccak-p`
   
-  Calculate an SHA-3 message-digest of data in `vector', which should
-  be a 1d simple-array with element type (unsigned-byte 8), bounded by
-  `start' and `end'.  The bit length of the message digest produced is
-  controlled by `output-bit-length', which can take on the values 224,
-  256, 288, 384 and 512, which is the default value.  Using the optional
-  `raw-keccak-p' keyword argument the SHA-3 mandated 01 suffix that is
-  appended to the actual message prior to padding can be elided to yield
-  message digests that match the original Keccak submission instead of
-  the actual SHA-3 standard.  Use this option only for compatibility
-  with historical implementations.
+  Calculate an SHA-3 message-digest of data in `vector`, which should
+  be a 1d simple-array with element type `(unsigned-byte 8)`, bounded
+  by `start` and `end`.  The bit length of the message digest produced
+  is controlled by `output-bit-length`, which can take on the values
+  224, 256, 288, 384 and 512, which is the default value.  Using the
+  optional `raw-keccak-p` keyword argument the SHA-3 mandated 01 suffix
+  that is appended to the actual message prior to padding can be elided
+  to yield message digests that match the original Keccak submission
+  instead of the actual SHA-3 standard.  Use this option only for
+  compatibility with historical implementations.
 
-- sha3:sha3-digest-stream stream &key (output-bit-length 512) raw-keccak-p
+- `sha3:sha3-digest-stream stream &key (output-bit-length 512) raw-keccak-p`
   
-  Calculate an SHA-3 message-digest of data read from `stream', which
-  should be a stream with element type (unsigned-byte 8).  The bit
+  Calculate an SHA-3 message-digest of data read from `stream`, which
+  should be a `stream` with element type `(unsigned-byte 8)`. The bit
   length of the message digest produced is controlled by
-  `output-bit-length', which can take on the values 224, 256, 288, 384
-  and 512, which is the default value.  Using the optional `raw-keccak-p'
+  `output-bit-length`, which can take on the values 224, 256, 288, 384
+  and 512, which is the default value. Using the optional `raw-keccak-p`
   keyword argument the SHA-3 mandated 01 suffix that is appended to the
   actual message prior to padding can be elided to yield message digests
   that match the original Keccak submission instead of the actual SHA-3
   standard.  Use this option only for compatibility with historical
   implementations.  
 
-- sha3:sha3-digest-file pathname &key (output-bit-length 512) raw-keccak-p
+- `sha3:sha3-digest-file pathname &key (output-bit-length 512) raw-keccak-p`
   
   Calculate an SHA-3 message-digest of the file specified by
-  `pathname'.  The bit length of the message digest produced is
-  controlled by `output-bit-length', which can take on the values 224,
-  256, 288, 384 and 512, which is the default value.  Using the optional
-  `raw-keccak-p' keyword argument the SHA-3 mandated 01 suffix that is
+  `pathname`.  The bit length of the message digest produced is
+  controlled by `output-bit-length`, which can take on the values 224,
+  256, 288, 384 and 512, which is the default value. Using the optional
+  `raw-keccak-p` keyword argument the SHA-3 mandated 01 suffix that is
   appended to the actual message prior to padding can be elided to yield
   message digests that match the original Keccak submission instead of
   the actual SHA-3 standard.  Use this option only for compatibility
   with historical implementations.
 
 Note that in order to generate a message digest of a string it will
-have to be converted to a simple-array with element-type
-(unsigned-byte 8) in the proper output-encoding.  This will have to
+have to be converted to a `simple-array` with element-type
+`(unsigned-byte 8)` in the proper output-encoding.  This will have to
 rely on implementation-specific functions and is not part of the SHA3
 library.
 
-The file keccak-reference.lisp contains a slow simple reference
+The file `keccak-reference.lisp` contains a slow simple reference
 implementation, and testdriver code, which allows testing of the tuned
 implementations against this reference and against test data available
 from the Keccak Site at: http://keccak.noekeon.org/KeccakKAT-3.zip
@@ -132,28 +134,35 @@ from the Keccak Site at: http://keccak.noekeon.org/KeccakKAT-3.zip
 The testcases from the Keccak test data can be run with the following
 form:
 
+```lisp
 (keccak:test-keccak-msgkat 
  "/Path/To/MsgKatDirectory"
  (lambda (total-bits bit-rate output-bits message)
    (declare (ignore total-bits bit-rate))
    (sha3:sha3-digest-vector message :output-bit-length output-bits :raw-keccak-p t)))
+```
 
 The adapted SHA-3 testcases from the Keccak Code Package test vectors
 available under https://github.com/gvanas/KeccakCodePackage/tree/master/TestVectors
 can be run with the following form:
 
+```lisp
 (keccak:test-sha3-msgkat 
  "/Path/To/MsgKatDirectory"
  (lambda (total-bits bit-rate output-bits message)
    (declare (ignore total-bits bit-rate))
    (sha3:sha3-digest-vector message :output-bit-length output-bits)))
+```
 
 This SHA-3 implementation is licensed under the MIT-style license
 contained in the file COPYING and the header of each source file.
 Many thanks go to the Keccak Team (Guido Bertoni, Joan Daemen, Michaël
-Peeters and Gilles Van Assche, cf. http://keccak.noekeon.org) for
+Peeters and Gilles Van Assche, cf. http://keccak.team) for
 their algorithm and excellent documentation and reference
 implementations.
 
 Please direct any feedback to pmai@pmsf.de.  A git repository of this
 library is available under git://github.com/pmai/sha3.git
+
+[SHA-3]: https://en.wikipedia.org/wiki/SHA-3
+[Keccak]: https://keccak.team/

common.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -185,16 +185,18 @@ Only supports atoms and function forms, no special forms."
   "Destructively pad the given message to the given bit-width according to 
 the Keccak 10*1 padding rules, optionally appending the FIPS 202/SHA-3
 mandated 01 suffix first, and return the padded message."
-  (let ((message-byte-length (length message))
-        (width-bytes (truncate bit-width 8)))
-    (setq message (adjust-array message (list width-bytes)))
+  (let* ((message-byte-length (length message))
+         (width-bytes (truncate bit-width 8))
+         (padded (make-array width-bytes
+                             :element-type '(unsigned-byte 8))))
+    (replace padded message)
     ;; FIPS 202 SHA-3 mandates the appending of a 01 suffix prior to the
     ;; final Keccak padding so that the first byte following the message
     ;; will be #b00000101 instead of #b00000001 for raw Keccak.
-    (setf (aref message message-byte-length) (if add-fips-202-suffix-p #x06 #x01))
+    (setf (aref padded message-byte-length) (if add-fips-202-suffix-p #x06 #x01))
     (loop for index from (1+ message-byte-length) below width-bytes
-          do (setf (aref message index) #x00)
+          do (setf (aref padded index) #x00)
           finally
-       (setf (aref message (1- width-bytes))
-             (logior #x80 (aref message (1- width-bytes))))))
-  message)
+       (setf (aref padded (1- width-bytes))
+             (logior #x80 (aref padded (1- width-bytes)))))
+    padded))

keccak-16bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-32bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-64bit.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

keccak-reference.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

pkgdef.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the

sha3.asd

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the
@@ -42,7 +42,7 @@
   :author "Pierre R. Mai <pmai@pmsf.de>"
   :maintainer "Pierre R. Mai <pmai@pmsf.de>"
   :licence "MIT/X11"
-  :version "1.1.0"
+  :version "1.1.2"
   #+sbcl :depends-on #+sbcl ("sb-rotate-byte")
   :components ((:file "pkgdef")
                (:file "common" :depends-on ("pkgdef"))
@@ -51,10 +51,10 @@
                #+(or (and :sbcl (not (or :x86-64 :alpha))) 
                      :cmucl
                      (and :ccl :64-bit-target)
-                     (and :lispworks :lispworks-64bit))
+                     (and :lispworks :lispworks-64bit (not :lispworks8)))
                (:file "keccak-32bit" :depends-on ("pkgdef" "common"))
                #-(or :sbcl :cmucl (and :ccl :64-bit-target) 
-                     (and :lispworks :lispworks-64bit))
+                     (and :lispworks :lispworks-64bit (not :lispworks8)))
                (:file "keccak-16bit" :depends-on ("pkgdef" "common"))
                (:file "sha3" 
                       :depends-on ("pkgdef" 
@@ -64,8 +64,10 @@
                                    #+(or (and :sbcl (not (or :x86-64 :alpha)))
                                          :cmucl
                                          (and :ccl :64-bit-target)
-                                         (and :lispworks :lispworks-64bit))
+                                         (and :lispworks :lispworks-64bit
+					      (not :lispworks8)))
                                    "keccak-32bit"
                                    #-(or :sbcl :cmucl (and :ccl :64-bit-target)
-                                         (and :lispworks :lispworks-64bit))
+                                         (and :lispworks :lispworks-64bit
+					      (not :lispworks8)))
                                    "keccak-16bit"))))

sha3.lisp

@@ -1,6 +1,6 @@
 ;;;; SHA3 --- Secure Hash Algorithm 3 (Keccak) Implementation
 ;;;;
-;;;; Copyright (C) 2012 -- 2016 PMSF IT Consulting Pierre R. Mai.
+;;;; Copyright (C) 2012 -- 2018 PMSF IT Consulting Pierre R. Mai.
 ;;;;
 ;;;; Permission is hereby granted, free of charge, to any person obtaining
 ;;;; a copy of this software and associated documentation files (the