The problem was test case for EMSA-PSS verify failed *sometimes*, like once in 100 times or once in a month. It was really annoying and made this procedure unreliable.
Because of this random frequency, I suspected it was random number generator. In the test case, it uses secure random number generator so that it generate different number each time. (well, thank god I used secure random, otherwise I would never notice this bug.)
So first step to fix this bug was create a proper (improper?) state of PRNG. To create it, I made this code;
(import (crypto) (math) (getopt)) (define key-pair (generate-key-pair RSA :size 512 :prng (pseudo-random RC4))) (define valid-rsa-message (string->utf8 "test message")) (define prng (pseudo-random RC4)) (with-args (command-line) ((c (#\c "count") #t "1")) (let ((count (string->number c))) (do ((i 0 (+ i 1)) (r (read-random-bytes prng 100) (read-random-bytes prng 100))) ((= i count) r)))) (let* ((rsa-sign-cipher (cipher RSA (keypair-private key-pair))) (rsa-verify-cipher (cipher RSA (keypair-public key-pair))) (em (sign rsa-sign-cipher valid-rsa-message :prng prng))) (verify rsa-verify-cipher valid-rsa-message em))And this shell script;
#!/bin/sh for i in `seq 1 $1` do count=`expr $i + 100` echo $count `sash -Lext/crypto crypto.scm -c $count` doneThen ran the script and check which number was the key number! After the inspection, the number was 181.
Now, it's time for debug. once I could find the PRNG state, it was really simple to fix. The problem was the signed message's first 2 bytes. RSA operation deletes left most 0's so verify procedure needs to add removed 0's in front of the message. However previous implementation did not add more than 2 zeros. That was the problem.
So I modified to add propert 0's in front of the message, and now it works!
I hope Sagittarius is now a bit more reliable. Even though I have no idea if it was the only problem that causes test case failed.