Decrypting AES-GCM in multiple parts loses plaintext. Each C_DecryptUpdate after the first returns at most ulTagBits/8 bytes however much it owes — and returns CKR_OK while doing so, so the loss is silent. Only the tag check in C_DecryptFinal notices, with CKR_ENCRYPTED_DATA_INVALID. A caller that does not check that return gets truncated plaintext and no error.
v1.5.3, --features pqc, Ubuntu 26.04 / OpenSSL 3.5.5, linux/amd64. Also on v1.5.2.
Reproducing
Attached gcm_multipart.c (single file, -ldl only) encrypts 128 bytes with one C_Encrypt, then decrypts the 144 byte ciphertext in 64 byte chunks. The plaintext is a counter, 00 01 02 ..., so the dumps below read directly.
Each chunk is a single C_DecryptUpdate into a large buffer, not the NULL-probe-then-call convention of §5.2 — the failure does not depend on that.
cc -I <pkcs11 headers> -o gcm_multipart gcm_multipart.c -ldl
./gcm_multipart /usr/lib/pkcs11/libkryoptic_pkcs11.so <pin>
pkcs11-spy trace
Full log attached. The three updates and the final call:
9: C_DecryptUpdate [in] 64 [out] 48 CKR_OK
00 01 02 03 ... 2D 2E 2F correct
10: C_DecryptUpdate [in] 64 [out] 16 CKR_OK
30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F
stops at 0x3F, owed through 0x6F
11: C_DecryptUpdate [in] 16 [out] 16 CKR_OK
B3 2F BE 22 CC 8C 32 E1 18 70 58 94 34 BE 19 EC
not plaintext
12: C_DecryptFinal CKR_ENCRYPTED_DATA_INVALID
Call 10 returns correct plaintext but stops 48 bytes early. By call 11 the buffering is out of step: the 16 bytes supplied are the authentication tag (C6 90 C4 40 ..., the last 16 bytes of the ciphertext), and they are decrypted and returned as if they were ciphertext — leaving nothing for C_DecryptFinal to verify. Those output bytes depend on the randomly generated key, so they differ between runs; the attached log is the run quoted here.
Totals: 48 + 16 + 16 = 80 bytes returned for 128 bytes of plaintext. Expected 48, 64 and 16, each holding back the 16 byte tag.
Other failing combinations
Found while running p11ex's test suite against kryoptic; 17 of 54 data-size × chunk-size combinations fail. Chunks ≤ 16 always pass, since the correct output never exceeds the cap. Larger chunks fail unless the whole stream fits in one update, or every later update happens to owe exactly 16 bytes.
At 128 bytes of plaintext:
| chunk |
updates returned |
expected |
| 17 |
[1,16,…,16,8] = 121 |
128 |
| 32 |
[16,16,16,16,16] = 80 |
[16,32,32,32,16] |
| 48 |
[32,16,16] = 64 |
[32,48,48] |
| 64 |
[48,16,16] = 80 |
[48,64,16] |
| 100 |
[84,16] = 100 |
[84,44] |
| 256 |
[128] = 128 |
passes — single update |
Expected
Updates return all available plaintext, holding back only the trailing ulTagBits/8 bytes, and C_DecryptFinal verifies the tag.
Attachments
Decrypting AES-GCM in multiple parts loses plaintext. Each
C_DecryptUpdateafter the first returns at mostulTagBits/8bytes however much it owes — and returnsCKR_OKwhile doing so, so the loss is silent. Only the tag check inC_DecryptFinalnotices, withCKR_ENCRYPTED_DATA_INVALID. A caller that does not check that return gets truncated plaintext and no error.v1.5.3,
--features pqc, Ubuntu 26.04 / OpenSSL 3.5.5, linux/amd64. Also on v1.5.2.Reproducing
Attached
gcm_multipart.c(single file,-ldlonly) encrypts 128 bytes with oneC_Encrypt, then decrypts the 144 byte ciphertext in 64 byte chunks. The plaintext is a counter,00 01 02 ..., so the dumps below read directly.Each chunk is a single
C_DecryptUpdateinto a large buffer, not the NULL-probe-then-call convention of §5.2 — the failure does not depend on that.pkcs11-spy trace
Full log attached. The three updates and the final call:
Call 10 returns correct plaintext but stops 48 bytes early. By call 11 the buffering is out of step: the 16 bytes supplied are the authentication tag (
C6 90 C4 40 ..., the last 16 bytes of the ciphertext), and they are decrypted and returned as if they were ciphertext — leaving nothing forC_DecryptFinalto verify. Those output bytes depend on the randomly generated key, so they differ between runs; the attached log is the run quoted here.Totals: 48 + 16 + 16 = 80 bytes returned for 128 bytes of plaintext. Expected 48, 64 and 16, each holding back the 16 byte tag.
Other failing combinations
Found while running p11ex's test suite against kryoptic; 17 of 54 data-size × chunk-size combinations fail. Chunks ≤ 16 always pass, since the correct output never exceeds the cap. Larger chunks fail unless the whole stream fits in one update, or every later update happens to owe exactly 16 bytes.
At 128 bytes of plaintext:
[1,16,…,16,8]= 121[16,16,16,16,16]= 80[16,32,32,32,16][32,16,16]= 64[32,48,48][48,16,16]= 80[48,64,16][84,16]= 100[84,44][128]= 128Expected
Updates return all available plaintext, holding back only the trailing
ulTagBits/8bytes, andC_DecryptFinalverifies the tag.Attachments