c - getting a segmentation fault with RSA_public_encrypt

Question

Here's my code:

#include <openssl/evp.h>
#include <openssl/pem.h>
#include <string.h>

int main (void)
{
        char publicKey[] = "-----BEGIN RSA PUBLIC KEY-----
"
"MIIBCgKCAQEAqBa0jeqfHO8CFZuHyN5WgdTd3uThkU/I9lR+bb2R9khVU1tYhiyL
"
"Gfnm051K0039gCBAz9S7HHO2lqR/B1kiEGwBzg83N3tL2UzQXXbpxJz0b7vV8rXZ
"
"fB9j+FCAtD5znittXRjWrNqlEyJyxK+PuxNF3uPGWSjkYtRKv2f5HCXsfRJah40w
"
"5zU+OKZsZMrSSxweYN9Jwc++oMaSPCsGKog6NPbkbcTK8Akveof8v5rpG50I/zcQ
"
"9dCD69qJcmhm4ai4fAZFJlsFH2HmxV6DERFs3TNyzGlSQ1gd/XLER7U9nlrOUT87
"
"mXreUIXSkAFdCrlHXHOzj0eodfN8IfHjnQIDAQAB
"
"-----END RSA PUBLIC KEY-----
";
    char plaintext[] = "enp6";

    // base64 decode plaintext
    EVP_DecodeBlock(plaintext, plaintext, strlen(plaintext));

    BIO* bo = BIO_new(BIO_s_mem());
    BIO_write(bo, publicKey, strlen(publicKey));
    EVP_PKEY* pkey = 0;
    PEM_read_bio_PUBKEY(bo, NULL, NULL, NULL);
    BIO_free(bo);
    RSA* rsa = EVP_PKEY_get1_RSA(pkey);
    char ciphertext[RSA_size(rsa)];

    RSA_public_encrypt(strlen(plaintext), plaintext, ciphertext, rsa, RSA_NO_PADDING);
    printf("%d
", (int) strlen(ciphertext));
}

I'm trying to run it by doing gcc -x c test2.c -lcrypto && ./a.out. Any ideas as to what I'm doing wrong?

question from:https://stackoverflow.com/questions/65905203/getting-a-segmentation-fault-with-rsa-public-encrypt

Answer 1

Your code contains several mistakes. The direct cause of the segfault is the fact that the variable pkey is never assigned any value after initializing it with 0. EVP_PKEY_get1_RSA(pkey) segfaults as a consequence. You probably intended to do

EVP_PKEY *pkey = PEM_read_bio_PUBKEY(bo, NULL, NULL, NULL);

However, with the PEM that you posted, that will not work. The header of the PEM indicates that this is an RSA key stored in "traditional" format. PEM_read_bio_RSAPublicKey is capable of reading such keys:

RSA *rsa = PEM_read_bio_RSAPublicKey(bo, NULL, NULL, NULL);

Then your choice of RSA_NO_PADDING when invoking RSA_public_encrypt is not right. In order to use that, the plaintext must be the exact same length as the key size. You could try using RSA_PKCS1_PADDING instead.

Finally, like mentioned in the comment section as well, using strlen on cipher text is incorrect, because it is not a 0-terminated string.

In general, you should check all return values for all OpenSSL functions invoked, to see when/if something went wrong.