The mask generation function (MGF) is always MGF1 from section B.2.1 of PKCS1 (currently there is no other MGF function defined).
Note that the scheme hash function and the MGF hash function can be different.
In this Toolkit, the default hash function is SHA-1.
You can specify a different hash function using the nOptions argument.
If you do so then the MGF1 hash algorithm will also be same hash function (this is recommended practice).
Add the PKI_MGF_MGF1SHA1 option to force the MGF hash function to be SHA-1 (which, strictly speaking, is still the default).
You cannot set any other combinations of scheme and MGF1 hash functions in this Toolkit.
The default salt length for RSA-PSS is hLen, the length of the output of the hash function in bytes.
You can change the salt length when creating a signature using the PKI_PSS_SALTLEN_* options.
In this Toolkit, the "trailer byte" for RSA-PSS is always 0xbc and the label L for RSA-OAEP is always the empty string; these cannot be changed.
Note also that there are physical limitations in the length of message digest used in the MGF function with RSA-PSS and RSA-OAEP: you cannot, for instance, use the SHA-512 MGF digest function with a 1024-bit RSA key (the digest is too long).