To provide assistance in developing command lines for your own particular situation, here are some samples.
Assuming that the installer application is located in the downloads directory:
C:\Downloads>QWS3270 Setup 5.4.302.exe DEFAULT_SSL_LIBRARY=OpenSSL DEFAULT_SSL_FIPS=1
This sets the default security SSL/Library to use OpenSSL for any session which specifies the Auto-Select option, and sets FIPS compliance to TRUE, using the .EXE installation file.
C:\Downloads>QWS3270 Setup 5.4.302.msi DEFAULT_SSL_LIBRARY=SChannel
This sets the default security SSL/Library to use SChannel for any session which specifies the Auto-Select option, using the .MSI installation file.