To provide assistance in developing command lines for your own particular situation, here are some samples.
Assuming that the installer application is located in the downloads directory and the particular version is 5.4.302:
C:\Downloads>QWS3270 Setup 5.4.302.exe DEFAULT_SSL_LIBRARY=OpenSSL DEFAULT_SSL_FIPS=1
This sets the default security SSL/Library to use OpenSSL for any session which specifies the Auto-Select option, and sets FIPS compliance to True.
C:\Downloads>QWS3270 Setup 5.4.302.msi DEFAULT_SSL_LIBRARY=SChannel
This sets the default security SSL/Library to use SChannel for any session which specifies the Auto-Select option. The de facto default for FIPS (False) will used.
C:\Downloads>QWS3270 Setup 5.4.302.exe DEFAULT_SSL_FIPS=1
This sets the FIPS compliance to True. The de facto default for SSL/Library (SChannel) will be used.