In this post, I will cover a basic overview of the SSO 5.1 architecture.The figure below illustrates the high level architecture of SSO in vSphere 5.1.

Problems with the Services

In vCenter 5.1, the SSO system is comprised of two services:
 1. VMware vCenter Single Sign-On
 2. RSA SSPI
The primary Single Sign-On service is basically a Tomcat web server which is used to provide access to a series of web applications. The current status of this service represents only the status of this Tomcat server, and has no bearing on the status of the functionality of the web applications themselves. When this service showed as Running, it would often lead VMware administrators to come to the mistaken conclusion that Single Sign-On was operating correctly. This was not always the case, and often the only way to know if a given application was running was to check the time stamp on the related log files.
The RSA SSPI service is used to access the existing Windows login token, for use with the “Windows Session Credentials” login option. Due to the naming convention of this service, many people were not even aware of the existence of this service or its function.

Problems with the Database

Single Sign-On requires a simple database to store configuration files. In SSO 5.1, all supported vCenter database types could be used for the Single Sign-On. This was done for customer convenience, but the allowance for an external database also led users to over emphasize the importance and size of this database. Obviously this database needs to be available for SSO to operate, but the size and number of transactions to this database were both very small.
The database also led to the majority of the configuration and installation failures, mostly due to the fact that SSO supported only a “JDBC” style connection to the database, and not the more commonly recognized “Named Pipe” connection style which is created via an ODBC/DSN. Essentially, the database server must be listening on a specified TCIP/IP socket in order for SSO to connect, and this was often not the default configuration for many database servers. Again, a big headache for many customers.
In addition, since the existing SSO 5.1 code did not support communication via Integrated Windows Authentication, the database needed to be created properly prior to install. This required the end-user to run a script against his database server to create the database and table spaces. This is an unwieldy procedure which often caused confusion and installation failures.
Finally, this model did not support database replication, which was required to configure a proper Multisite environment. This required manual CLI commands for replication and backup, and also caused more confusion during installation and configuration.

Problems with the Deployment Modes

The designers of SSO 5.1 tried to provide a series of possible deployment modes which could cover all possible use cases which customers might desire. In the end they settled on four possible deployment modes:
 1. Basic
 2. Primary
 3. Multisite
 4. High Availability (HA)

The Basic deployment is a single SSO node which can not be linked to another. This mode is automatically used when using the “Simple Install” option in vCenter 5.1. The advantages to this mode are that it is simple conceptually, it supports Local OS Users & Groups as an identity source, and it is easy to install. The disadvantages included an inability to joined a “vCenter Linked Mode” configuration, it had no failover capacity, and since the node could not be converted to a more advanced deployment mode, it often caused management difficulties as the customers environment grew and changed.
The Primary deployment represented a parent node which had no specific role, other than the fact that it could have Multisite and HA nodes join it to create the desired type of configuration. Note that this is essentially the only difference between a Primary and a Basic node. In addition, the Primary node lacked the ability to utilize the Local OS Users & Groups directory on the host system.
Multisite and HA deployments represented different use cases. Multisite allowed for an SSO 5.1 node in geographically disparate locations, each with its own local configuration and identity sources. High Availability allowed for the SSO 5.1 node to function as a failover system in case of an outage of the Primary Node. Multisite generally worked as described, but unfortunately the High Availability system often presented many challenges due to design limitations.
In the end, the number of deployment modes (and in many cases the nomenclature itself) presented an overwhelming number of options. Many customers felt confused by the choices, and often ended up choosing the wrong deployment mode during installation. Since there was no simple method to change deployment modes after install, this resulted in a great deal of wasted time all around.

Problems with Authentication

The primary method of authentication used in SSO 5.1 is to link to Identity Sources using a system known as an LDAP Simple BIND. When an LDAP client connects to the server and an LDAP session is created, Simple BIND sends the users Distinguished Name and Password. The BIND operation establishes the authentication state for a session.
SSO 5.1 also supports anonymous BIND (with empty DN and password) which allows the connection to run in an anonymous state. Unfortunately this method caused many issues in multi-forest environments. In many larger environments, this system would fail to get proper group information regarding users, and there were several bugs which interfered with One and Two-way Trusted Domain configurations.

Problems with System Domain Directory

The SSO 5.1 system introduced a Directory which was installed locally and could be used to create users and groups. This was an extremely simple directory structure when compared to things like Active Directory. It was limited to creating users, assigning them to groups, and using vCenter to assign permissions to those principals.
For most people who used SSO 5.1, there is only a single user which they would know about: admin@System-Domain. This administrator user was the SSO super user, and could be used to administer and configure SSO via the vSphere Web Client. During upgrades to version 5.5, this would get access to VCenter by default.

Problems with Master Password

Another lesser known element of SSO 5.1 was the concept of a master password. This master password was not associated with any specific user, even in the System-Domain directory. The password was configured during installation to automatically be the same as the password given to the admin@System-Domain user.
The distinction was important, because the Master password was not automatically changed when the admin@System-Domain user password was changed. Due to the poor communication regarding this password to the end-users, many administrators would find themselves unable to perform certain administrative tasks against an SSO 5.1 environment because this password had been “lost”. This would result in a full reinstall of Single Sign-On, and subsequent re-registration of other components, which could result in many wasted hours.
vTip: Upgrade to version 5.5 and use the newer SSO and avoid these headaches. :)