Clusters

Intro
Installation
SysAdmin
Accounts
Transfer
Access
Directory
Data
Clusters
Miscellaneous
Licensing
HowTo
  • Traditional File-Sharing Approach
  • Account-Sharing Approach
  • Benefits of the Account-Sharing Approach
  • Supported Services
  • Configuring DNS
  • Configuring Backend Servers
  • Configuring Frontend Servers
  • Symmetric Clusters
  • Fault Tolerance and Static Clusters
  • Dynamic or Super-Symmetric Clusters
  • When your site serves more than 150,000-200,000 accounts, or when you expect a really heavy IMAP/WebMail traffic, you should consider using a Cluster configuration.

    A Cluster is a set of server computers that handle the mail load together. The CommuniGate Pro software allows you to use any set of OS and hardware platforms in a CommuniGate Pro Cluster.

    If your site serves many domains, you may just install several CommuniGate Pro servers and distribute the load by distributing domains between the servers. In this case you do not need to employ the special Cluster Support features. However if you have one or several domains with 100,000 or more accounts in each, and you cannot guarantee that clients will always connect to the proper server, i.e. when you need dynamic load balancing and a very high fault tolerance, you should implement a CommuniGate Pro Cluster on your site.

    To use CommuniGate Pro servers in a cluster, you need a special CommuniGate Pro Cluster License.

    Please read the Scalability section first to learn how to estimate your mail server load, and how to get the most out of each CommuniGate Pro Server running your multi-server (cluster) site.


    Traditional File-Sharing Approach

    If you try to use any "traditional" mail server, the only way to distribute your mail load between several servers is to use some sort of network file sharing protocol and let several mail servers access the same account files on one or several file servers.

    These file-sharing based solutions have the following problems:

    The CommuniGate Pro Server supports External INBOXes, so it can be used for File-Sharing based systems, too. Still (because of the problems listed above) such systems cannot be recommended for use at carrier-scale sites.


    Account-Sharing Approach

    The CommuniGate Pro system allows you to build clusters based on mail protocols, rather than file-sharing protocols.

    Any CommuniGate Pro Server can act as an account server, providing access to its accounts for other CommuniGate Pro Servers.

    You can use a symmetric cluster, or you can use the Frontend/Backend topology. We will discuss the Frontend/Backend systems here: they require more servers, but they can also give you more benefits. A symmetric cluster is a cluster where the same servers are used for both Frontend and Backend types of activity.

    Frontend servers accept TCP/IP connections from client computers (usually - from the Internet). In a pure Frontend / Backend setup there is no accounts created on the Frontend servers, but nothing prohibits you to serve some accounts and/or mailing lists directly on the Frontend servers.

    Backend servers are the servers where all (or most of) accounts reside.

    When a client establishes a connection with one of the Frontend servers and sends the authentication information (user/account names), the Frontend server detects on which Backend server the address account actually resides, establishes a connection with that server, and then acts as a proxy:


    Benefits of the Account-Sharing Approach

    The Account-Sharing setup has the following benefits:

    You can use both Account Servers and File Servers: Backend servers can store the actual account files on File Servers. In this case:


    Supported Services

    The CommuniGate Pro clustering features support the following services: The WebUser Interface module maintains user sessions even if subsequent page requests come to the Backend server through different Frontend servers.


    Configuring DNS

    For each domain served by a CommuniGate Pro cluster, a DNS record should be created. The A-record for the domain should include the IP addresses of one, two or all Frontend servers. You should use a load-balancing DNS server that directs clients to several Frontend servers distributing the load as evenly as possible.


    Configuring Backend Servers

    Each Backend server should have a unique main domain name. These domains are not required to be "real" domain names, i.e. the names known to the DNS. You can use simple names as s1.cluster, s2.cluster, etc.

    SMTP

    The outgoing mail traffic generated by regular (POP/IMAP) clients is submitted to the site using the A-records of the site domains. As a result, the submitted messages go to the Frontend servers and they are distributed from there.

    Messages generated by WebUser clients and messages generated automatically (using the Automated Rules) are generated on the Backend servers. Since usually the Backend servers are behind the firewall and since you usually do not want the Backend servers to spend their resources maintaining SMTP queues, it is recommended to use the forwarding feature of the CommuniGate Pro SMTP module. Select the Forward to option and specify the domain name that resolves into the IP addresses of all (or some) Frontend servers. In this case all mail generated on the Backend server will be quickly sent to the Frontend servers and it will be distributed from there.

    Central Directory

    If you want to maintain a Central Directory that lists all accounts on all Backend servers, you should maintain it as an LDAP server on one of the servers. On all [other] Backend servers, you should replace the default Directory.tdb file with the Directory.ldb file that describes the LDAP server keeping that unified directory. Make sure that records in the Central Directory contain the #home field: that field will keep the name of the Backend server hosting the account.

    See the Central Directory section for more details.


    Configuring Frontend Servers

    A Frontend server:

    You should open the Cluster Settings page using any browser and specify the main domain names of your Backend servers, along with the server IP addresses:
    Member NameMember Address

    If an address is routed to a domain listed in this Cluster table, the Frontend server uses the CommuniGate Pro Clustering features to connect to the Backend server at the specified address and performs the requested operations on the Backend server.

    Using the Router

    You can use Router tables to provide account home location information.

    You should use the same techniques as those used to specify explicit routing for certain E-mail addresses: you specify which accounts should be "routed" to which server.

    If accounts in the domain.com domain have to be distributed between 5 Backend servers n1.clust, n2.clust, n3.clust, n4.clust, n5.clust, then you can use the first letter of account names to distribute the accounts:

    When a connection with the Frontend server is established and the client software provides the account name bill in the domain.com domain, the Access module (POP, IMAP, WebUser, etc.) processes the bill@domain.com address with the Router, as explained in the Access section.

    Using the specified Router Table rules, the bill@domain.com address is routed to the bill%domain.com address on the n1.clust SMTP host.

    Without Cluster Support enabled, the Access module reports the account is not local error if the specified address is routed to the SMTP module.

    When Cluster Support is enabled, and the name of the "SMTP host" is listed in the Cluster Members list, the Access module contacts the specified Backend server (n2.clust, i.e. 206.40.74.195) and tries to open an account there. If account is successfully opened on the Backend server, the Frontend server sends the positive response to the client software and the session begins.

    Using the Central Directory

    While Router-based Clustering is the simplest method from the technology point of view, that method may not be sufficient for large sites where accounts and domains are created and removed very often and it quickly becomes very difficult to update all those Router tabled manually.

    If your site maintains a Central Directory for all accounts on all back-up servers, the Frontend servers can use that Directory to find the back-server hosting the addressed account.

    You can use methods described in the Central Directory section to provide a Directory for all Servers in your Cluster. Make sure that the Central Directory contains the #home field (attribute): this attribute contains the name (Main Domain Name) of the Server hosting that account.

    Enable the Directory-Based Clustering Option in the Router Settings of all your Servers. With this option enabled, the CommuniGate Pro Server makes an additional step when routing messages to a local domain: if an addressed account is not found on that server, the server send a request to the Central Database. If a record for this address exists and contains the #home field, the message is routed to the Backend server specified in that field.


    Symmetric Clusters

    Symmetric Cluster is a set of CommuniGate Pro Servers each operating as a Frontend and as a Backend server.

    For example, you can distribute domain accounts between 4 servers in your cluster, and let each server accept incoming connections for that domain (or domains). If the server A detects that an addressed account is located on the server B, the server A starts to operate as a Frontend server, connecting to the server B as to the Backend server storing the addressed account. At the same time, if an external connection is established with the server B and that server detects that the addressed account is located on the server A, the server B acts as a Frontend server, opening a connection to the server A and using it as a Backend server:


    Fault Tolerance and Static Clusters

    The Clusters described in the previous chapters are also called static clusters because every account is always served by a dedicated Backend server.

    Static Clusters survive a complete failure of any of Frontend servers: as long as at least one Frontend server is up and running, users can have access to all site accounts.

    If one of Backend servers fails, it does not have any effect on accounts served with other Backend servers.

    To restore access to the accounts processed with the failed server, the account storage should be made accessible to any other Backend server. You can either:

    After a sibling Backend server gets physical access to account storage of the failed server, you should modify the Routing so all [Frontend] servers now contact the new "home" for those accounts. If your sites uses Central Directory for Clustering, simply update all directory entries where #home=failed server setting the #home field to the new home server.


    Dynamic or Super-Symmetric Clusters

    The Static Clusters described above can be used to handle really large (practically unlimited) Internet sites, providing 7x24 site access. In a rare case of a Backend server failure, the Static Clusters continue to operate and access to accounts on the failed server can be restored within 1-10 minutes (depending on how easily the disk storage can be reassigned and how fast Routing can be updated).

    If it is necessary to provide 100% site uptime and 24x7 access to all accounts even if when some of Backend servers fail, the Dynamic or Super-Symmetric Cluster should be used.

    Backend Servers in a Dynamic Cluster have access to all site accounts, i.e. they share the account disk storage. The most common method to implement this type of Cluster storage is by employing dedicated File Servers.

    A Dynamic Cluster Servers do not use OS-level file-locks to synchronize account access operations. Like in a Static Cluster, only one Server has access to an account at any given moment, and all other Servers work through that Server if they want to access that account. But this assignment is not static: any server can open any account if it is not opened with some other server. If an account is opened with some other server, all account operations are conducted via that server.

    This architecture provides 100% uptime: if any of Backend servers fail, all accounts are still available for other Backend server - without any operator intervention.


    Configuring the Cluster Controller

    One of the Backend servers in a Dynamic Cluster acts as the Cluster Controller. It synchronizes all other servers in the Cluster and executes operations such as creating shared domains, creating and removing accounts in the shared domains, etc.

    You should install the CommuniGate Pro software on the Controller computer and configure it using the standard procedure for single-server installations.

    Cluster Controller can serve both shared domains (that will be served with the entire cluster) and local domains that are served only with the Cluster Controller itself. The Main Domain is always a local domain, and you can use it to create accounts for Cluster administrators.

    When initial configuration is completed, stop the CommuniGate Pro Server.

    Create a directory that will contain shared domains. You should create that directory on a storage unit that will be available for all Cluster Backend servers (on a file server, for example). Place a link to that directory into the CommuniGate Pro base directory, and name that link SharedDomains. Make sure that the Controller computer has all file access rights to create, remove, read, and write files and directories inside the SharedDomain directory.

    If you migrating from a single-server configuration, you may want to make some of your existing domains shared, so they will be served with the entire Cluster. In this case you should move the domain directory from the {base}/Domains directory into the {base}/SharedDomains directory.

    Add the --ClusterController flag to the CommuniGatePro startup script, and restart the CommuniGatePro Server.

    Use the WebAdmin Interface to verify that the Cluster Controller is running. Use the Domains page to check that:

    Use the Create Shared Domain button to create additional domains that should be served with the Cluster.

    Use the WebAdmin Settings->Access page to modify the PWD service settings. Each Cluster member (Backend and Frontend) opens 2 PWD connections to the Cluster Controller, so the maximum number of channels should be increased at least by
    2*(number of Backend servers + number of Frontend servers)

    Since additional PWD connections can be opened by Frontend and Backend servers to serve administrator and user requests, it is better to increase the number of channels by:
    5*(number of Backend servers) + 3*(number of Frontend servers)

    Use the WebAdmin Interface to open the Settings->General->Cluster page. Enter the IP addresses of your Backend and Frontend servers. These servers do not have to be running at this moment - this setting just tell the Controller which addresses should be treated as Cluster addresses.

    After the Cluster Controller is configured, the site can start to serve clients (if you do not use Frontend servers). If your configuration employs Frontend servers, at least one Frontend server should be launched in the Cluster mode.


    Adding a Backend Server

    A Dynamic Cluster can work with just one Cluster Controller operating as a Backend server. You can add additional Backend servers to the Cluster at any moment, and you can remove them from the Cluster for maintenance. If a Backend server fails, all shared domain accounts that were open on that server at the time of failure become unavailable. They become available again within 30-60 seconds, when the Cluster Controller detects the failure. A Backend server failure does not cause any mail loss.

    You should install the CommuniGate Pro software on a Backend Server computer and configure it using the standard procedure for single-server installations.

    A Backend server can serve both shared domains (that will be served with the entire cluster) and local domains that are served only with that Backend Server itself. The Main Domain is always a local domain, and you can use it to create accounts for Backend server administrators.

    Use the WebAdmin Interface to open the Settings->General->Cluster page. Enter the IP addresses of your Backend and Frontend servers.

    When initial configuration is completed, stop the CommuniGate Pro Server.

    Place a link to the shared domains directory (created on a shared storage unit) into the CommuniGate Pro base directory, and name that link SharedDomains. Make sure that the Backend computer has all file access rights to create, remove, read, and write files and directories inside the SharedDomain directory.

    Add the --ClusterMember address flag to the CommuniGatePro startup script. The address parameter should be the IP address of the Cluster Controller. Restart the CommuniGatePro Server.

    Use the WebAdmin interface to verify that the Backend Server is running. Use the Domains page to check that all shared domains are visible and that you can administer accounts in the shared domains.

    Use the WebAdmin Settings->Access page to modify the PWD service settings. Additional PWD connections can be opened by Frontend and Backend servers to serve administrator and user requests, so you should increase the number of PWD channels by:
    2*(number of Backend servers) + (number of Frontend servers)

    When the Cluster Controller and at least one Backend Server are running, they both can serve all accounts in the Shared Domains. If you do not use Frontend Servers, load-balancing should be implemented using DNS round-robin or similar technique that distributes incoming requests between all Backend Servers.


    Adding a Frontend Server

    Frontend Servers allow you to:

    You can add additional Frontend servers to the Cluster at any moment, and you can remove them from the Cluster for maintenance. If a Frontend server fails, no account becomes unavailable and no mail is lost. While POP and IMAP sessions conducted via the failed Backend server are interrupted, all WebUser Interface session remain active, and WebUser Interface clients can continue to work via remaining Frontend Servers.

    You should install the CommuniGate Pro software on a Frontend Server computer and configure it using the standard procedure for single-server installations.

    A Frontend server can serve both shared domains (that will be served with the entire cluster) and local domains that are served only with that Frontend Server itself. The Main Domain is always a local domain.

    Use the WebAdmin Interface to open the Settings->General->Cluster page. Enter the IP addresses of your Backend servers there.

    When initial configuration is completed, stop the CommuniGate Pro Server.

    Add the --ClusterFrontend address flag to the CommuniGatePro startup script. The address parameter should be the IP address of the Cluster Controller. Restart the CommuniGatePro Server.

    Use the WebAdmin interface to verify that the Frontend Server is running. Use the Domains page to check that all shared domains are visible.

    When Frontend Servers try to open one of the Shared Domain accounts, the Controller directs it to one of the running Backend Servers, distributing the load between all available Backend Servers.


    Security Issues

    The Frontend-Backend topology allows you to protect the site information and Backend Servers not only if one of the Frontend Servers is crashed because of some type of network attack, but even if the Frontend Server OS is "cracked" and an intruder gets the complete ("root") access to the Frontend Server OS using a security hole in that OS.

    To protect the site from these "cracks":

    These measures do not cause any problem for your users that have the domain administrator rights and want to administer their shared domains (using WebAdmin Interface or CLI). They also do not cause any problem for your regular users that want to use the PWD module to update their passwords.


    Dynamic Cluster Administration

    The CommuniGatePro Dynamic Cluster distributes not only the account (mail) processing tasks, but the Domain/Account Administration tasks, too. A Server and/or Domain Administrator can connect to any server in the Cluster and perform all operations using the WebAdmin Interface or CLI - in exactly the same manner these operations are performed on a single-server site.

    An administrator can change the settings of an account that is opened on a different Cluster server, and if an administrator uploads a modified version of some WebUser Interface file, the domain WebUser Interface Cache is automatically modified on all Cluster Servers.

    Note: most of the Domain-level update operations, such as updating Domain Settings, Default Account Settings, WebUser Interface Settings, and Domain-Level Alerts may take up to 30 seconds to propagate to all servers in the Cluster. Account-Level modifications come to effect on all servers immediately.

    When you assign IP addresses to a shared Domain, CommuniGate Pro does not complain when it detects non-local addresses. For shared domains, CommuniGate Pro accepts all local addresses from the specified set. This allows you to assign several IP addresses to a shared domain (either directly or using a DNS A-record), which correspond to local IP addresses of your Frontend servers. Each Frontend Server will find its local IP address in the specified set and assign it to that shared domain. The Frontend Servers ensure that the account names passed to backend servers are fully qualified, so Backend Servers do not need to have any local IP address assigned to any shared domain.


    CommuniGate® Pro Guide. Copyright © 1998-1999, Stalker Software, Inc.