In the real world today you seldom if ever set up a server (or workstation) in isolation. Inevitably there are many services involved and these are usually hosted on different servers: a web server farm, DHCP, printing, LDAP (single sign-on), DNS, NFS, Samba, NIS, etc. In this situation there can be many dependencies between servers, so that changing the configuration of any one of them may impact others. Here are a few ways these dependencies can hurt the unwary:
This problems can become especially acute in a de-centralized environment, where some "uppity" SA over-rides the standard configuration with local changes.
Every organization has information stored about its IT infrastructure, and this information must be kept up-to-date as changes are made to any configurable items (or CIs), and consulted for dependency information when planning any changes.
Configuration management is the set of tasks needed to juggle the configuration of all servers, routers, switches, firewalls, PCs, etc, and all application configuration on them. The SA must check if changes to any CIs have been recorded correctly, including any dependencies that may have changed, and monitor the status of all IT components.
Configuration management also covers patching systems and applications throughout the enterprise. In larger organizations this requires some help (including policies, tools, and servers) or the task is impossible. There may be legal requirements for doing some sort of configuration management.
Most SAs use some collection of tools to manage some part of system configuration: webmin, command-line tools, central software repositories, etc. However there is no unified approach taken in the usual case, to configure a network of hosts. Starting with:
The systems configuration tasks are:
Some tools do exists to solve the overall tasks of configuration. You define the system specification, the location of the hosts, repositories, etc., and let the tool do the rest. Such tools include cfengine (old), Bcfg2, puppet, etc.
Other tools are more commonly used, but only help with some of the tasks. Such tools include using databases (referred to as the configuration management database or CMDB), which should not be confused with asset management. (Qu: What kind of queries do you think would be useful?) Other tools include a repository of software and patches, a variety of shell scripts, open source, and commercial tools, all used to monitor the system and apply updates, and revision control systems.
Other issues to consider are that updates and reconfigurations are made on a production network, so it is not okay to flood the net with updates all at once. No software update should require physical access to the host. No update should crash the host, as it may have live users. Hosts may not have the previous configuration assumed (the user or some SA may have changed it) so any update must carefully check. Finally, dual-booted PCs may be running a different OS than the update is for!
Patches are applied to applications and to the OS, usually to address flaws (but sometimes to add features). These can be critical to apply, e.g., security patches. Software vendors have a responsibility to produce such critical patches in a timely manner. Sometimes a flaw just discovered existed for some time, and it may be necessary to have different patches for each supported version of the software.
Ideally applying a patch will not affect any system users. Even simple code flaws may not be easy to fix, however. The WebDAV issue fixed in Microsoft Security Bulletin MS03–007 was an example. While the exploit happened in WebDAV, the actual problem occurred in a kernel function used by more than 6,000 other components in the operating system. A simple code flaw is no longer easy to fix when you have 6,000 callers. Some of those callers may actually be relying on what you have now determined to be flawed behavior and may have problems if you change it. (This sometimes leads to patches for the patches!)
It is important for the vendor to ensure that one patch doesn't undo the good work of some prior patch. Also, some patches may depend on earlier ones. (To avoid that problem Microsoft issues cumulative patches, so if you use the latest one you get all prior ones too.)
Patches must be easily deployable.
In some cases, a patch is to the source code of some application or
operating system.
This is often the case with open source software such as the
Linux kernel.
Such a patch is actually a file containing the output of the diff
command, and can be used to update the old version of the source code to the new
version using the patch command.
Then the code must be re-compiled and re-installed.
This can be quite complex to do!
For proprietary software the source code is not available, and the only choices are to ship the patch as a binary diff that modifies the executable and other files to the new version, or to ship a completely updated version of the software, which must then be installed.
Most systems include some sort of patch management system, or software package system, or both. Examples of patch management systems include MS Windows Update, Apple Software Update Service, and Solaris AutoPatch. Examples of package systems include Windows Installer, and Debian, Solaris, Slackware, and Red Hat package management systems.
Most package and patch management systems can run pre- and post- install scripts with the package/patch. Small configuration changes may be made by using the patch management system. A package/patch is created that just runs some scripts without actually installing anything else. When the patch is deployed, the scripts run and can change configuration files.
Patches should use a naming scheme that identifies the software being patched (including it's version number, the software's new revision number (once the patch is successfully applied), and the architecture (e.g., PC, IA–64, etc.) and locale (e.g., English or Japanese version).
Patches should be single files that contain a description of the problem being addressed by the patch, details of the software and version the patch is for, what prior patches if any this patch depends on, and any other instructions or information users of the software may need (e.g., "You will need to reboot after installing this patch", or "You must change your configuration after installing this patch").
Patches ideally support an un-install, or rollback option. Patches should also add an entry to a log file for patches.
Patches must be protected from tampering, and support verification. This is typically done with digital signatures, or at least with a checksum/digest/hash. Patches can be made available with a secure web server, or some other TLS-protected service.
Some patches may not come from the vendor. This can happen with an enhancement patch, or if the vendor is slow in supporting its customers. You must be very careful with such third-party patches! Often installing one voids your support contract with the vendor. At the very least it can adversely affect future patches from the vendor.
Patches must be deployed, and customers informed when new patches are available. This can be done automatically but that is rarely a good option for servers. Often an email notice is sent (today RSS can also be used).
Notifications should include important information, including detailed information on the problem, affected components and their versions, instructions and information about installing the patch and any changes required by the patch, where to download the patch from, and possible work-arounds for the problem in case the patch can't be applied immediately. The notification should also describe the severity and urgency of the patch (e.g., a security patch with major implications, but with no known exploits, versus a patch with minor implications but with current exploits known). If you have an urgent patch it may be necessary to skip some compatibility testing and deploy it quicker.
Once you have been notified of the availability of a patch, you need to identify all hosts that need the patch. This includes servers, workstations, and don't forget remote users (e.g., those who work from home or use laptops or notebook computers.) Note some servers run multiple instances of some software, so you need to remember to patch each instance.
Once you have downloaded (and verified) the patch, you next need to test it, to make sure it is compatible with your critical software. Installing a security patch is useless if it crashes you mission-critical applications!
If installing a patch requires down-time, you may have to delay installing it on production servers until a maintenance window is available. Installing patches on clusters and grids usually requires special planning and support.
To ensure your organization is fully protected it is important to monitor patch compliance. You may need to decide to isolate un-patched hosts until they are patched. Other aspects of patching should be monitored as well, for management reporting purposes: patch download and install rates, reliability, and whether or not required reboots/restarts have been done after patch installs.
Patch management policies for patch deployment can be one of three types. You will probably use a combination of these, for different types of patches/updates: