Un-Xhiering MFCF

Introduction

At MFCF's June 17 Management meeting, Wayne asked me to investigate getting rid of Xhier. The initial investigation showed that the functionality that Xhier provides is very pervasive throughout the various services that MFCF provides. If we wished to continue providing the same level of service, eliminating Xhier would be very expensive. A followup report on August 21 investigated the services we currently provide, many of which exist simply for historical reasons, having no significant client demand. Eliminating them would greatly reduce the impact of no longer being able to rely upon Xhier.

What follows is a combination of those two reports.

Does Xhier refer to the various infrastructure tools currently associated with that name, or to the software that currently makes use of it, or to the philosophy behind its design, or to all of the above? People have varying ideas as to what the term Xhier refers to, and conflation is a constant problem, so it could be any or all of these.

It would help if we were all familiar with the Xhier philosophy. If you haven't read it recently, please read Xhier Philosophy and Goals (Theory)https://www.math.uwaterloo.ca/mfcf/info/overviews/xhier-philosophy and Xhier Overview (Practice)https://www.math.uwaterloo.ca/mfcf/info/xhier.

Briefly, xhier is a comprehensive suite of tools providing hierarchical system configuration management and software packaging and distribution in a uniform way across a heterogeneous environment. Although its rich feature set was important to us in the past (which is why it was designed that way), it may be the case that some functionality is no longer needed.

There may be popular open-source alternatives to cover most, or perhaps all, of what we need.

This report reviews what services xhier provides, explains the reasons for those services, suggests how we might revise those services, and points out some problems. It also itemizes some features we still require. Appendices address what others on campus are doing, and provide a few initial comparisons to popular alternatives.

Commonly Used Xhier Services

Some of the most common services provided by Xhier are:

Code Portability and Centralized Compilation

The various architectures we use often provide different and incompatible functions and structures. Xhier provides compatibility libraries (e.g. libcfix) and header files (e.g. <mfcf/libc/xxx.h>) that allow code to be written independently of the architecture on which they are to be built and run.

All source code is maintained on one machine. One can make a change to some code, then use Xhier-provided programs (e.g. xh-sdist) that distribute this source to the architecture masters, compile it using whatever programs and libraries are appropriate for each architecture, and install the update into the software package. Later, the updated files will be automatically distributed to all clients machines that request the package. All this can be done in the very few seconds that it takes to type the one simple command.

Each week, Xhier automatically compares the build dates on the source master against those on the architecture master and reports to each package maintainer with a list of files that need to be (re)built (typically because the arch master was unavailable at the time the software was rebuilt).

Automated Distribution

Xhier automatically distributes software packages from the architecture masters to their client machines.

Xhiered packages can be requested at any level of the distribution hierarchy, and distribution restrictions (e.g. for licensed software) can be controlled by the higher levels.

Centralized and Non-centralized Administration and Configuration

Administration and configuration can be centralized at several levels, from the shared configuration on the source master, to architecture-specific, administration-specific (e.g. MFCF or CSCF), region-specific, or individual machine.

In particular, much software makes use of xhier regions, where several machines (often of different architecture) share a common set of home directories and user accounts (e.g. the student.math, general.math, and iqc.math regions).

Uniform Presentation

In addition to the uniform environment provided to the package maintainers for building and distributing their source, Xhier provides standard ways for packages to be configured. Regardless of the package, local system administrators know exactly where to look to determine each package's configuration, crontab entries, weekly installation scripts, etc.

Similarly, since packages can be built independently of which architecture they are installed on, the set of commands and services provided by each package is identical on all machines. E.g. users don't need to know that they must use one option flag on Linux and a different one on Solaris.

Other Benefits

Xhier allows multiple versions of the same package to be installed at the same time on the same macine, each version using different configuration. One package would be considered the default, but users could easily put a different version into their personal search paths. Most common Linux and Solaris packages do not allow this.

Xhier installs as little as possible into the vendor space. In particular, such things as configuration, crontab, and boottime scripts are not made available to the operating system until the package had confirmed that it has been installed correctly. Most common Linux and Solaris packages do not treat system changes specially, and so could end up making themselves available even though parts of the package are absent, broken, or misconfigured.

Major Xhier Services — Reasons for Providing

Uniform Presentation for Users

The need for this service is created by our policy of making multiple architectures appear as similar as possible to our users.

Preservation of Vendor Space

The need for this service is created by our policy of allowing people to use a machine as if we hadn't made any changes to it. (It also reduces the work required to install system patches and perform OS upgrades.)

Multiple Versions of Software

The need for this service is created by our policy of allowing users to have access to different versions of the same package on individual machines.

Code Portability and Centralized Compilation

The need for this service is created by our policy of providing the same software on multiple architectures.

Uniform Presentation for Administrators

The need for this service is created by our policy of making administration of multiple software packages as easy to learn and manage as possible.

Centralized and Non-centralized Administration and Configuration

The need for this service is created by our policy of providing a hierarchical administration structure.

Automated Distribution

The need for this service is created by our policy of providing software on multiple hosts of the same architecture.

Automation and Scaling

The need for this service is not policy, but a matter of staff wanting to do things in the most productive way possible. Maintaining ten machines should not require 10 times as much work as maintaining one. Similarly, when there was a large enough number of people (critical mass) working with Xhier, all campus groups involved with it benefited from the work done by other groups.

Essential Services

Most of, or perhaps all, the above services are essential in order to fulfil past and present MFCF policies. They are not necessarily essential in order to fulfil our mandate, and in many cases not essential in meeting the current needs of our clients. Even in cases where those needs exist, they might be provided by other means.

Updating our Services

Uniform Presentation for Users

Xhier was developed to meet a real need. At one time we had nearly 20 different architectures in our student labs and research environment. Users would have become very confused had they met with significantly different behaviour depending upon which terminal they happened to have sat down to use.

We now support only 4 major architectures (Mac, Windows, Solaris, Linux), and only two of them are similar enough that anyone might be confused by their minor differences. (This raises the issue of whether there is still any need to support both Solaris and Linux.)

Preservation of Vendor Space

Virtual machines are effectively free, easy to generate, and have a more pristine environment than what we are currently providing.

Multiple Versions of Software

Again, without the policy of providing one large seemingly homogeneous environment, the need for providing multiple versions of software on the same machine goes away. Where there is a need for a specific old version, a virtual machine for instance could be set up specifically for that purpose.

(On the other hand, a more heterogeneous environment will conflict with our more recent policy of presenting a simplistic view to our clients (e.g. linux.math rather than having to know about specific cpu### names).)

Code Portability and Centralized Compilation

And again, if we do not have a policy of providing a uniform environment, the need for building the same software on all architectures becomes much less. The amount of software that we require in multiple places might actually be quite small.

Uniform Presentation for Administrators

This is still a good goal, but given the large number of non-local packages that we now provide and support, even achieving perfection among the xhiered packages would not have a great overall effect.

Centralized and Non-centralized Administration and Configuration

Having IQC able to set their own software policies and configuration, within those set by MFCF, within those set by IST, has proven to be useful. But more and more, groups are setting up smaller self-contained computing that doesn't need the generic environment we provide.

Centralized configuration for the machines MFCF maintains is still a good and necessary goal, but without a policy that requires hierarchical administration, there are tools other than Xhier that provide this service.

Automated Distribution

While in some ways better than, and in some ways not as good as Xhier, these same centralized configuration tools (e.g. Puppet) can provide, or greatly help with, the distribution of software.

Automation and Scaling

Moving to a more heterogeneous environment will make the automation and scaling of some tasks more difficult.

Eventually, it's possible that we might be in a position to develop modules for open some source packages that might benefit a community much larger than the current Xhier world.

Significant Policy Changes

If MFCF were to make large changes to many of our long-time policies regarding what software services we provide, as described above, most of the need for Xhier would go away.

We would support only one major version each of Windows, MacOS, and Linux. The first two are outside the scope of this report, but all Linux machines would be administered from one central place by one single authority (presumably MFCF).

Possible Directions

Retaining Xhier's Level of Service

Xhier hasn't outlived its usefulness, it's outlived the people that know how to use it.

Replacing all aspects of Xhier, while retaining a similar level of service to maintainers and clients would require considerable work:

• Change all currently used source code to use standard Makefiles, custom tailored to build correctly on each architecture.
• Change all existing source code to have architecture-dependent code to replace the portability provided by Xhier.
• Define standard places to install packages.
• Provide some method for compiling, installing, and distributing packages.
• Provide some method for centralized and non-centralized configuration.
• Provide some equivalent to xhier regions.
• Build, distribute, and configure all packages using the new methods.
• Modify existing user accounts (e.g. .cshrc and .profile) to know about the new locations of software.
• Remove all files associated with Xhier.

We are aware of some external packages (e.g. Puppet, Bcfg2) that can provide a part of the above (configuration and to some extent distribution), but no existing package or collection of packages that can provide it all. (E.g. see Building Source Code in the Appendix.) A large part of this task would effectively be the reinvention of Xhier under a different name.

Furthermore, unless we can convince CSCF, IST, and other Campus groups to follow our lead, we would lose support for many xhiered packages that they currently maintain (e.g. the number and maintainers of xhiered packages that are currently on our Solaris systems: 80 MFCF, 50 CSCF, 40 IST, 50 other).

A reasonable estimate of the time required to fully provide the current level of service to the package maintainers and clients would be in man years.

The benefit to such an undertaking would be nowhere near the cost. The effort would be much better spent teaching staff how to use Xhier, and perhaps even how to improve it. But this late in the game, that too would be a very expensive proposition.

Simplifying Our Level of Service

We don't currently know what software our clients need that is dependent on Solaris and not provided by Linux. Once we have eliminated mail services, in particular IMAP, we should be able to get a much better view of how our student and research regions are actually being used by our clients. The usage statistics for October or November should give us a good sample to work with.

We do know that a significant number of programs and software packages currently relying on Xhier for compilation, configuration, distribution, etc. are used by MFCF itself.

Some of those are part of our centralized administration and accounting systems, and so really need to exist on only one host. But others are required on all our client hosts. In either case, the current source base will not build or install without the tools and environment provided by Xhier.

We might be able to reduce the amount of such software by eliminating or reducing the need for the services that they provide. This will require considerable study and further policy changes. Even then, for what remains as essential we will need to make large changes to our source and our procedures.

Essential Packages

A survey of MFCF staff indicates that (at least) the following software packages, some currently highly dependent on Xhier, are needed.

Essential Programs

In addition, a number of specific programs were requested.

Some people also suggested packages that they thought our clients might need.

Responses have been received from Chris R., Jim, Lori, Ray, and Robyn.

For completeness and to eliminate personal bias, the tables above include all submissions, even those that might have immediately been omitted.

(More investigation is needed into why some of these packages and programs are needed.)

Questions

Many questions remain unanswered. Some are technical, requiring more investigation, while others are more a matter of policy and direction.

• Should MFCF still use and provide Solaris?
• Should MFCF still provide a student.math region?
• Should MFCF still provide a general.math region?
• Should MFCF change its accounting and billing policies and procedures?
• Should MFCF …?

Next

Once we have a better understanding of where we hope to be in a year or two, we can start working toward that goal.

A good place to start would be to construct a small environment of say one central server and a separate region of two hosts; to build and install our essential software on them; and to ensure that configuration, resource management, accounting, etc. all work as required.

Once that is working to our satisfaction, we can add more hosts into this new world, and phase out our current environment.

We already administer several Linux machines without the benefit of Xhier. Perhaps we could start building a new environment using them, eventually possibly incorporating resource management software that Christopher is developing.

Appendix 1 — Building From Source

Most xhiered source can be built with a one-line xh-imakefile for the package. Similarly each program could have a one-line file specifying the name of the program, its language, and its intended use (e.g. general users or system maintainers), often with additional lines specifying required libraries or other compile-time values.

These xh-imakefiles are completely portable across all supported architectures (currently Solaris, IRIX, several versions of Linux, and MacOS).

I'm not aware of any open-source or commercial product that provides this capability. Web searches turn up similar requests, but the answers are generally No, build by hand using the vendor's native environment.. See examples below (emphasis mine).

It is possible, but for each individual package (and perhaps command) one would have to write a very package-specific and OS-specific configuration file. And each of the thousands of source files might need to be tailored to handle the various OS-specific differences.

Even within a specific architecture, such as Linux, and even for pre-built software, there isn't any one standard method of performing installation and updating. E.g. RedHat has yum, Ubuntu has apt-get, SuSE has zypper.

I know it's difficult to prove non-existence, but I'd say there simply is nothing approaching the simplicity or power of xhier's generic build-and-install xh-imakefile.

Below are typical responses to the question of using Puppet or CFEngine to maintain and build source code.

https://ask.puppetlabs.com/question/23/how-can-i-use-puppet-to-build-from-source/

How can I use Puppet to build from source? (asked Nov 13 '12)

…
I have a webserver and I want to download, unpack, configure and compile and install apache. How can I get Puppet to do this for me only once?
…

answered Nov 28 '12

…
While you could have a series of exec{} resources that each check that the commands need to be run have been run and build relationships between them for ordering, you do not want to go down that road.

All software installed on a system should be done through that OS's packaging software, not through compilation. Then you can just use a package{} resource. This also gives you the benefit of leveraging the packaging software that acts as a source of truth for installed packages and generally knows the files on disk for each package.
…

answered Dec 18 '12

Using native packages is almost always the sanest way to go, as other answers suggest. That said, Puppet as a framework is capable of supporting build-from-source style application deployments.

A generic stub of what a defined type for building from source might look like: https://gist.github.com/2597027

The gist is unfinished and most notably does not include "unless" statements or refreshonly parameters, but gets the idea across in a simple way.

https://groups.google.com/forum/#!topic/help-cfengine/tQScSHf-C8Y

Can CFEngine track packages installed from source (RedHat)

So I have searched this group as well as google and trudged through a portion of the documentation but I am unclear if CFEngine can track and report packages which are installed from source
…

CFEngine leverages native packaging systems. Source code has literally no uniformity or anything even remotely resembling standardization.

That being said, CFEngine can very easily manage source based packaging systems (e.g., BSD ports, Homebrew). If you can find (or devise) a standard convention for source based packages then CFEngine can manage it for you.

However, I would strongly urge you not to. Creating native packages on most systems is not that hard. And there's always fpm.
…

Appendix 2 — IST

(From an August 1 meeting with Mike Borkowski, Jason Gorrie, and Shawn Winnington-Ball.)

Xhier

Things are generally frozen as far as Xhier is concerned. Current packages and architectures will continue to be supported, but there is no development and little maintenance. Demand for the service is expected to continue to decline.

Configuration and Distribution

In the non-Xhiered world, IST mostly uses CFEngine, but they believe that Puppet is more useful and are moving toward using that instead.

Puppet is good at providing configuration and consistency checks on a large number of machines.

Rather than distributing software updates, Puppet reports where native software needs to be installed or updated. This process does not scale the way Xhier does, and so requires more individual attention for each machine. The number of supported machines is not expected to significantly increase however, and in many cases massive updates can be made by periodic re-imaging and where necessary letting the owners re-do their own customization.

Administration

Unlike Xhier, Puppet does not provide an administration hierarchy. This is not a problem for IST itself, as it does not currently make use of the administration and region hierarchy that Xhier provides. Individual units, such as MFCF, or within it IQC, would have to run their own Puppet server, but could make use of modules provided by IST.

Source

IST doesn't have much locally written software, and what they do have is distributed to only a small number of machines, all of the same architecture. Losing Xhier's ability to maintain portable source for multiple architectures will not be a significant issue for them.

Appendix 3 — CSCF

(From an August 2 meeting with Fraser Gunn.)

Xhier

Like MFCF, CSCF cares about this issue, but has fuzzy plans, and isn't doing much about it yet.

They would miss Xhier's ability to provide multiple versions of the same package, independently configured, on the same machine. This can't be done easily with many Linux packages. Xhier is also much better at ensuring that packages are properly installed before it makes changes to the system configuration.

Source

Unlike MFCF, CSCF is well on the way to moving to a homogeneous environment with only one architecture. They are also arranging that their new packages are created in Debian package format.

Having everything on Debian/Ubuntu eliminates the need for portable code, though it does mean that they may eventually end up with a lot of architecture-specific code that will eventually make moving to or incorporating any other system an expensive task.

Much of the need for many xhiered packages has gone away over the years due to improvements in vendor software. Previously it was very inconsistent from one architecture to another, was lacking in features (which had to be added locally), and was often buggy, with slow or nonexistent response for fixes.

But in terms of xhiered software, they are still very dependent on some locally written packages, such as the accounts management system.

Configuration and Distribution

CSCF has experimented with CFEngine and Bcfg2, but don't really like them. Puppet seems like a much more promising system, especially if used with GIT for content management.