MFCF's RT — Alternatives and Improvements

0. Introduction

0.1 Outline

MFCF's RT — Alternatives and Improvements
- 0. Introduction
  - 0.1 Outline
  - 0.2 Request
  - 0.3 Survey
- 1. Features Of Our Existing RT System
  - 1.1 Missing
  - 1.2 Good
  - 1.3 Bad
  - 1.4 Ugly
- 2. Possible Directions
  - 2.1 Design our own request tracking system from scratch.
    - Advantages
    - Disadvantages
  - 2.2 Upgrade our existing RT system to version 3
    - Advantages
    - Disadvantages
  - 2.3 Transfer our data to IST's RT system
    - Advantages
    - Disadvantages
  - 2.4 Go with an existing non-RT system
    - Advantages
    - Disadvantages
  - 2.5 Stay with our current software
    - Advantages
    - Disadvantages
- 3. Recommendations
- 4. Unanswered Questions

0.2 Request

   From tserviss@math.uwaterloo.ca Fri Dec  1 05:50:48 2006
   From: "Tom Serviss" <tserviss@math.uwaterloo.ca>
   To: "'Ray Butterworth'" <rbutterworth@math.uwaterloo.ca>
   Subject: RT survey project

   I would like for you to assess the RT system.  We would like to have a
   report which outlines the strengths, weaknesses and constraints posed
   by the current system.  We would also like a similar analysis of the
   newer version of the RT system used by IST.  Once these assessments are
   complete, we would like you to make a recommendation as to whether we
   should stay with the old system, move to an instance of the new system
   on the IST server, or look at developing our own system.

0.3 Survey

   From rbutterw Mon Dec  4 14:41:27 2006
   From: "Ray Butterworth" <rbutterworth@math.uwaterloo.ca>
   Reply-To: rbutterworth@mfcf.math.uwaterloo.ca
   To: mfcf-staff
   Subject: RT Survey - please respond

   Please take a few minutes over the next week or two to respond
   to this.  I'd like to hear from everyone.

   I'll compile (and probably paraphrase) the results without
   attributing them to any individual.

   If you include personal comments that you don't want included in
   the survey (e.g. something specific enough that it identifies you),
   please mark them as such.

   We're doing a survey of our RT system, trying to find out
   what we as users consider good about it and what bad.
   We aren't necessarily going to replace it with something
   completely different, but there are newer versions of the
   software available and there is a lot of room for customization.

   So, we're asking you to mail your comments to me (rbutterworth)
   about (at least) these topics:
     - What features do you really like and wouldn't want to lose?
     - What features are missing that you'd like added?
     - What features and bugs do you dislike and would like corrected?
     - Anything about the way we do or don't use the RT system within MFCF?

   They could be about the database structure itself, the web-form
   interfaces, documentation, or any other aspect you want to discuss.

   I expect that the "like" list will be fairly small and similar
   for most of us, but if there are some obscure features that you
   rely upon and think that others might not even know about,
   be sure to mention them.  (If we do make significant changes
    as a result of this, we don't want people to suddenly discover
    useful features have become no longer available.)

   The "add" list can be whatever you want.  Some things will of
   course be effectively impossible to add, but it's surprising
   how many apparently complicated things can actually be simple
   to implement.

   As for the "don't like" list, we expect to see a lot of suggestions.
   They might be simple bug reports about things that don't work as
   advertised, complaints that things don't work as expected,
   suggestions for how things could work differently, or even
   suggestions for significant reorganization and design.

   For bugs, specific details would be nice to get, but for
   everything else even vague generalities are acceptable.

   And please keep in mind who is compiling the results, and so
   realize that no complaint or suggestion is too trivial to consider.

   Thanks.

1. Features Of Our Existing RT System

Our current request tracking system was adapted by MFCF from one of the earliest RT releases and highly modified for our own purposes (mostly by Fraser Gunn and Bill Ince, and later by Isaac Morland, all now with CSCF).

It is heavily used by all staff within both groups, and has become an essential tool in our daily routine, providing a current and historical record of:

what has been done
what still needs to be done
who is responsible for doing it
all non-routine work performed by the department
requests and bug reports that are not significant enough to procede with, but which are worth recording. (Anyone working on a more serious item might find it convenient to handle some of these related items at the same time. Similarly, repeated reports of the same problem could raise its priority.)

Most of the following comments result from a survey of MFCF staff. They are not comprehensive, but do illustrate the kind of things that people care enough about that they have taken the time to report them. I've tried not to imply any relative merit among the many suggestions listed in this report. It is worth noting though that almost none of the suggestions received contradicted or conflicted with each other, and that except for a few minor points, I would agree with them all.

The features of our current request tracking system are divided into four categories:

Things that are missing.
Nice features that we'd like to retain.
Bugs that should be corrected.
Design flaws.

1.1 Missing

Our current system has some obvious deficiencies and could benefit from some useful additions. Many of these are already handled by, or could easily be added to, a newer version of RT.

Allow file attachments (diagrams, screenshots, other non-text data).
Handle mime-types in automatically appended mail.
Force an item's status to stalled when any of the items it depends on are still open.
When all the items that an item depends on have been closed:
- change the item's status to open
- notify the item's owner that the item is no longer stalled
Automatically notify the requester when an item is assigned.
Automatically notify the requester when an item is closed.
Automatically notify the owner when an item is nearly due.
Allow users to set preferences on items indicating the kinds of activities for which they want to be sent mail.
Remember more personal web-form preferences.
Provide new web forms for our clients:
- Simpler version of the run-on sentence.
- Simpler form to create new items.
- History of all their jobs.
- Simple summaries of their open items.
Provide intermediate privilege levels. e.g. to allow department heads to look at requests associated with their department.
Automatically reopen an item if the requester adds info. (Only if no owner? i.e. if mail wasn't sent to someone.)

1.2 Good

It also has some locally added features that, as far as we know, are not provided by stock software:

All updates get very visible date/author stamps, so that appended comments and mail don't appear merged together, something that would make it difficult to determine who said what when. Amazingly, this is not true of IST's own RT configuration.
Dependencies, See-Alsos, etc. are expanded to display the subject line, with dependencies graphically expanded to represent relationships.
Hostnames are expanded to show support level etc.
Client phone extensions, office numbers, departments, etc. are automatically filled in.
Programs to analyze workflow statistics (e.g. graphs of number of closed items).
A subscription code field allows us to track and report time spent on CS work for which we get reimbursed.
New fields allow cost-estimates and other-ids.
The run-on-sentence makes it easy for non-techies to request very specific selections.
Hyperlinks are generated for RT#'s embedded in comments and mail.
Most empty fields are not displayed, but there is a mechanism for expanding them when one wants to set their value.

1.3 Bad

Bugs in our current RT system are too numerous to list, but here are some that people find most annoying:

The first person to make a change to an item with an empty Responsible field gets credit for assigning responsibility to the owner. e.g. rbutterworth set Responsible to lcsuess, which is not true. If RT is going to fill in this field, it should give appropriate credit: e.g. RT set Responsible to lcsuess.
When sending mail, if one empties the To and Cc fields, and sends only to a Bcc list, RT magically sends to the requester anyway. By being so helpful RT can create embarrassing situations.
Updates received via mail are appended as is, including multiple format versions of the message and encoded data, making them noisy, ugly, or illegible. Only plain-text messages should be appended; anything that can't be properly formatted should be returned to the sender with appropriate suggestions.
Another bad feature of mail is caused by people that use the evil bottom quoting style promoted by Micosoft and other mail software. Bottom quoting results in large amounts of irrelevant or redundant material being included with each message. Unfortunately this is a user-education issue, and given who is on the other side, it's one we aren't likely to win.
After sending mail, RT goes to a Display window. One then has to use back, back, and refresh to be returned to the original Update window.
The Claim button drops all existing owners. It should only move or add you to the head of the existing list.
Some records contain bad characters (e.g. carriage return and tab) that cause problems for analysis programs.
The web forms have too much unnecessary whitespace taking up valuable real estate.
People use the term wish list for two different purposes. One of those has been used since the beginning, and is wired into the software. The other should be renamed to shopping list to avoid confusion. (But that's people re-education, not a software change.)

1.4 Ugly

Even when our system does work as intended, certain aspects of its fundamental design make it inconsistent and awkward to use and understand:

The web interface for our clients is too technical and not simple enough. It should be much easier to submit a request.
The login and identification process is unnecessarily awkward.
If fields are not entered in the correct order, some previously entered data will be cleared and will have to be reentered. Previous administrations have felt this was a user-education issue. Murphy would disagree.
Many of our local customizations are perhaps not necessary, and we should avoid having to carry them forward to future versions. Some candidates include:
- using insecure identd to authenticate the consultant user.
- requiring an xhiered environment.
The due-date and urgency fields aren't easy to understand.
- Given the urgency choices of 24 hour and urgent, which is more urgent?
- There an emergency urgency and an emergency queue, the two not necessarily matching on any specific request.
- There is a by date urgency, which can be filled in even though no due-date is specified.
- Similarly, one can specify a due-date without setting the urgency to by date.
There should be a clean separation between importance and urgency, where urgency can be calculated from the due-date, and importance can be thought of as how bad it would be if the item weren't completed by that date.
Conflating queues and priorities doesn't make sense. I don't know whether the concept of combining these two values into one came with the original RT or we invented it ourselves, but either way, it's a bad idea.
- The queues, activities, and statuses are poorly defined:
  - Queues:
    - Emergency
    - Current
    - Upcoming Requests
    - Upcoming Projects
    - Wish List
  - Activity:
    - Research
    - Teaching
    - Administration
    - Infrastructure
    - Student Requests
    - Unknown
  - Status:
    - Done
    - Timeout
    - Can't
    - Noreply
    - Notreproducible
    - Forwarded
    - Merged
    - Forget
    - Gaveup
    - Sigh
  Instead, these attributes should be orthogonal, and each set of categories should be mutually exclusive and collectively exhaustive. The Status values should reflect how we view the item, and will change with time. The Queue and Activity values should be part of the original definition of the item and should normally never change.
  - Move Current, Emergency, and Wish List from Queue to Status, as additional open statuses.
  - Merge the Activities into the Queues:
    - Ongoing Duties (a new category)
    - Research (now in Requests queue)
    - Teaching (now in Requests queue)
    - Administration (now in Requests queue)
    - Infrastructure (now in Projects queue)
    - Student Requests (now in Wish List queue)
    - Unknown (eliminate this one; if we don't know what kind of problem it is we shouldn't maintain an item for it).
- It should be noted that other campus groups base their queues on the people assigned to do the work rather than on origin and purpose.
Many of us have ongoing duties that are not reflected in specific RT items. Having items therefor would allow RT to present a more accurate view of how all our staff spend their time. Such ongoing items would also provide documentation that would make it much easier for others to assume our duties should we be away for a significant period of time. Previous administrations have been opposed to creating a queue for such things.
Some people feel that there are too many methods by which clients may enter requests, including web-forms, newsgroups, telephone, gripe commands, e-mail, consultants. Others feel that the more ways we offer for clients to tell us their problems, the more likely they are to do so, in a way that is most comfortable for them.
Many other problems are not due to the specific software implementation, but are almost entirely caused by how people use, or don't use, request tracking systems. Although it has significantly improved over the last year (with Jim Bell's asking us to clean up our queues, and Robyn's method of handling larger projects), the way MFCF uses our request tracking system could be much better. In particular, our clients should be able to look at our queues and get a reliable image of what we are working on and what our priorities are.
- Managers should take greater control over item priorities.
- Managers should do more to prevent late jobs. Most items don't have due-dates, but despite that, 5% of our open items are already late. If we plan on encouraging the assignment of more due dates without correcting this current problem, it will only become worse, and the due-date field will lose its usefulness and become meaningless.
- Staff should be a little more conscientious so that Managers won't have to spend much time dealing with the above two points.
- We should more closely follow proper procedure. e.g. many items marked current haven't been worked on in years, while other items that are upcoming show a lot of recent activity.
- Some items don't get assigned for a long time (e.g. there are still some that have been unassigned since October).
- We should make sure that each item has a meaningful title. e.g. if a client submits a request as new account, we should immediately change it to something like create fbaggins accounts on student.math and Nexus. i.e. someone looking at only a list of subjects should be able to have a good idea of what each item is about. Blaming inappropriate subject lines on clients is no excuse.
- Once corrected, subjects should remain stable. Continually changing the subject to describe what needs to be done next to complete the item isn't the right way to do it. Either the item should be broken into several sub-items, or the summary field should be used to indicate the current state of the item.
- We need stricter rules about who can make changes to the owners field, and under what circumstances.
All of this is simply a matter of getting management to set and enforce policies.

2. Possible Directions

Each of several solutions to our problem has obvious advantages over the others, but none of them are strong enough to be the obvious right choice.	Each of these solutions also has obvious disadvantages. In general these disadvantages can be seen in terms of the amount of human work they will require.
2.1 Design our own request tracking system from scratch.
Advantages The most ambitious option would be to replace the RT system with a completely new in-house design. This would enable us to: implement it in a language that will run far more efficiently than the Perl interpreter and be easier to understand and maintain. eliminate every annoying feature of the existing system. add every missing feature. have full control over every aspect of the software.	Disdvantages This would be, by far, the largest project, requiring a significant amount of dedicated labour, not only for the initial design and implementation, but also for ongoing maintenance and updating. And for those that consider past investments as a useful factor when making decisions, it would make it virtually impossible for us ever to switch to the real RT system, regardless of how bad our own system turns out.
2.2 Upgrade our existing RT system to version 3
Advantages Version 3 provides many advantages over the ancient version of RT that we are running now. Our base version was picked up so long ago that I haven't been able to find anything listing all the features that the vendor has added since then. But some obvious enhancements include: attachments enforced dependencies enforced policies custom fields better documentation better defined API ability to override default functionality a command-line interface full i18n support mason-perl preprocessing of web pages significantly improved source code easier addition of local modifications easier preservation of local modifications across version upgrades Use of Mediawiki to manage content.	Disdvantages This would require extensive work to incorporate the local modifications that we consider worth keeping. In particular, we have created many customized web-forms and extra tables, and the software has access to other databases (e.g. inventory, uwdir). Fraser Gunn, of CSCF, has now created an item to determine what features we added to the original version: RT#56353 - RT - document the changes we made and our design decisions. Just as some people use the pretentious British style of spelling words with an -ise ending, rather than the correct everywhere -ize, RT's writers use the American alternate (like lite) spelling requestor, rather than the requester spelling that is correct everywhere.
2.3 Transfer our data to IST's RT system
Advantages Using IST's existing system (version 3) would provide all the advantages of upgrading our own system, without having to provide or maintain any hardware. It would also mean that MFCF wouldn't incur the overhead of managing the RT system or the machine(s) on which it runs, nor of installing vendor updates or backing up the data. We would also have a choice of: Sharing IST's configuration and database, which would provide full support from IST. Having IST support a separate instance for us, which would allow us to make modifications and additions that IST might not otherwise allow.	Disdvantages In addition to the problems mentioned above, this would require a little extra work to incorporate the local modifications that we consider worth keeping. And if we went with the shared-data option, we would have to renumber all of our RT items and might even not be allowed to implement some modifications.
2.4 Go with an existing non-RT system
Advantages No obvious advantage.	Disdvantages RT's web site says: Every ticketing system sucks. Here at Best Practical we're really proud of the fact that RT sucks less than everything else out there ... .. I've not seen anything that contradicts this fact.
2.5 Stay with our current software
Advantages Staying with our current software offers two very significant advantages over switching to a new version: It's free. It's already done. And staying our course would not preclude making changes to eliminate many of the existing annoyances.	Disdvantages While we can change some of the features that we don't like, and add new ones that we would like, it's unlikely that we'd ever bother to modify the underlying code that does all the work, which presumably the vendor has improved somewhat during the previous ten years. We'd also miss out on features that might be too difficult to retrofit into the old code. And we'd have to track new features and improvements that the vendor adds and reimplement them ourselves (given our track record with the existing package, that isn't likely to happen). The other choices would almost force us to perform a massive cleanup of the existing data. If we stay with the current software, experience shows that we're unlikely to spend the time on this desirable side-effect.

Each of several solutions to our problem has obvious advantages over the others, but none of them are strong enough to be the obvious right choice.

Each of these solutions also has obvious disadvantages. In general these disadvantages can be seen in terms of the amount of human work they will require.

2.1 Design our own request tracking system from scratch.

Advantages

The most ambitious option would be to replace the RT system with a completely new in-house design. This would enable us to:

implement it in a language that will run far more efficiently than the Perl interpreter and be easier to understand and maintain.
eliminate every annoying feature of the existing system.
add every missing feature.
have full control over every aspect of the software.

Disdvantages

This would be, by far, the largest project, requiring a significant amount of dedicated labour, not only for the initial design and implementation, but also for ongoing maintenance and updating.

And for those that consider past investments as a useful factor when making decisions, it would make it virtually impossible for us ever to switch to the real RT system, regardless of how bad our own system turns out.

2.2 Upgrade our existing RT system to version 3

Advantages

Version 3 provides many advantages over the ancient version of RT that we are running now. Our base version was picked up so long ago that I haven't been able to find anything listing all the features that the vendor has added since then. But some obvious enhancements include:

attachments
enforced dependencies
enforced policies
custom fields
better documentation
better defined API
ability to override default functionality
a command-line interface
full i18n support
mason-perl preprocessing of web pages
significantly improved source code
easier addition of local modifications
easier preservation of local modifications across version upgrades
Use of Mediawiki to manage content.

Disdvantages

This would require extensive work to incorporate the local modifications that we consider worth keeping. In particular, we have created many customized web-forms and extra tables, and the software has access to other databases (e.g. inventory, uwdir).

Fraser Gunn, of CSCF, has now created an item to determine what features we added to the original version: RT#56353 - RT - document the changes we made and our design decisions.

Just as some people use the pretentious British style of spelling words with an -ise ending, rather than the correct everywhere -ize, RT's writers use the American alternate (like lite) spelling requestor, rather than the requester spelling that is correct everywhere.

2.3 Transfer our data to IST's RT system

Advantages

Using IST's existing system (version 3) would provide all the advantages of upgrading our own system, without having to provide or maintain any hardware. It would also mean that MFCF wouldn't incur the overhead of managing the RT system or the machine(s) on which it runs, nor of installing vendor updates or backing up the data.

We would also have a choice of:

Sharing IST's configuration and database, which would provide full support from IST.
Having IST support a separate instance for us, which would allow us to make modifications and additions that IST might not otherwise allow.

Disdvantages

In addition to the problems mentioned above, this would require a little extra work to incorporate the local modifications that we consider worth keeping. And if we went with the shared-data option, we would have to renumber all of our RT items and might even not be allowed to implement some modifications.

2.4 Go with an existing non-RT system

Advantages

No obvious advantage.

Disdvantages

RT's web site says: Every ticketing system sucks. Here at Best Practical we're really proud of the fact that RT sucks less than everything else out there ... ..
I've not seen anything that contradicts this fact.

2.5 Stay with our current software

Advantages

Staying with our current software offers two very significant advantages over switching to a new version:

It's free.
It's already done.

And staying our course would not preclude making changes to eliminate many of the existing annoyances.

Disdvantages

While we can change some of the features that we don't like, and add new ones that we would like, it's unlikely that we'd ever bother to modify the underlying code that does all the work, which presumably the vendor has improved somewhat during the previous ten years. We'd also miss out on features that might be too difficult to retrofit into the old code. And we'd have to track new features and improvements that the vendor adds and reimplement them ourselves (given our track record with the existing package, that isn't likely to happen).

The other choices would almost force us to perform a massive cleanup of the existing data. If we stay with the current software, experience shows that we're unlikely to spend the time on this desirable side-effect.

3. Recommendations

It's clear from the responses that changes need to be made to our current request tracking system, and given the lack of conflict between the suggestions, it's fairly clear what they need to be:

Simpler and more useful forms for clients.
Simpler and more obvious terminology and structure.
Less gratuitous local customization
Better documentation and education of both staff and clients.
More well-defined policies, and stricter enforcement thereof by management.

If MFCF were still in its golden age, I'd have no trouble recommending that we design our own system from scratch, market it worldwide, and become rich and famous. Unfortunately that isn't the case, and such a choice would require a large commitment of human time, a resource that, as far as I know, we can't really afford now. And even if we could afford it, perhaps there are other projects on which that time could be better spent.

If we are truly lacking in resources, staying with our current system seems like the only choice, though we could spend some time, in cooperation with CSCF, to clean up some of its more annoying features.

Otherwise, if MFCF is willing to commit the necessary resources, the best direction to take seems to be with asking IST to host our own RT instance. This would give us more freedom than sharing their system, while eliminating much of the ongoing responsibility of maintaining the hardware, data backups, and vendor updates.

4. Unanswered Questions

Even after we have made a firm decision about which direction to proceed, there will still be a number of issues to resolve:

Who will do the work? Most of the experience and expertise with RT is in CSCF and IST. And the software itself is mostly Perl based.
Will CSCF go along with our decision, stay as they are, or make a similar decision in parallel?
Many of the existing items are of relevance to both groups, so if we do separate our items from the CS data, how will we divide it up?
Would all the historic closed items (which are of practical use as reference articles) be moved into the new database? If not, what would we do with them?
The most troubling problem is that of transferring our existing data to the new system. We might have to convert from version 1 to version 2, and then from version 2 to version 3. Or worse, ours might actually be so old and customized that we'll have to write ad hoc processes to do the conversion. Or perhaps, given how dissimilar the database tables are (the new RT has twice as many tables, only one of which shares the same name with our current tables), we won't be able to move our old data at all. (One easy out would be to continue using the old system for existing items, and the new system for new items. This sounds like it would be awkward and impractical, but if CSCF were to continue supporting the old system and allow us to continue leeching, it would not be an unrealistic possibility.)
This report has not considered the relative merits of individual suggestions. Any decision as to which are worth implementing and which aren't still has to be made, but that would perhaps be easier once we know which direction we will be taking.