Joris Evers

Subscribe to Joris Evers: eMailAlertsEmail Alerts
Get Joris Evers: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: Airline Information Technology

Efficient, Comfortable & Secure Airlines with IT: Article

Virtual Case Study: How An Airline Can Find Efficiency With Unix

How can an airline fly today? By going way off scale with Unix. The story of SafetyJet International - Part 2

Development rational and processes
The decision to try to develop what amounts to a real-time operating system for an airline is a potential business killer and correspondingly questionable. My fundamental arguments for doing it are:

  1. The SafetyJet business model is not a good fit for any established system that we're aware of. The one we liked best, built around a core reservation system that provided the foundation for Southwest Airlines' success and is now powering several other airlines to pre-eminence in their markets, seemed to require more than 100 ancillary systems and is dependent on MPE/3000 -- a product that HP has recently end-of-lifed.

    The vendor assures us that continuity will not be a problem and we accepted that, but it is my belief that good software reflects both the business it serves and the mentality of its developers. As companies using other major applications ported from world-class integrated database environments (e.g. PICK, OS/400) have demonstrated many times, the magic that makes the thing an unusual success in its original environment is usually missing from the ported product.

  2. You don't get competitive advantage by copying the other guy's software. At best, you'll always be at a disadvantage because his people have more experience with it than yours do.
  3. The actual costs and risks are not substantially different from those involved in setting up all of the interfacing and related middleware pieces needed to string together the 100 or more packaged applications we would need to license, and make work, if we went the licensed software route.
  4. Winning at the airline game is a matter of gaining and holding many small advantages. A core system built around an all-encompassing operational optimization model should, if it works, deliver those better than any other approach can.
  5. If successful, our systems operating cost will be less than 25 percent of competitor costs and translate to long-term competitive advantages for SafetyJet.
  6. A successful custom system can, if managed properly, provide greater long-term security with lower downstream risk, than any licensed alternative regardless of safeguards.

Development isn't as frightening as it used to be. We're not going to be spending years building a static specification nor will we be spending time and effort on database development, front-end development, or any of the internal or network infrastructure -- all of these pieces are fully predefined. What we are going to be doing is connecting a complex mixed integer linear programming formulation to a rapid prototyping environment that gets us nearly instant user feedback on functionality.

Many of the ancillary functions will, furthermore, be licensed from third parties and then run at arms length for security reasons. None of the document and EDI/interface management systems will, for example, be either home grown or directly linked to the production systems. Whether licensed or mandated, these systems will run independently of the real-time airline operating system with fully proceduralized manual intervention on any required data transfers.

Internally to the real-time system, we see the scheduler driving the revenue cycle just as it drives the expenditure side. In effect, marketing determines a basic set of flight plans well in advance but reality, in the form of weather, Murphy's Law, regulatory action, and passenger/freight demand, will determine the actual minute-by-minute schedule. The core idea, therefore, is to start with a basic solution, add demand and reality information, and then operate all parts of the business according to the best schedule the constraints will allow. In that vision, the two high-volume sources of the demand and reality information needed are the ticketing and operations pieces -- that's one reason they'll be built using a shared database.

The biggest single problem with developing this system is that we don't have a live airline to practice on. Normally, as described in my Unix Guide to Defenestration, I strongly prefer adaptive prototyping: build something, hand it over, fix or replace it, repeat until perfect. Here, however, we don't have a working airline within which to develop the functional prototypes needed.

This means bootstrapping the system at the same time that we bootstrap the airline. That's even scarier than the development decision by itself -- but I don't see many choices. As a result, my development plan is to do all three at the same time:

  1. The business model
  2. The system
  3. The airline

To provide a starting point and get some idea of the complexity, I asked two teams, each with three people one of whom is a constraint programming expert, to develop a structural framework that:

  1. Has a common Web front-end for all users.
  2. Handles making, changing, or dropping reservations based on having the customer (or call center agent) pick a starting point, a destination, and a latest arrival time to generate one or more choices from which a decision can be made.
  3. Adopts a database structure that allows the Solver pre- and post-processes to pick up constraint data, and write solutions, with an absolute minimum of delay.
  4. Can provably deal with conflicting reservation attempts; last-minute changes; bar code and/or ticket printing errors; missed freight scans; and, incorrect passenger counts, while immediately incorporating resource allocation changes coming from the Solver.

One of the rules I make for this is that nothing will be done as a database stored procedure although triggers are allowed for database products lacking native referential integrity support. This maximizes code re-use and hedges against database licensing schemes based on processor use.

When Oracle imposed processor-based licensing recently it became obvious that applications with modularized external code could limit their database use to purely database functions and needed far fewer processor licenses than those that relied on stored procedures. For many companies, that meant millions of dollars in license fees went to Oracle when better application design could have avoided the expense.

Compared to the real thing, the result was trivial but the winning team's 16-day masterpiece did let us simulate a multiple airport, multiple user reservation system and demonstrate direct linkage, for both passengers and freight, to both the Solver and operations control application components. Interestingly, but uninformatively, their demo Solver set-up ran LINDO solutions for a 25,000-row, 80,000-column problem in less than two seconds using an 8-gigabyte, dual 400-MHz, Sun 450 with locally attached disk.

Advertising expense vs technical credibility
The winning team used Unify Vision and ran the demo on U2K rather than Informix ODS, but the tools are database-independent and scale, within the Unix/Smart display architecture, essentially linearly as you add users.

On a personal basis, I have used this product set for demos since 1988, but have yet to have a client adopt it for serious development work and production. Without exception, they've declared Unify is likely to go under real soon now and bought into better-marketed products that died or were absorbed by Microsoft or CA.

I've never understood this -- just as I don't now understand people buying Microsoft Office rather than adopting OpenOffice.Org because the fact that it costs big bucks gives them more faith in its future. If you can explain it without using words like "stupid," please send me a note!

Combined with the results obtained by the Forte/iPlanet team, we developed a consensus that perfect execution would let us tie the licensed pieces together under one Informix ODS based application with less than 16 man-years of effort and four major adaptive iterations of the core user interaction pieces. We'll probably never find out how realistic that is, but it produced the brilliantly indefensible $2 million dollar estimate that went into the subsequent business plan.

Staffing
Finding qualified staff isn't easy. By the time we're done, we'll need about 65 people, 60 of them regular IT staff, and the other five comprising the office of the CIO.

Many of these people need special skills, particularly those who will work with Boeing or FAA/DOT-mandated systems, the Solver, and the development tools we'll be selecting. Our general recruiting strategy is to recognize that we don't need them all today, start with a few good people, and let those people recruit others for us.

Two of our team members, for example, are full-time professors -- one of whom teaches several operations research courses -- and we'll count on them to recruit some promising graduate students for us. For other start-up jobs we will use existing team members and people recruited via advertisements in the two metropolitan areas we'll be headquartering in. The key decision here, however, is to hire good people as we come across them rather than waiting for the actual jobs to materialize -- at worst, we can always sell their services to third parties as consultants.

Once people are in place, we'll adopt the usual Unix strategies:

  1. Everyone is assigned basic personal responsibilities that require some expertise, initiative, and commitment to carry out.
  2. People are encouraged to work directly with users and act as user advocates within the data center.
  3. People are encouraged to swap themselves into, and out of, whatever project team needs, or doesn't need, their expertise. Team members can delegate jobs, but not responsibility. Someone hired as a DBA can, for example, delegate some tasks to someone previously limited to Solaris sysadmin work but retain full personal responsibility for database integrity and availability -- while actually spending most of his time helping to get documents to display properly on PDAs carried by mechanics. Done properly, this combination of personal responsibility with tremendous job flexibility, produces highly productive, happier, workplaces -- and graduates who become CIOs in later life.

As usual, there are two key goals to this:

  1. To ensure that users, not systems management, dictate most systems decisions.
  2. To ensure that systems people get cross-trained on just about everything in the operation but never lose their sense of personal responsibility for one or more key functions.

Capital cost
The full capital spending plan has more than 600 entries ranging from a $50,000 software license with $500,000 in installation and formulation support services, to a set of $225 stackable hubs. Actual expenditures are scheduled to occur in parallel with the airline bootstrap operation. The first six months of work, for example, will take place in one center with only a Sun V880 server and some workstations on a local network. To stay on schedule we will need, however, to commit to some steps -- like data center construction or CPLEX licensing -- well before the matching approvals and go-aheads come in. To reduce our exposure we hope, therefore, to negotiate risk-sharing arrangements with some key vendors (e.g., Sun) under which we can cancel receipt or delay payment at the last minute if regulatory approvals are denied.

The first five months will see us spend about $1.8 million on hardware, licenses, and staff. The next three or four, exclusive of contractor commitments, will about double that but the major expenditures start about 60 days before the first airplane arrives. During those last 60 days all the networks and servers will go in while the last of the staff are brought up to speed in both data centers and last minute panic sets in. We'll blow through something like $9 million during that period. That's serious money, but much less than what the airline will spend on reconditioned highway buses, departure desk installations, or airport facility re-construction.

During the initial operational phase:

  1. Both data centers will have dual Sun 6800 machines, each with 24 CPUs, 128-GB of RAM, and about 10 TB of on-line storage.
  2. The 20 local offices will account for about 800 of the 17-inch SunRays, 200 printers, 20 wireless barcode reader/scanner networks, about 100 NCD 21-inch smart displays, and just over 200 handhelds.
  3. There will be about 20 of the 21-inch NCDs allocated to the distributed call center each with a DslPipe or comparable VPN capable router.
  4. The two administrative offices will each have about 80 of the NC900 smart displays, 40 or so SunRays, and a handful of Mac G4 workstations.
  5. We will need about 90 IT workstations, mostly NC900 smart displays but including about a half-dozen Model 60 graphics displays and a few Sunblades for higher speed applications.
  6. We will need a Windows 2000 server and perhaps as many as a dozen PCs -- and a full-time support person -- to handle government- and vendor-mandated communications functions requiring Windows clients as well as format conversions for incoming Windows file formats.

Supporting all this will take about 18 people in each center and three in the office of the CIO. That will grow rapidly to about 65 when we ramp up for full operation -- well in advance of the arrival of more airplanes and approvals. At that time:

  1. Each center will need about nine people for basic network and related "power-on" activities.
  2. The development teams that started as:
    • two prototyping teams of seven people and a full time joint development co-ordinator to ensure directional continuity as both groups work with the airline bootstrap operation to adapt, expand, or invent the required code. As a matter of policy all code will be written twice, once by each team, with adoption decided after users, and other developers, have exercised both sets of changes to the evolving prototype;
    • an independently chartered development team of two will work on the statistical quality control and geophysical reporting applications. This team will cover both data centers with one member posted at each center;
    will carry on refining and improving the system.
  3. The major document management sub-systems will have staff dedicated to their care and feeding. This adds three more bodies to each center.
  4. The tax compliance team working within finance will have dedicated IT support in each data center. This adds two more people to the total.
  5. The EDI and other third-party data exchange systems will have at least one person specifically assigned to them in each center.
  6. Each data center will need at least one person to baby-sit the few Windows operating systems on PCs needed to deal with incoming file conversions and meet mandated client environments for regulatory and supplier notifications.
  7. The user support team will comprise about four people in each center. Their role will not, however, be that of traditional help desk staff because SafetyJet will not deploy Windows to users and so needs no support staff in that role. What these people will do, instead, is install local office gear, provide training and related support, and liase between local office users and the data centers. These people, in other words, will primarily act as roving ambassadors and teachers.
  8. The audit and security group will be independently staffed and report directly to the president and board but be housed within the two data centers. Their staff will grow from one person at each location during startup to about one person for each three to five destinations as SafetyJet matures.

For a total complement of about 65 IT people and perhaps as many as 20 others who work out of the data centers but are not carried on the Systems budget.

Getting these people in place will be hardest part of the scale-up process needed to sustain 100-airplane operation. The technology part is made easy by the SPARC architecture, which lets us run the same code on anything from a single-user workstation to a 15K.

Depending on the Solver decision, scaling up will involve either:

  1. Adding a Starfire 15K (or its descendent) to each data center, migrating the RTAOS applications from the two 6800s to the new machine and dedicating the former to backup and experimental processing roles.
  2. Migrating the Solver component of the RTAOS from the 6800s to the Linux (or Solaris) GRID.

There is no code or significant procedural change within the overall system beyond that needed to accommodate the Solver solution chosen.

More Stories By Paul Murphy

Paul Murphy wrote and published 'The Unix Guide to Defenestration'. Murphy is a 20-year veteran of the IT consulting industry.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.