Joris Evers

Subscribe to Joris Evers: eMailAlertsEmail Alerts
Get Joris Evers: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: ERP Journal on Ulitzer

ERP Journal: Article

Virtual case study: Cutting IT costs with Unix

How to reduce IT manpower, hardware and software costs, and improve services.

(LinuxWorld) -- Editor's note: The following scenario is that of a new systems manager brought into a small company "to bring systems costs under control." The company, its staff and products are fabrications but the situation presented, and the remedies offered, reflect the author's recent experience with real-world clients that were overly optimistic about the prospects for e-commerce applications. Whackabilly is imaginary, but the conditions, decisions, and outcomes described are broadly based on real events.

I am the new Information Technology (IT) vice president at Whackabilly Toy. My company makes the Ventables series of training aids and the Whackabilly line of deformable plastic dolls along with the Microsoft Windows software that goes with them. The products are made out of a dense foam material incorporating up to about 30 percent sand for weight and feel, and have one or more pressure sensors embedded near the center. The toys are typically about eight inches high. The training aids usually nearly life size. All products ship in Styrofoam kits complete with instruction manual, CD-ROM, and mouse cable splitters for both serial and PS2 connectors. The Whackabillies also include a foam-rubber mallet.

When the dolls are deformed the software grabs the amount, direction, and initial duration of the deformation to trigger an appropriate action. For the toys, this is a sound and graphics display set according to user preferences. For the training aids, it is usually a measure of the source, direction, and strength of the pressure applied. We ship a default set of these with each product but also provide an exchange service for users who make their own and wish to share them with others. These, needless to say, are often hilariously obscene and far more creative than the cartoonish violence and sound effects our lawyers let us ship.

Our biggest sellers, by volume, resemble public figures. The greatest revenue growth is now coming from serious training aids like our Heimlich model.

The manufacturing operation is reasonably simple. The sensors and Application Specific Integrated Circuit (ASIC) are custom built for us in Singapore and shipped to an extruder in Detroit which produces the dolls. Kit assembly, packaging, and shipping are co-located with the executive offices and sales center in Omaha.

The company currently employs about 109 people of whom 35 people are in sales and 26 are systems staff. Of these 26, only two work on the software shipped as part of the product. Fourteen are fully committed to the e-commerce effort and two or three work on it part time.

Today, all computer work is done using Windows-based gear. The previous director of IT precipitated his own demise by requesting nearly $1 million to upgrade everything to Windows 2000 on new Compaq hardware. The president's words to me early on in the relationship were, "Bob was out of control and wanted to throw money, good money, away on gear we don't need." Nevertheless he agreed to give me $400,000 for new capital investment and to front another $400,000 from next year's IT budget to support changes needed now.

Action agenda

In the longer term the strategy will be to turn this place around; to transform the IT role from that of an expensive after-thought to being a key part of the business. I'm taking two small steps in that direction right up front:

  1. Working with the VP of Sales, evaluate Palm Pilots and new Web services software to replace 22 IBM ThinkPads now checked out by sales people when they work outside of the office.
  2. Work with the VP of Finance ostensibly to develop the cost/benefit case for spinning-out our current Web-based retail operation as a separate company, but really to build rapport and get her to understand that systems costs are not measured by expenditures but by foregone opportunities -- just as the cost of a cold is measured by time off work and not by the cost of the drugs needed to control it.

In the short-term, however, there are urgent problems that need to be addressed:

  1. The most difficult of all the transition jobs starts with meeting with each member of the IT staff and explaining the changes to come. Most of them know the company can't sustain present IT staffing levels without an enormous, highly improbable boost in sales, but that doesn't relieve me of responsibility for making the cuts. As usual, this won't be made easier by the fact that some are genuinely likeable but incompetent, some are afraid to talk honestly, some are competent but angry, and a few have user or executive relationships to be wary of. No matter what I feel, however, at least half of the 26 have to go quite soon to get staffing down to anything remotely reasonable. Ideally, another two or three will go next year.
  2. The e-commerce effort threatens to eat the company and now requires two rackmounts of NT servers. This needs to be moved to a single Unix machine to eliminate the need for 24 x 7 staffing and cut workload from nine people plus a manager to part-time work for two. Not only are costs out of control, but volume is low and the sales people are unhappy because this thing is really a Web store. What Sales needs is automated fulfillment with customers placing, and tracking, their own orders.
  3. The e-commerce Unix machine will get a write-only laser disc sub-system to squirrel away copies of all outgoing and incoming e-mail. Today, the company has no defense against someone who:

    • Claims they e-mailed us something, but didn't.
    • Claims to have received an e-mail no one here sent.
    • Claims workplace harassment or other allegations arising from e-mail abuse.

    Put some disk-control procedures in place and we'll have the best of all defenses -- unimpeachable records.

  4. We're turning down orders (!) because the product software doesn't run on Linux or Mac OS.
  5. There's lots of user unhappiness ostensibly about being stuck with two-and-a-half-year-old Windows 98 machines because of all the money going into the e-commerce venture. Of course, the complaint is really about the failure by the systems group to meet expectations. To meet expectations, the company needs to review and upgrade its design software, clean up the executive information system, and try to bring more functionality to Sales while getting upgrade and support costs under control.

The plan

The technology plan is easy. We will:

  1. Swap all production services onto two Sun 850s.
  2. Transition users to "smart displays" initially with software running on the existing NT servers and gradually shift them to purely Unix software. (We discuss more about smart displays later.)
  3. Require Linux on company-supplied home machines used to access office documents.
  4. We're going to go looking for new and better software for the design, sales, and reporting functions.

The IT staff plan is equally straightforward. Next week, I'll tell a special IT staff meeting that:

  • Five people will not be attending because they've been let go the night before.
  • Ten more have to go by Christmas.
  • All 15 will get three-months severance, and we'll carry their medical coverage until March 31st of next year.
  • We will be hiring two new Unix sysadmins with considerable Solaris and RDBMS experience as quickly as we can find qualified applicants.
  • The two-person product group is being expanded by one temporarily, preferably by transferring an existing staffer, to add Linux and Mac OS X compatibility to the product.
  • The average salary will rise, by the late next year, from around $38,000 now to around $50,000 and the people who earn the right to stay can expect more responsibility, more training, and more scope for personal initiative.

At about the same time the VP of Finance will send a personal e-mail to all employees outside IT informing them of these changes. She will also tell them that people whose desktop PC gets replaced by a smart display will have the option of taking the PC, minus licensed software, home as a gift from the company. Since most are reasonable quality Compaq 500s and 550s they'll make great Linux machines. We'll host a series of Linux clinics for employees timed to coincide with shutdown of the machines for office use.

Attendees will load Linux on their own machines and get a tutorial introduction to the system and key tools such as the OpenOffice.org suite, Pipeline-based VPNs, and the advantages of remote X Window System access.

The Linux installation and training offer is far more important than it looks. Physical control of the systems has to be centralized for cost, security, and performance reasons. Functional control has been pushed out to the users. That's the reverse of the present situation in which users have physical control of desktop computers but little or no say in how they are used.

By trying to get as many home users as possible to switch to, and learn, Linux I'm hoping to apply the maxim that "knowledge is power" to help shift systems control toward the user side. If I can create a pool of knowledgeable users, they will not only be more effective in working with IT staff on identifying and developing new services but their independent knowledge and information access will also allow them to function as a check against any power grabs by IT staff.

The budget

There are two months left in this year's $1.2 million IT staffing budget, and I've committed to reducing that by $400,000 next year. As a result, my change budget will be spent on:

Servers
I'm ordering two new Sun Ultra 10 workstations with 21-inch screens, and two 850s each with 4 CPUs, 8 gigabytes of RAM, and two sets of 6 x 36-gigabytes drives on the condition Sun provides free iPlanet licensing, an immediate loaner 450 or comparable until delivery and acceptance of the 850s, and fully test and configure all three on arrival here -- including setting up the iPlanet e-mail/scheduling services and the HP Sure Store 400 Optical disk writer ($12K). Since I'm prepared to pay retail (about $175K all-in) and the dot bomb hype is well behind us, I have every confidence Sun will agree.

I will convert two of the existing Web servers to run Windows 2000 Advanced with Terminal Server by stripping everything I can out of the other two and shutting those down. That'll give me four 550-MHz Xeons in each one along with 8 gigabytes of RAM and 260 gigabytes of accessible disk. Licensing will be an expensive throw away at about $76,640.

Upgrade from NT server to Win2K Advanced Server with 25 CALS $1,9191
Upgrade from NT BackOffice to Win2K Servers with 50 CALS $3,895
Upgrade to 80 CALS $18,560
Terminal Service with 80 CALS $10,476
Upgrade 80 Microsoft Office 98 licenses for TS $19,120
Total Microsoft Licensing $53,074
Add Citrix Metaframe license for 70 concurrent users $22,500
Total Licensing $76,640
1Pricing is from the Microsoft and Citrix sites as of Nov. 6, 2001.

However, this will enable the users to continue using the software they know as they get used to the smart display environment. As we bring in replacements for the accounting and other software they use now, I'll be able to phase this stuff out instead of endlessly paying to upgrade or replace it.

Staffing
My goal is to be down to 12 full-time staff by February 1 of next year at a buy-out cost of about $190,000 including the extended medical coverage. I'll give the remaining staff an immediate raise to about $44,000 on February 1 and then raise that again to about $50,000 in August. That should meet the commitment to spend no more than $800,000, fully burdened, on salaries and related costs with a handful of change left over for emergencies.

Initially, two people will be assigned to each server team and be responsible for the full range of operations from making sure the UPS has working batteries through to user desktops but that will increase to three as applications come off the Citrix systems. In the longer run, the three product-support people will get the additional responsibility of running any truly necessary Windows stuff on their test systems. Two will be assigned to new project development initially, and I'll need one to play vice-principal looking after security, discipline, vendor meetings, and paperwork, for a total complement of 12.

Smart displays
I'm ordering three new Macintosh G4 workstations, one a cheap model for testing and development at about $3,000, and two with dual CPUs and various Adobe tools for use in packaging and advertising materials design. The high-end Macs cost about $9,000 each mainly because of the fancy 22-inch LCD displays and the Adobe licenses. Almost everyone else will get an NCD NC900 smart display with a standard 21-inch CRT at $1,900 each for a total of $165,300 for 87 units.

What's a "smart display"? As discussed in A strategic comparison of Windows vs. Unix, it is a desktop graphics device intended to handle the user interface components of an application running on a server. Most provide and extend the capabilities of such predecessors as the "dumb" terminal, X terminals, and the Microsoft "thin client."

Networking
Today, there are firewalls, routers, internal switches, and various other layers of gear that keep a couple of people busy. All of that is going away, as is the current 10-megabit/second service connection to a local carrier. WorldCom will be putting in two burstable T1s to each of our two service nets at a combined annual cost that's a little more than we're paying now. We'll get less nominal bandwidth but higher actual performance and much better stability.

The T1s will be hooked up to two Pipeline 220s at about $2,000 each to provide basic firewall protection where it belongs -- in the router. Internally, all the other stuff is going away in favor of a 16-port switch stack that feeds unmanaged hubs in user areas at an estimated cost, mostly for re-wiring and new power connections in user offices, of about $11,400.

Externally, WorldCom will also provide company-paid home user connectivity for sales and executive personnel who currently have, or are otherwise authorized to get, company-paid home PCs. These will become Linux machines connected to the Internet via Ascend Pipelines, again with built-in firewalls but, more importantly, with built-in VPN capabilities that let traffic to and from the office system be fully protected without imposing burdens on other traffic.

Software conversion
The existing Web services software is eating an enormous amount of resources. Not only does it need two rackmounts of little servers to run, it needs a 24 x 7 babysitting service and an additional staff of three day-shift programmers who develop for it. The project manager, who is definitely on my short list for next week, shamefacedly admitted both Code Red and Nimda felled Whackabilly. It's actually worse than that. They had to re-install NT on the four servers used for user file exchanges in early September, and promptly got nailed again by Code Red and Nimda.

The people in Sales tell me that most of this stuff does fundamentally the wrong things. The user file exchange works well but what they want for the other stuff looks a lot more like a CRM/supply chain system from PeopleSoft or Oracle than the combo Web store and batch order service that seems to have evolved here. The application does not, however, warrant the real heavyweight stuff because there's no ERP backend to link to and no need for one either. We're just too simple to justify the overhead involved.

The argument for assembly of a custom solution is very strong. I want to be running on the new software January 1 so I'm hedging my bets. We're getting four vendors of packaged solutions in to show their products to our sales staff; running an internal design competition for a better custom solution; and, asking two companies, one using Unify Vision and the other specializing in PHP/PostgresSQL applications, to provide custom designs and matching quotes.

Impact summary

Impact of changes at Whackabilly Toy Inc.
  1. IT staffing cut from 26 to 12
  2. Total IT cost for 2002 cut from $2.6 million to $1.6 million
  3. Macintosh, Linux, and BSD access added to product line
  4. Marketing materials development moved to Macs
  5. All major systems now made redundant
Calendar 2002 Projections
Before ChangesAfter Changes
Impact on Users Average desktop application crashes per week1 21 0
Average server failures per week 12 0
Performance (Load 1,000 e-mail messages) 2 minutes 1-3 seconds
Screen size 15-inch, 800 x 600 21-inch, 1600 x 1200
Access to software Windows 98/Back Office Concurrent, multi-host, Win2K/Unix
Access from home Dial-in, PC-Anywhere Autoconnect VPN
Impact on Costs (in thousands) Staffing Costs (fully burdened) $1,200 $800 (66%)
Cost of Separations $75 $115
Planned Capital Expenses $1,036 $435 (42%)
Other operating expenses (mainly licensing & networking) $325 $172 (53%)
Total Direct Dollar Impact $2,636$1,582 (60%)
1Failure rates are based on the average time between failures (188 hours) for all Microsoft Windows branded operating systems as computed from numbers reported by bugtoaster.com.

From a short-term perspective, these actions seem to be about using server consolidation to get a handle on costs and related issues like system failure and information security. It is true that, outside of product testing, we'll be going from 14 NT servers to two Unix servers and eliminating 107 desktop computers while dropping half the IT staff and a third of our staffing costs, but those savings are not the point. As discussed in my book The Unix Guide to Defenestration, the Unix transition is not an end in itself but a means to an end. The goal isn't to swap one set of hardware and software for another but to swap one culture for another.

Today, users have desktop machines over which they have no control. They think they do and refer to these machines as "theirs." In reality, these can't be used for anything more productive than playing Solitaire if the servers go down -- and the servers go down far too often. When we're done with the Unix transition the users will have smart displays on their desktop and these too will turn into scrap if the servers go down -- but these servers won't do that. Everything on them is redundant, we will have separate teams running the two machines, and independent power and network connections for them. Sometimes one will fail, sometimes the other will fail, but the probability of both failing in the same interval is close to zero. Direct user control isn't any stronger than it was with Windows servers, but the pretense will be gone and reliability improved.

In the past, IT people were encouraged to mind their own business. User ideas for systems services had to pass through various management filters to be considered. In the future, IT people will focus on working directly with users and new ideas can be evaluated and acted on within hours.

A few systems people will, of course, try to maintain their previous power relationships with users but many of those users will have adopted Linux at home and have developed very good understandings of what can, or cannot, reasonably be done. IT staffers who try this, whether new hires or veterans who managed to hide those attitudes from me, will find it impossible to bluff knowledgeable users. As a result, real power over systems decisions will migrate to the users, counter balancing the centralizing tendency of the Unix and smart display architecture.

Will everything in this example be rosy by March of 2002? No, there will probably be outstanding issues on Web services and executive reporting, but costs will be under control, the relationship between users and systems staff will have changed, and control will be where it belongs -- in user hands.

More Stories By Paul Murphy

Paul Murphy wrote and published 'The Unix Guide to Defenestration'. Murphy is a 20-year veteran of the IT consulting industry.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.