Transport for NSW scrambles to patch servers missing fixes released in 2007

But IBM Australia has only a ‘skeleton crew’ on duty, missed deadlines, will move people from other projects for fix

 

Around a third of servers at Transport for New South Wales, the public transport department in Australia’s largest most populous state, need security patches, some dating back to 2007. But IBM, which provides IT services to the agency, doesn’t have enough people dedicated to the the job to get it done in the planned timeframe or in a manner that will let the agency operate as it desires.
The Register understands that Transport for New South Wales (TfNSW) runs a mixed fleet of AIX, Solaris, Red Hat Linux and Windows servers, all of which need patching. It is unclear what applications run on the un-patched servers, or their sensitivity, but TfNSW has mobilised an effort to quickly catch up on its patching.
IBM, however, has found itself with just a “skeleton crew” at the agency due to personal circumstances and staff being moved to other, higher-priority jobs. The company has therefore not been able to implement all of TfNSW’s desired changes or keep up with its client's requests, leaving many servers without patches. Some of the fixes were released as far back as 2007. We understand IBM is not responsible for the tardy patching effort.
Sources tell The Register IBM has called for teams working at other clients to lend staff to sort things out at TfNSW, as while offshore labour will be involved it can only do so much when on-premises mission-critical servers require reboots. The request for help is an offer other teams dare not refuse.
IBM’s therefore tried to find specialists in all the operating systems mentioned above, preferably with patch-preparation expertise, for a few weeks work. Whoever is recruited is in for a torrid time: we’re told midnight shifts and weekend work will be required as change windows are scheduled beyond business hours.
An IBM spokesperson told The Register such shout-outs for assistance are not unusual. "IBM shifts resources on a continuous basis, based on clients' project requirements and the need for skills. This is common with any services delivery organisations operating a shared services model."
The problems at TfNSW seem to have come about in part due to Meltdown patches throwing other plans out of kilter. The resulting mess has created a requirement for change windows so long and so numerous that TfNSW has balked at the effort required, further complicating patching plans.
The Register understands IBM can't hire new people fast enough to address the problem, a state of affairs that is perhaps the result of IBM's made numerous rounds of redundancies and decision to stop hiring contractors. IBM has described such changes as ensuring its business is an appropriate size.
But in this case it appears IBM Australia has so little fat, its TfNSW team can't cover a handful of staff becoming unavailable. And with new contract hires forbidden, it can't make a quick fix.
Ironically, sources tell The Register that one of the few exceptions to the contractor ban is hires made by offshore teams seeking a better liaison in the nations where IBM clients reside.
Source