Two reports suggest this is what happens when government agencies hollow out
Australia's census all-but failed due to a combination of poor design, bad operational decisions, human error and numerous lazy and/or bad decisions that could have been avoided had warnings about corporate culture been heeded and Australian government agencies properly educated about what it takes to deliver digital services.That's The Register's summary of two reports into the contentious events of August 9th, when Australia's online census went down after a suspected denial of service attack saw a router rebooted, but fail to restart because IBM had never tested what would happen if it turned it on and off again. IBM has claimed it would never have had to touch the power button if ISPs it hired did their jobs properly.
Enough back story for now: to the reports, namely the Senate Economics References Committee's 2016 Census: issues of trust and the Review of the Events Surrounding the 2016 eCensus (PDF) by Alastair MacGibbon, Special Adviser to the Prime Minister on Cyber Security.
The latter report is not kind to any of the participants in the census, as MacGibbon observes that “One of the government’s most respected agencies – the Australian Bureau of Statistics (the ABS) – working in collaboration with one of the technical world’s most experienced companies – IBM – couldn’t handle a predictable problem.” MacGibbon says “ABS and IBM emphasised some areas of security – the confidentiality and integrity of data – while underinvesting in the availability of the system.”
He also notes that the plans devised by the ABS and the IBM were signed off by the Australian Signals Directorate (ASD), a signals intelligence and security advisory agency that he says “has “outstanding expertise for supporting agencies, but not the capacity to service the clear need across government.”
MacGibbon also mentions the Australian Public Service Commission (APSC) Capability Review of 2013 that found the ABS to be “insular, inward looking, reactive” and recommended an overhaul of its culture. That overhaul largely hasn't happened, leaving the ABS “an exemplar of established government practice: ticking the boxes, but not appreciating the challenges change presents.” Matters weren't helped by the fact the agency lacked a Chief Statistician [effectively CEO – Ed] during much of 2014 when census preparation commenced.
The Senate Committee's report also notes the capability review and offered the following observation:
“The ABS' funding has been eroded over a number of years while the demands and expectations placed on the organisation have increased. Accurate data is critical for the provision of public services and for businesses making investment decisions in Australia. The ABS is a world leading statistical agency, and to remain such it requires funding to maintain current capacities, meet new demands, and develop the skills necessary to provide quality outputs.Again, the ABS' decision-making is called into serious question and resource levels are raised as one reason for its decisions. Here's the Committee's view at section 6.84 of its review:
“Questions regarding the validity of the ABS' actions should be focused on the years and months before the 2016 census when the decisions were made that would manifest themselves on 9 August 2016. The confirmation that the census would proceed, the delayed development of an eCensus solution, the use of a limited tender and the erosion of internal capacity to adequately oversee the development of the eCensus are all serious concerns that may contributed to the events of 9 August 2016.”MacGibbon's report suggests ABS culture led IBM to be considered a safe option, but suggests that sentiment left it “locked itself in with a trusted partner” and “denied itself the opportunity to leverage conditions and capabilities that were changing rapidly over the decade.”
That worries MacGibbon as much as anything else: his report says the failure of the census shows that beyond a couple of dedicated agencies, Australia's government just doesn't know what it takes to run digital services.
Dud DDoS defences
The organisational issues both reports identify led to the adoption of what proved to be a dud DDoS defence.The Senate Committee report offers the observation that “It goes without saying that the eCensus website should have had the capacity to withstand what was a relatively minor attack.”
“Further, the appropriateness of Island Australia must also be questioned given that some components of the eCensus—such as password resets—required access to international servers.”
MacGibbon criticises IBM for its approach to implementing Island Australia, which saw it test the DDoS protection only once the census form was live and then only for ten minutes. Those tests did not consider the impact the “Island Australia” plan to block traffic from non-Australian IP addresses would have on other internet service providers (ISPs). IBM is also felt not to have issued proper instructions to one of its contracted ISPs. That ISP didn't help matters by failing to configure its service for IBM's data centre.
Even if all the DDoS protection plans had worked, MacGibbon finds it the “Island Australia” plan is not a widely-accepted DDoS defence tactic. He also points out that geo-blocking had the potential to harm the census. Here's his observation on the matter:
Parts of the eCensus system – including the Softlayer mail relay used for password resets – are located offshore, and hence the eCensus system would not function as intended while ‘Island Australia’ was active. Additionally the access point to the internet for a number of Australians – including Vodafone customers located in NSW or those using VPNs – is via an international location. Those affected would have been unable to use the eCensus system while ‘Island Australia’ was active without explanation as to why.To The Register's mind the extracts above are a smoking gun for all involved: IBM came up with a dud defence mechanism and it was signed off by the ABS and ASD.
MacGibbon doesn't hold back: he says IBM had a contractual obligation to provide DDoS protection and failed.
Questions of privacy
But there's plenty of others who need to have a long hard look at themselves in the mirror. Both reports conclude that cross-agency and cross-ministry plans to prepare for, and respond to, foreseeable incidents were poorly thought out and executed. Escalation plans were either poor or non-existent.The ABS comes in for plenty of criticism for just not engaging meaningfully with the community amid escalating concerns about the decision to retain name and address data.
But perhaps the biggest question of privacy to be answered is why IBM has been able to settle this matter confidentially. Australia's government, citing commercial-in-confidence provisions in IBM's contract, has revealed that a settlement has taken place but we'll never know what happened.
Australian prime minister Malcolm Turnbull has given a radio interview in which he said the IBM settlement covers the $30m it cost the ABS to get the census site up and running. But a single vague hint tells us nothing about just what went wrong and what agencies considering IBM need to ask of the company or themselves.
There also appears to be no sanction for IBM beyond the settlement, yet MacGibbon's report says “between 1 January 2013 and 19 August 2016 IBM was awarded 777 contracts across the Commonwealth Government with a total value of $1.55 billion.”
As The Register recently revealed, a recent $4m tender for a new supercomputer seemingly goes out of its way to offer IBM a chance to re-platform a significant code base. Without transparency of IBM's role in #censusfail, it's harder to know if sending more work Big Blue's way is a sound idea.
That Australia has not been able to shine a light on the settlement perhaps just re-enforces the two reports' observations that under-resourced agencies struggle to do things well. Other nations practicing austerity measures, or making rapid moves to digital service delivery, may wish to take note of Australia's experience
Source: http://www.theregister.co.uk/2016/11/25/ibm_clearly_failed_ddos_protection_for_australias_censusfail/