Author Archives: Daragh O Brien

About Daragh O Brien

Daragh O Brien is the Managing Director of Castlebridge Associates. This site has been one of his side projects for a decade. It needs some love and attention...

Green Card, Red Faces

9 Replies

The United States Government is being sued in a massive class-action suit representing Green Card applicants from over 30 countries which alleges that the United States unfairly denied 22,000 people a Green Card due to a computer blunder.

This story is reported in the Irish Times and the Wall Street Journal.

It is not in the remit of this blog to debate the merits of awarding working visas on the basis of a random lottery, but this is precisely what the Green Card system is, offering places to 50,000 people each year based on a random selection of applications submitted over a 30 day period. According to the WSJ:

In early May, the State Department notified 22,000 people they were chosen. But soon after, it informed them the electronic draw would have to be held again because a computer glitch caused 90% of the winners to be selected from the first two days of applications instead of the entire 30-day registration period.

Many of these 22,000 people are qualified workers who had jobs lined up contingent on their getting the Green Card. The WSJ cites the example of a French neurospyschology PhD holder (who earned her PhD in the US) who had a job offer contingent on her green card.

The root causes that contributed to this problem are:

  1. that the random sampling process did not pull records from the entire 30 day period, with the sampling weighted to the first two days of applicants, with 90% of the “winners” being drawn from the first two days.
  2. There was no review of the sampling process and outputs before the notifications were sent to the applicants and published by the State Department. It appears there was a time lag in the error being identified and the decision being taken to scrap the May Visa Lottery draw.

The first error looks like a possible case of a poorly designed sampling strategy in the software. The regulations governing the lottery draw require that there be a “fair and random sampling” of applicants. As 90% of the applicants were drawn from the first two days, the implication is that the draw was not fair enough or was not random enough. At the risk of sounding a little clinical however, fair and random do not always go hand in hand when it comes to statistical sampling.

If the sampling strategy was to pool all the applications into a single population (N) and then randomly pull 50,000 applicants (sample size n), then all applicants had a statistically equal chance of being selected. The fact that the sampling pulled records from the same date range is an interesting correlation or co-incidence. Indeed, the date of application would be irrelevant to the sampling extraction as everyone would be in one single population. Of course, that depends to a degree on the design of the software that created the underlying data set (were identifiers assigned randomly or sequentially before the selection/sampling process began etc.)

This is more or less how your local State or National lottery works… there is a defined sample of balls pulled randomly which create an identifier which is associated with a ticket you have bought (i.e. the numbers you have picked). You then have a certain statistical chance of a) having your identifier pulled and b) being the only person with that identifier in that draw (or else you have to share the winnings).

If the sampling strategy was to pull a random sample of 1666.6667 records from each of the 30 days that is a different approach. Each person on each day of application has the same chance as anyone else who applied that day, with each day having an equal chance at the same number of applicants being selected. Of course it raises the question of what do you do with the rounding difference you are carrying through the 30 days (equating to 20 people) in order to still be fair and random (a mini-lottery perhaps).

Which raises the question: if the approach was the “random in a given day” sampling strategy why was the software not tested before the draw to ensure that it was working correctly?

In relation to the time lag between publication of the results and the identification of the error, this suggests a broken or missing control process in the validation of the sampling to ensure that it conforms to the expected statistical model. Again, in such a critical process it would not be unreasonable to have extensive checks but the checking should be done BEFORE the results are published.

Given the basis of the Class Action suit, expect to see some statistical debate in the evidence being put forward on both sides.

Unhealthy Healthcare data

2 Replies

For a change (?) it is nice (?) to see stories about healthcare IQ Trainwrecks that don’t necessarily involve loss of life, injury, tears, or trauma.

Today’s Irish Examiner newspaper carries a story of the financial impacts of poor quality data in healthcare administration. At a time when the budgets for delivery of healthcare in Ireland are under increasing pressure due to the terms of the EU/IMF bailout of Ireland, it is essential that the processes for processing payments operate efficiently. It seems they do not:

  1. Staff continued to be paid pensions where they retired from one role and then re-entered the Health Service in a different role (HSE South)
  2. Absence of Controls meant staff who were on sick leave with pension entitlements being paid continued to be paid when they returned to work (HSE South)
  3. Pensions were calculated off incorrect bases for staff who were on secondment/shared with other agencies (HSE South)
  4. Inaccurate data about the ages of dependents resulted in overpayments of death in service benefits (HSE South).
  5. “Inappropriate” filing systems were resulting in “needlessly incurring wastage of scarce resources” (HSE Dublin/Mid Lenister)

 

Poor quality information costs between 10% and 35% of turnover in the average organisation. So the HSE may not be too bad. But the failure of controls and processes resulting in poor quality data leading to financial impacts is all too familiar.

Electoral finger flub causes kerfuffle

Leave a reply

Via the twitters and google comes this story from Oh Canada about the unforeseen confluence of an election, the adoption of new technology (QR codes), and a careless fingerflub that has resulted in a bit of embarassment for a Liberal party candidate.

This is the comedic counterpoint to our story last month of the finger flub that resulted in death and lawyers.

It seems that staffers working for candidate Justin Trudeau fat fingered the creation of the QR code that is being used on his posters. Instead of the code containing a URL for the Liberal Party they hit the “U” key instead, creating a URL that sent people to a “lifestyle” site that promoted the use of lubricants in sexual activity.

Sadly Luberal.ca has been taken down at the request of the party, and it seems that they may be in discussion to buy the domain name from the current owner. The candidate has tweeted about the issue on his twitter feed, and staff have been dispatched out to replace the offending QR code with a corrected version.

All of which adds up to cost and resource headaches for an election candidate who probably had other things planned for his staff to be doing at this stage in the campaign.

Of course, we remain slightly concerned that, given that it is April 1st this may be too good a story to be true. But in that case take it as a parable of what could happen, not necessarily a report of what did!

Gas by-products give a pain in the gut

1 Reply

Courtesy of Lwanga Yonke comes this great story about how the choice of unit of measure for reporting, particularly for regulatory reporting or Corporate Social Responsibility reports can be very important.

The natural gas industry’s claim that it is making great strides in reducing the polluted wastewater it discharges to rivers is proving difficult to assess because of inconsistent reporting and a big data entry error in the system for tracking contaminated fluids.

The issue:

Back in February the Natural Gas industry in the US released statistics which appeared to show that they had managed to recycle at least 65% of the toxic waste brine that is a by-product of natural gas production. Unfortunately they had their data input a little bit askew, thanks to one company who had reported data back to the State of Pennsylvania using the wrong unit of measure – confusing barrels with gallons.

For those of us who aren’t into the minutiae of natural gas extraction, the Wall Street Journal helpfully points out that there are 42 gallons in a barrel. So, by reporting 5.2 million barrels of wastewater recycled instead of the 5.2 million gallons that were actually recycled, the helpful data entry error overstated the recycling success by a factor of 42.

Which is, co-incidentally, the answer to Life the Universe and Everything.

According to the Wall Street Journal, it may be impossible to accurately identify the rate of waste water recycling in the natural gas industry in the US.

Not counting Seneca’s bad numbers — and assuming that the rest of the state’s data is accurate — drillers reported that they generated about 5.4 million barrels of wastewater in the second half of 2010. Of that, DEP lists about 2.8 million barrels going to treatment plants that discharge into rivers and streams, about 460,000 barrels being sent to underground disposal wells, and about 2 million barrels being recycled or treated at plants with no river discharge.

That would suggest a recycling rate of around 38 percent, a number that stands in stark contrast to the 90 percent recycling rate claimed by some industry representatives. But Kathryn Klaber, president of the Marcellus Shale Coalition, an industry group, stood by the 90 percent figure this week after it was questioned by The Associated Press, The New York Times and other news organizations.

The WSJ article goes on to point out that there is a lack of clarity about what should actually be reported as recycled waste water and issues with the tracking of and reporting of discharges of waste water from gas extraction.

At least one company, Range Resources of Fort Worth, Texas, said it hadn’t been reporting much of its recycled wastewater at all, because it believed the DEP’s tracking system only covered water that the company sent out for treatment or disposal, not fluids it reused on the spot.

Another company that had boasted of a near 100 percent recycling rate, Cabot Oil & Gas, also Houston-based, told The AP that the figure only included fluids that gush from a well once it is opened for production by a process known as hydraulic fracturing. Company spokesman George Stark said it didn’t include different types of wastewater unrelated to fracturing, like groundwater or rainwater contaminated during the drilling process by chemically tainted drilling muds.

So, a finger flub on data entry, combined with lack of agreement on meaning and usage of data in the industry, and gaps in regulation and enforcement of standards means that there is, as of now, no definitive right answer to the question “how much waste water is recycled from gas production in Pennsylvania?”.

What does your gut tell you?

 

Calculation errors casts doubt on TSA Backscatter safety

Leave a reply

It is reported in the past week on Wired.com and CNN that the TSA in the United States is to conduct extensive radiation safety tests on their recently introduced backscatter full body scanners (affectionately known as the “nudie scanner” in some quarters).

An internal review of the previous safety testing which had been done on the devices revealed a litany of

  • calculation errors,
  • missing data and
  • other discrepancies on paperwork

In short, Information Quality problems. A TSA spokesperson described the issues to CNN as being “record keeping errors”.

The errors affected approximately 25% of the scanners which are in operation, which Wired.com identifies as being from the same manufacturer, and included errors in the calculation of radiation exposure that occurs when passing through the machine. The calculations were out by a factor of 10.

Wired.com interviewed a TSA spokesperson and they provided the following information:

Rapiscan technicians in the field are required to test radiation levels 10 times in a row, and divide by 10 to produce an average radiation measurement. Often, the testers failed to divide results by 10.

For their part, the manufacturer is redesigning the form used by technicians conducting tests to avoid the error in the future. Also, it appears from documentation linked to from the Wired.com story that the manufacturer spotted the risk of calculation error in December 2010.

Here at IQTrainwrecks.com we are not nuclear scientists or physicists or medical doctors (at least not at the moment) so we can’t comment on whether the factor of 10 error in the calculations is a matter for any real health concern.

But the potential health impacts of radiation exposure are often a source of concern for people. Given the public disquiet in the US and elsewhere about the privacy implications and other issues surrounding this technology any errors which cast doubt on the veracity and trustworthiness of the technology, its governance and management, and the data on which decisions to use it are based will create headlines and headaches.

 

The Wrong Arm of the (f)Law

Leave a reply
Courtesy of Steve Tuck and Privacy International comes this great story from the UK of how a simple error, if left uncorrected, can result in significantly unwelcome outcomes. It is also a cautionary tale for those of us who might think that flagging a record as being “incorrect” or inaccurate might solve the problem… such flags are only as good as the policing that surrounds them.
Matthew Jillard lives on Repton Road in a suburb of Birmingham. In the past 18 months he has been raided over 40 times by the police. During Christmas week he was raided no fewer than 5 times, with some “visits” taking place at 3am and 5am, disturbing him, his family, his family’s guests, his neighbours, his neighbour’s guests….
According to Mr Jillard,
9 times out of 10 they are really apologetic.
Which suggests that 1 time out of 10 the visiting police might annoyed at Mr Jillard for living at the wrong address(??)
The root cause: The police are confusing Mr Jillard’s address with a house around the corner on Repton Grove.
(scroll the map to the right to find Repton Grove)
Clancy Wiggum from the Simpsons
Not a spokesman for West Midlands Police

View larger map
Complaints to the police force in question have been met with apologies and assurances that the police have had training on how important it is to get the address right for a search. Some officers have blamed their Sat Nav for leading them astray.
Given the cost to the police of mounting raids, getting it wrong 40 times will be putting a dent in their budget. Also, the costs to the police of putting right any damages done to Mr Jillard’s home due to the incorrect raids (which have included kicking in his door at 3am on Christmas Day) will also be mounting up.
The police have said that “measures” have been taken to prevent Mr Jillard’s home being raided, including putting a marker against his address on the police computer systems. None of these measures appear to have stopped the raids, which come at an average frequency of more than one a fortnight (40 raids in 18 months).
This Trainwreck highlights the impact of apparently simple errors in data:
  1. Mr Jillard’s home is being disturbed without cause on a frequent basis
  2. His neighbours must be increasingly suspicious of him, what with the police calling around more often than the milkman
  3. The police force is incurring costs and wasting man power with a continuing cycle of fruitless raids.
  4. The real target of the raids are now probably aware of the fact that the police are looking for them and will have moved their activities away from Repton Grove.

8 year old orphaned by a fat finger key stroke error

1 Reply

Daragh O Brien has written and presented in the past for the IAIDQ on the topic of how the legal system and information quality management often look at the same issues from a different perspective, ultimately to identify how to address the issues of the cost and risk of poor quality.

This was brought home very starkly this morning in a case from the UK High Court which has opened the possibility of six figure damages being awarded to an 8 year old boy who was orphaned by a data quality error.

A single key stroke error on a computer cost a mother her life from breast cancer and left her eight-year-old son an orphan, the High Court has heard.

Two urgent letters informing the single mother of hospital appointments were sent to the wrong address – because the number of her home was typed as ’16’, instead of ‘1b’.

Read more: http://www.dailymail.co.uk/news/article-1366056/Mistyped-address-leaves-mother-dead-cancer-son-8-orphan.html#ixzz1GfRPOOHJ

In a tragic series of events a young mother discovered a lump on her breast. She was treated in hospital and given the all clear, but continued to be concerned. Her GP arranged further tests for her but she never received the letters due to a simple mis-keying of her address which meant she never received her appointment letters. As her cancer went untreated for a further 12 months by the time she was diagnosed her only treatment option was palliative care. Had she been treated in time, the Court heard, she would have had a 92% chance of survival for another 10 years.

Her doctor admitted liability arising from the failure of the surgery to follow up with the the woman on her tests, which might have uncovered that she hadn’t received the letters.

The Court dismissed an argument by the defence that the woman should have followed up herself, on the grounds that, while they would never know what had been in her mind, she had already been given an “all clear” and that she was likely either trying to get on with her life or may have been scared to return to the doctor.

A key lesson to be learned here is that ensuring accurate information is captured at the beginning of a process is critical. Equally critical is the need for organisations where the data is potentially of life and death importance to ensure that there is follow up where the process appears to have stalled (for example if expected test results are not received back from a hospital).

A simple error in data input, and a failure of or lack of error detection processes, has been found by the UK High Court to be the root cause for the death of a young mother and the orphaning of an 8 year old boy.  This is a SIGNIFICANT legal precedent.

Also, the case raises Data Protection Act compliance issues for the GP practice as sensitive personal data about a (now deceased) patient was sent to the wrong address.

RELATED POST: Daragh O Brien has a related post on his personal blog from 2009 about how Information Quality is getting some interesting legal support in the English legal system.

Smart Grid, Dumb Data

Leave a reply

In September 2010 a massive gas explosion ripped through the San Francisco suburb of San Bruno, not too far from San Francisco International Airport. The explosion was so powerful it was registered as a magnitude 1.1 earthquake.

Subsequent investigations have identified that poor quality data was a contributory factor in the disaster. According to Fresnobee.com

The cause of the deadly rupture has not yet been determined, but the PUC said it is moving ahead with the penalty phase after the National Transportation Safety Board recently determined that PG&E incorrectly described the pipe as seamless when in fact it was seamed and welded, making it weaker than a seamless pipe.

Read more: http://www.fresnobee.com/2011/02/25/2285689/pge-faces-big-fine-over-gas-pipeline.html#

According to the San Francisco Chronicle the problems with PG&E’s data were nothing new, with problems stretching back almost 20 years.

Omissions or data-entry errors made when the system was developed – and left uncorrected – may explain why PG&E was unaware that the 1956-vintage pipeline that exploded in San Bruno on Sept. 9, killing eight people, had been built with a seam, according to records and interviews. Federal investigators have found that the explosion started at a poorly installed weld on the seam.

Continue reading →

So exactly HOW pregnant is he?

5 Replies

From the #dataquality correspondents on Twitter comes this great story of a classic IQ Trainwreck.

Hilton Plettell is pregnant and is expected to deliver in 7 months, according to the NHS. They’ve invited him to a scan to see his bundle of joy.

Yes. We did say HIM and HIS, because Hilton is a 50 year old department store merchandising manager. But that is not the end of the IQ Trainwreck here.

  1. The hospital he was directed to is 162 miles from his home (a long way to travel with the full bladder needed for an ultrasound scan).
  2. A sticker attached to the letter correctly identified Mr Plettell as being Male.

So, 3 errors or inconsistencies in the letter which indicate a Data Quality kerfuffle in the NHS (at least in Norwich).

A spokesperson for the hospital thanked Mr Plettell for raising the issue with them and indicated they were undertaking a Root Cause Analysis to see where their processes and procedures could be improved to prevent this type of obvious error.

We can’t help but wonder if the root cause might be similar to the problem encountered by DataQualityPro.com’s Dylan Jones last year, which we reported here in June 2009.

The story is covered in the Daily Male  Mail, which reproduces a picture of Mr Plettell’s hospital letter (but that image is copyright so we can’t republish it here).

The importance of context

1 Reply

Data is often defined as “Facts about things” and Information is often defined as “Facts about things in a context”.

From Lwanga Yonke (IAIDQ Advisor and one of the visionaries behind the CIQP certification) comes this great example of where, without consistent application of context, it is possible for the Data to give rise to poor quality and misleading information.

Sign showing population, feet above sealevel and year founded with the data totalled

Image linked from "thepocket.com"

What we see in the sign opposite are three distinct contexts:

  1. A count of the population (562)
  2. The height of the town above sealevel (2150)
  3. The year the town was founded (1951)

And of course, when we see a column of figures our instinct (since our earliest school days) is to add them all up… to give us 4663.

Of course, that figure is meaningless as information, and is also poor quality data.

I have personally experienced similar “challenges of context” in tracking back root cause analyses in Regulatory Compliance projects.. the stakeholder pulling the incident reports together didn’t consider context and as such was comparing apples with ostrich eggs (if he’d been comparing apples to oranges at least they’d both have been fruit).

I’d love to hear your stories of Contextual conundrums that have lead to poor quality data and erroneous Information.