Category Archives: Government Process IQ Trainwrecks

No child left behind (except for those that are)

Leave a reply

Steve Sarsfield shares with us this classic tale of IQ Trainwreck-ry from Atlanta Georgia.

An analysis of student enrollment and transfer data carried out by the Atlanta Journal-Constitution reveals a shocking number of students who appear to be dropping out of school and off the radar in Georgia.  This suggests that the dropout rate may be higher and the graduation rate lower than previously reported.

Last year, school staff marked more than 25,000 students as transferring to other Georgia public schools, but no school reported them as transferring in, the AJC’s analysis of enrollment data shows.

Analysis carried out by the State agency responsible was able to track down some of the missing students. But poor quality information makes any further tracking problematic if not impossible.

That search located 7,100 of the missing transfers in Georgia schools, state education spokesman Dana Tofig wrote in an e-mailed statement. The state does not know where an additional 19,500 went, but believes other coding errors occurred, he wrote. Some are dropouts but others are not, he said.

In a comment which should warm the hearts of Information Quality professionals everywhere, Cathy Henson, a Georgia State education law professor and former state board of education chairwoman says:

“Garbage in, garbage out.  We’re never going to solve our problems unless we have good data to drive our decisions.”

She might be interested in reading more on just that topic in Tom Redman’s book “Data Driven”.

Drop out rates consitute a significant IQ Trainwreck because:

  • Children who should be helped to better education aren’t. (They get left behind)
  • Schools are measured against Federal Standards, including drop out rates, which can affect funding
  • Political and business leaders often rely on these statistics for decision making, publicity,  and campaigning.
  • Companies consider the drop out rate when planning to locate in Georgia or elsewhere as it is an indicator of future skills pools in the area.

The article quotes Bob Wise on the implications of trying to fudge the data that sums up the impact of masking drop outs by miscoding (by accident or design):

“Entering rosy data won’t get you a bed of roses,” Wise said. “In a state like Georgia that is increasingly technologically oriented, it will get you a group of people that won’t be able to function meaningfully in the workforce.”

The article goes on to highlight yet more knockon impacts from the crummy data and poor quality information that the study showed:

  • Federal standard formulae for calculation of dropouts won’t give an accurate figure if there is mis-coding of students as “transfers” from one school to another.
  • A much touted unique student identifier has been found to be less than unique, with students often being given a new identifier in their new school
  • Inconsistencies exist in other data, for example students who were reported “removed for non-attendance” but had  zero absent days recorded against them.

Given the impact on students, the implications for school rankings and funding, the costs of correcting errors, and the scale and extent of problems uncovered, this counts as a classic IQTrainwreck.

The terror of the Terrorist Watch list

Leave a reply

A source who wishes to remain anoynymous sent us this link to a story on Wired.com about the state of the US Government’s Terrorist watch list.

The many and varied problems with the watch list have been covered on this blog before.

However, the reason that this most recent story constitutes an IQTrainwreck is that it seems that, despite undertakings to improve quality, the exact opposite has actually happened given:

  • The growth in the number of entries on the list
  • The failures on the part of the FBI to properly maintain and update information in a timely manner.

According to the report 15% of active terrorism suspects under investigation were not added to the Watch list. 72% of people cleared in closed investigations were not removed.

The report from the US Inspector General said that they “believe that the FBI’s failure to consistently nominate subjects of international and domestic terrorism investigations to the terrorist watchlist could pose a risk to national security.”

That quote sums up why this is an IQTrainwreck.

Continue reading →

Irish State Exam leak being studied.

Leave a reply

A serious complication has emerged in Leaving Certificate exams run by the Irish State each year.. An exam Superindtendent accidentally distributed the wrong paper in one exam centre earlier this week. He put out the exam questions for Paper 2 of the English examination, which wasn’t the subject being examined. The paper was, it seems, only on students’ desks for a few minutes before the error was noticed. However, in this age of twitter, bebo, myspace, facebook and such things, details of the exam questions were soon being discussed in school yards the length and breadth of the country. 

To make matters worse, the Superintendent in question failed to notify the Department of Education until more than 6 hours after the paper was leaked.

As with all things governmental, an investigation is underway. Denials of responsibility have issued from various entities involved.  The Superintendent in question has been dismissed. The exam is being rescheduled, causing disruption to study timetables across the land.

But an examination (no pun intended) of the facts reveals a telling IQ Trainwreck.

One of the factors that determines the quality of information is the quality of information presentation. Indtroducing ambiguity into visual information invites error. Tom Redman, in his book Data Driven, describes the presentation of information as a key step in how information is used and a key part of its complexity. Redman tells us that a number of disciplines need to come together to make even the simplest information and data useful, including:

Presenting data in ways that make it easy for customers to understand and use them. Only in this last step do data and information contribute to internal operations and decisions…

Packaging two sets of highly sensitive information in highly similar packaging which is similar enough that a warning is required makes it hard for customers (Redman uses “customers” to mean the actual consumers of the information – in this case the Superintendent) invites misunderstanding and error.

Yes, the Superintendent could have and should have double checked the paper was the right paper before handing it out, but a key contributing cause was the use of overly similar packaging for both exams.

  • The Superintendent didn’t report a leak of sensitive information in a timely manner

All too often this happens in business. A laptop gets stolen, a memory stick gets mislaid, sensitive information gets left on a train. A key element of the response to this kind of problem is knowing that there is actually a problem, so early reporting to authorities of the leak is imperative. Had the State Examinations Commission had the information in a timely manner perhaps the cost of fixing the gaffe would be less.

  • The cost of remedying the issue is now put at approximately EUR 1 Million

The solution that the Department of Education and State Examinations Commission has come up with is to run a totally new exam paper on Saturday. That means:

  • Extra costs for transport for students to the exams (where State-funded school transport is used)
  • Extra salary costs for Superintendents and their assistants
  • Extra salary costs for school staff who are required to be on-site during exams.
  • The costs of printing a whole new batch of exam papers.

And of course, it being a Saturday:

The SEC is finalising arrangements for a deferred sitting of English papers for a small number of students from the Jewish community after getting legal advice that asking them to sit an exam on their Sabbath, when their religion prohibits it and it is against their conscience, could have been unconstitutional. All other students will be expected to attend, in line with other timetabled exams.

For more on that particular complication, see the Irish Times’ detailed story.

So, why is this an IQTrainwreck?

  1. The similarity in packaging on the exam papers was a key root cause. This is (or should be) a straightforward process of ensuring that all exam subjects and levels are distinctly colour coded and ensuring that packaging is not similar. Issuing a reminder is simply trying to inspect a defect out of the process. Yes, the Superintendent has to carry responsibility as well for not double checking but avoidable similarity should have been avoided (ergo preventing the confusion)
  2. The lack of rigour regarding the reporting of the accidental distribution of the wrong paper is inexcusable. 
  3. The cost impact of the error is extremely significant, particularly given the current state of Irish Government finances. EUR1 Million is a challenging amount to find in your budget at short notice.
  4. The disruptive impact on students during a stressful time can’t be underestimated. 
  5. The further complication presented by Ireland’s multi-culturalism adds further challenges (and potentially costs) for the SEC, the Department, and the students.

(On that note of multiculturalism, one is left wondering if the ISM school in Tripoli, Libya that offers the Irish Leaving Certificate to its students will have received their replacement exam papers yet of if they are even aware of the issue.)

I am not a number – I’m a human being!

Leave a reply

Information Quality professionals (and indeed quality management professionals in general) often recite a mantra that good quality begins at the beginning of a process, that it must be designed in, and that defects need to be fixed as close to the start of the information chain as possible.

A post today on DataQualityPro.com from Dylan Jones highlights the significant truth that lies behind all these statements.

Dylan’s son was born in April. The first thing the State did for him was to slap an identifier on him. The second thing they did (to summarise Dylan’s excellent and forensic post) was to make a mess of linking the local hospital ID to a National patient record.

That error propogated and resulted in the parents of another child over 90 miles away getting an appointment for a medical checkup relating to Dylan’s son. It seems that the efforts made to correct the error Dylan spotted when his son was born haven’t propogated half as fast as the original error.

And that’s the problem. How many other processes and silo’d systems has this error propogated into? How many more times in Dylan’s son’s life will be be confused with another child 90 miles away? What other ‘life-events’ will this error impact? In future, how will he find himself trapped by his number?

Ultimately, Dylan’s son is not a number, he’s a human being.

We recently posted a long Trainwreck on the problems with Google Health due to poor quality information. It is possible that an error like the one affecting Dylan’s son could result in incorrect patient data about Dylan’s son being transferred to this type of electronic patient record. Who would be responsible for the impacts if that information was acted on in haste without Dylan (or Mrs Jones) being there to point out that the information was wrong?

Given that in an Irish hospital in 2003 the medical staff failed to act on an error in an expectant mother’s chart and delivered a baby 39 days prematurely, despite the parent’s insistence that there were errors in the chart, highlights that simple errors in medical records can have significant impacts. That the baby in question died further highlights that these impacts can be catastrophic, which means that the standard of care for quality in medical records needs to be high.

Dylan’s investigations also uncovered some other weaknesses/flaws in patient data quality which are unsettling. We’d suggest you take a look at Dylan’s post for more details on those.

For actual inconvenience and annoyance to Dylan’s family, and for the potential for catastrophic loss or injury, this counts as a definite IQTrainwreck.

Kid from 6th Sense works on Economic Stimulus – he stimulates dead people

3 Replies

The bad joke in the headline aside, this story (which comes to us via Initiate Systems on Twitter, who linked to it from WBALTV in Baltimore USA) reveals a common type of IQ Trainwreck – the “sending things to dead people” problem.

As we know, the US Government has been sending out Stimulus Cheques (or Checks, if you are in the US) to people to help stimulate consumer spending in the US economy. Kind of like a defibrillator for consumer confidence.

Initiate Systems picked up on the story of a cheque that was sent to Mrs Rose Hagner. Her son found it in the mail and was a bit surprised when he saw it. After all, he’s 83 years old and his mother has been dead for over 40 years. Social Security officials give the following explanation:

Of the about 52 million checks that have been mailed out, about 10,000 of those have been sent to people who are deceased.

The agency blames the error on the strict mid-June deadline of mailing out all of the checks, which didn’t leave officials much time to clean up all of their records.

Of course, one might ask why this was such a challenge when the issue raised its head in 2008 as well when a cheque was mailed to a man in Georgia which was made out to a Mr George Croker DECD (an abbreviation for deceased). The story, which was picked up by SmartPros.com at the time (and for the life of us we can’t see how it slipped under our radar), describes the situation as follows:

Richard Hicks, a Fulton County magistrate, says the $600 check arrived in Roswell this week and was made out to George A. Coker DECD, which, of course, stands for “deceased.”

Coker obviously won’t be able to do his bit to spur the consumer economy, which has Hicks puzzled and somewhat miffed.

“There’s a $9 trillion national debt and our government’s giving away money to dead people,” he told The Atlanta Journal-Constitution. “As a taxpayer, it offends the hell out of me.”

The Internal Revenue Service in Atlanta told the newspaper it didn’t know how many other DECD checks have been written nationwide since the 2007 returns are still being processed.

So, the issue has existed since at least 2008 and relates to data being used for a new purpose (sending cheques on a blanket basis). It would seem the solution that is being attempted is to inspect the errors out of the cheque batches before they are sent by the June dead-line. A better solution might be to:

  1. Apply some business rules to the process, for example “If recipient is older than 120 then verify – as the oldest person in the world is currently 115), or parse the name string to determine which social security records end with “DECD” or any other standard variant abbreviation for “deceased”.
  2. Embed these checks (not cheques) into the process for managing the master data set rather than applying them at ‘point of use’.

Building quality into a process, and into the information produced by and consumed by a process, reduces the risk of embarrassing information quality errors. Cleaning and correcting errors or exceptions as a bulk batch process is not as value-adding as actually improving your processes to prevent poor quality information being created or being acted on.

Why is this an IQ Trainwreck?

Well, the volumes of records affected and the actual cost are quite low so one could argue that the information is “close enough for government work”. However, government work tends to get political and a google search on this topic has thrown up a lot of negative political comment from opponents of the stimulus plan.

The volume and actual cost may be low, but the likely impact in terms of PR impact and time that might be required to explain the issue in the media highlights the often overlooked cost and impact of poor quality information – reputation and credibility.

Tax disc mailings… on the Double

Leave a reply

From our ever vigilant sources over at The Register comes this story of duplicated information resulting in confusion and costs to the UK Taxpayer.

It seems that the UK DVLA has issued duplicate tax discs to concientous motorists who renewed their motor tax on-line.

A DVLA spokesman told theRegister: “As a result of an error, a number of customers, who recently purchased tax discs on line or by phone, were issued with duplicate tax discs.

“Once the problem was identified, swift action was taken to rectify it. All customers affected are being sent a letter of apology and the erroneous discs have been cancelled.”

So. Let’s sum this one up…

  1. Poor quality information in a process resulted in the normal cost of the Motor tax process being higher than it should (because of duplicate postage and printing costs for the certificates sent in error).
  2. A further printing and postage expense will be incurred to apologise to motorists for the confusion
  3. Analysis will need to be done to identify all the affected motorists, which will require staff to be diverted from other duties or increased costs due to overtime or external IT contractor spend
  4. People might bin the wrong tax disc and find themselves technically in breach of the law.

This is a simple example of the costs to organizations of poor quality information. A classic IQTrainwreck scenario.

Duff Data dumps 1 million on the dole (social security)… in France.

1 Reply

The Register carries a story this week that clearly shows  the impact of poor quality information on people, particularly in this time of tightening economic conditions when getting a job is hard enough.

It appears that the French Government’s Police Vetting database is not very complete or accurate. According to the French Data Protection authorities (CNIL), this highlights the

serious issues over the provenance of data illustrate all too clearly what happens when the government starts to collect data on its citizens without putting adequate measures in place for updating and accuracy checking.

It would appear that there are errors in 83% of records, with a range of degrees of seriousness. The errors in the database arise as a result of “the simple mechanism of mis-recording actual verifiable data”. In other words, poorly designed processes,  poorly designed data creation/maintenance processes, poorly trained staff, and a lack of information quality control contribute to the errors.

But what  of the cost to the French economy? Well, every person who has been blacklisted in error by this system is potentially drawing social security payments. On top of that they are not paying taxes into the French economy.

If, in a month, they are drawing €100 in social insurance instead of paying €100 in taxes, the net loss to the French economy is €200 per month.  €200 x 1Million =€200 million per month, or €2,400,000,000 per year.

So, on the basis of a very rough guesstimate, the value to the French economy of fixing these errors is €2.4billion per year. Is that a business case?

Parents of dead children asked to choose school for them

1 Reply

According to BBC News Gloucestershire County Council has sent out letters to the parents of a number of children who had died asking them to choose a school as they approach school starting age!

This has resulted in a good deal of pain and grief for the parents and a significant embarrassment to the council and is one more strand in the perception that government agencies (both central and local) have a very bad record in in Information Management. The problem seems to have arisen because of poorly defined inter-agency information requests (i.e. the council seems to have asked the local NHS Trust for details of children born between certain dates and not specified that they should be still alive) and then poor checking of results (at least that’s what the council is blaming it on).

We all know that one can’t add quality so this PR disaster shows the importance of getting the spec right. It also raises questions: “How do the Council find out about children who have moved into the the area after being born within the purview of another NHS Trust?” 

Trusted Electoral Information

7 Replies

Introduction

Warning – this is a long and detailed examination of a complicated trainwreck

[Update] The IAIDQ has issued a press release on this topic…Election Throws A Spotlight On Poor Data Quality. [/update]

In every democracy citizens must be able to trust that the State will not impede their right to vote through any act or omission on the part of the State or its agents. Regular visitors to the iqtrainwrecks.info blog will know that Ireland has it’s fair share of problems with its electoral register. Of course, that isn’t news.

However, the Washington Post has reported last weekend (18th October) that the US elections are being plagued by similar issues. The New York Times covers the same ground in this story from 9th October. With a slightly important vote coming up on the 4th of November, that is news

In a saga that has found its way to the US Supreme Court (in at least one case so far), voters are being forced to re-establish their eligibility to vote before the election on November 4th. As the Post points out, “many voters may not know that their names have been flagged” which could “cause added confusion on Election Day”.

So what is going on (apart from the lawyers getting richer of the inevitable law suits and voters finding themselves reduced to just “Rs” as they lose their Vote)? Where is the trust being lost? Why is this an IQ Trainwreck?

A Change of Process and a Migration of Data

Under the Help America Vote Act, responsibility for the management of electoral registers was moved from locally managed (i.e. county level) to state administered. This has been trumpeted as a more efficient and accurate way to manage the accuracy of electoral lists. After all, the states also have the driver licensing data, social welfare data and other data sources to use to validate that a voter is a voter and not a gold fish.

However, where discrepancies arise between the information on the voter registration and other official records, the voter registration is rejected. And as anyone who has dealt with ‘officialdom’ can testify to, very often those errors are outside the control of the ‘data subject’ (in this case the voter). The legislation requires election officials to use the state databases first, with recourse to the Federal databases (such as social security) supposedly reserved as a ‘last resort’ because ,according the the New York Times, “using the federal databases is less reliable than the state lists and is more likely to incorrectly flag applications as invalid”.

Of course, for a comment on the accuracy of state databases I’ve found this story on The Risks Digest which seems to sum things up (however, as a caveat I’ll point out that the story is 10 years old, but my experience is that when crappy data gets into a system it’s hard to get it out). In the linked-to story, the author (living in the US) tells of her experience with her drivers license which insisted on merging her first initial and middle name (the format she prefers to use) to create a new non-name that didn’t match her other details. That error then propagated onto her tax information and appeared on a refund cheque she received.

In short, it would seem she might have a problem voting (if her drivers license and tax records haven’t been corrected since).

Accuracy of Master Data, and consistency of Master Data

The anecdote above highlights the need for accuracy in the master data that the voter lists are being validated against. For example, the Washington Post article cites the example of Wisconsin, which flags voters data discrepancies “as small as a middle initial or a typo in a birth date”.

I personally don’t use the apostrophe in my surname. I’m O Brien, not O’Brien. Also, you can spell my first name over a dozen different ways (not counting outright errors). A common alternate spelling is Darragh, as opposed to Daragh. It looks like that in Wisconsin I’d have high odds of joining the four members of their 6-strong state elections board who all failed validation due to mismatches on data.

In Alabama, there is a constitutional ban on people convicted of felony crimes of “moral turpitude” voting. The Governor’s Office has issued one masterlist of 480 offences, which included “disrupting a funeral” as a felony. The Courts Administrator and Attorney General issued a second list of more violent crimes to be used in the voter validation process. Unfortunately, it seems that the Governor’s list was used until very recently instead of the more ‘lenient’ list provided by the Courts Administrator.

Combine this with problems with the accuracy of other master data, such as lists of people who were convicted of the aforementioned felonies and there is a recipe for disenfranchisement. Which is exactly what has happened to a former governor (a Republican at that) called Guy Hunt.

In 1993 Mr Hunt had been convicted of a felony related to ethics violations He received a pardon in 1998. In 2008 his name was included on a “monthly felons check” sent to a county Registrar. Mr Hunt’s name shouldn’t have been on the list.

According to the Washington Post article, Mr Hunt isn’t the only person who was included on the felon list. 40% of the names on the list seen by the Washington Post had only committed misdemeanors. In short, the information was woefully inaccurate.

But it is being used to de-register voters and deprive them of their right tohave their say on the 4th November.

The Washington Post also cites cases where US citizens have been flagged as non-citizens (and therefore not entitled to vote) due to problems with social security numbers. Apparently some election officials have found the social security systems to be “not 100% accurate”. But this is the reason why they are supposed only to be used when the state systems on their own are insufficient to verify the voter. That’s the law（apparently).

Continue reading →