using Excel caused Covid-19 results to be lost

[Breelander]

You just cannot make this stuff up....

Excel: Why using Microsoft's tool caused Covid-19 results to be lost

... Public Health England (PHE) was to blame...The problem is that PHE's own developers picked an old file format to do this - known as XLS.

As a consequence, each template could handle only about 65,000 rows of data rather than the one million-plus rows that Excel is actually capable of. And since each test result created several rows of data, in practice it meant that each template was limited to about 1,400 cases. When that total was reached, further cases were simply left off.

For a bit of context, Excel's XLS file format dates back to 1987. It was superseded by XLSX in 2007. Had this been used, it would have handled 16 times the number of cases.

https://www.bbc.co.uk/news/technology-54423988

[servodude]

As a consequence, each template could handle only about 65,000 rows of data rather than the one million-plus rows that Excel is actually capable of. And since each test result created several rows of data, in practice it meant that each template was limited to about 1,400 cases. When that total was reached, further cases were simply left off.

For a bit of context, Excel's XLS file format dates back to 1987. It was superseded by XLSX in 2007. Had this been used, it would have handled 16 times the number of cases.

Why would anyone try to use a spreadsheet for this
- there are a myriad of better ways to store info of this type ..... just as there was in 1987!

Well at least they've sorted it out now

To handle the problem, PHE is now breaking down the test result data into smaller batches to create a larger number of Excel templates. That should ensure none hit their cap.

- aw FFS!

- sd

[UncleEbenezer]

Why would anyone try to use a spreadsheet for this
- there are a myriad of better ways to store info of this type ..... just as there was in 1987!

That comment brings back my days (in the mercifully distant past) of working for UK companies in IT. Must be even more so with such a centralised Soviet approach, headed by a party-political appointee whose career inspires confidence only in the certainty of failure.

Top management play politics. Middle-management play buzzwords. The grunts keep their heads down, and anyone who points to the Emperor's sartorial eccentricity has their life made a misery. Anyone competent to elaborate on your comment desperately looks for a better job,

Well at least they've sorted it out now

To handle the problem, PHE is now breaking down the test result data into smaller batches to create a larger number of Excel templates. That should ensure none hit their cap.

I've got some spare duct tape, if it helps ...

[servodude]

That comment brings back my days (in the mercifully distant past) of working for UK companies in IT. Must be even more so with such a centralised Soviet approach, headed by a party-political appointee whose career inspires confidence only in the certainty of failure.

To be fair even back when I was pulling data from places for the Centralised Soviet Council of Strathclyde they were using a real database

Write stuff in in Foxpro on a Win3 machine
- deploy it in Recital on the Vax

but then, at the end of the cycle, copy and paste results it in to Excel to send to the Scottish Office

- sd

[Itsallaguess]

There's an interesting article here about historic Spreadsheet data-issues -

European Spreadsheet Risks Interest Group -

Public reports of spreadsheet errors have been sought out on behalf of EuSpRIG by Patrick O’Beirne of Systems Modelling for many years. There are very many reports of spreadsheet related errors and they seem to appear in the global media at a fairly consistent rate.

These stories illustrate common problems that occur with the uncontrolled use of spreadsheets. In many cases, we identify the area of risk involved and then say how we think the problem might have been avoided.

http://www.eusprig.org/horror-stories.htm

It seems to me that exposure of poor data-governance is often as much of a problem as the processes and tools that are used, but it's shocking to hear of these issues affecting some of these really important areas over the years.

Digitising and automating things can of course deliver tremendous results, but it often feels like the lack of necessary governance is where the really poor issues start to develop...

Cheers,

Itsallaguess

[AleisterCrowley]

viewtopic.php?f=71&t=25013&p=336463&hilit=excel#p336463

Anther example - as I said -'To err is human, to really foul things up requires a spreadsheet'

I use Excel for 99% of my data analysis work, usually with reports and files dragged from other systems
The thing is, everyone here has got Excel and most people are familiar with it, but I reckon only 20% are competent users as there is insufficient formal training.

[Lootman]

I am far from being an IT guy but even I know this is nuts. Back in the 1980s and 1990s in the City we had a team of people whose job was to model data and design proper databases. Using Excel is just amateur hour. It's fine for keeping track of my capital gains. For tracking 60 million people, not so much.

But then I hear the NHS and other government departments still use hierarchical databases from the 1960s, and have programmes written in Cobol. My local council still uses those so-called "dumb terminals", presumably attached to some creaking IBM mainframe from the 1970s. I guess their support staff are getting older and older.

[Infrasonic]

I am far from being an IT guy but even I know this is nuts. Back in the 1980s and 1990s in the City we had a team of people whose job was to model data and design proper databases. Using Excel is just amateur hour. It's fine for keeping track of my capital gains. For tracking 60 million people, not so much.

But then I hear the NHS and other government departments still use hierarchical databases from the 1960s, and have programmes written in Cobol. My local council still uses those so-called "dumb terminals", presumably attached to some creaking IBM mainframe from the 1970s. I guess their support staff are getting older and older.

I've just finished reading Edward Snowden's book Permanent Record (highly recommended) and some of the antiquated IT equipment and procedures he reports on that were in place at the NSA/CIA was a bit of an eye opener.
One guy waiting it out until his retirement appeared to do nothing until Snowden worked out he was the only one who knew how to replace the ancient tape backups every night, (no mention of any test restores to see if they actually worked...).

[modellingman]

But then I hear the NHS and other government departments still use hierarchical databases from the 1960s, and have programmes written in Cobol. My local council still uses those so-called "dumb terminals", presumably attached to some creaking IBM mainframe from the 1970s. I guess their support staff are getting older and older.

The banks are probably the biggest culprits here. Co-op, Lloyds, NatWest all have form here.

[modellingman]

Most amusing comment from El Reg on this fiasco:

Dido: disaster in, disaster out.

https://www.theregister.com/2020/10/05/ ... act_error/

[AleisterCrowley]

But then I hear the NHS and other government departments still use hierarchical databases from the 1960s, and have programmes written in Cobol. My local council still uses those so-called "dumb terminals", presumably attached to some creaking IBM mainframe from the 1970s. I guess their support staff are getting older and older.

The banks are probably the biggest culprits here. Co-op, Lloyds, NatWest all have form here.

I always assumed the public sector would be worst - no real incentives to modernise/perform well

[Infrasonic]

But then I hear the NHS and other government departments still use hierarchical databases from the 1960s, and have programmes written in Cobol. My local council still uses those so-called "dumb terminals", presumably attached to some creaking IBM mainframe from the 1970s. I guess their support staff are getting older and older.

The banks are probably the biggest culprits here. Co-op, Lloyds, NatWest all have form here.

I always assumed the public sector would be worst - no real incentives to modernise/perform well

I had an IT mate who was on pager call out years ago for a well known high street "ethical" bank, some of the stories he told me about how chewing gum and string based their internet banking set up was put me off ever using them. I'm sure many of the issues existed elsewhere with the other banks.
Fraud within the SWIFT interbank system runs into hundreds of millions worldwide (that they'll admit to in public). I suspect the real numbers are more like billions.

[AleisterCrowley]

Smaller building societies having steam powered IT wouldn't surprise me, but I can remember opening a new type of account at the Nationwide some years ago and the advisor had to write down all my details from System A then type them into System B, there being no central customer identity

[UncleEbenezer]

Most amusing comment from El Reg on this fiasco:

Dido: disaster in, disaster out.

https://www.theregister.com/2020/10/05/ ... act_error/

Except you seem to have misquoted. The comment there is properly in the 'recursive' tradition of GNU, PINE, et al.

Dido in Disaster Out.

(alas, the missing comma is original too).

[Lanark]

I am far from being an IT guy but even I know this is nuts. Back in the 1980s and 1990s in the City we had a team of people whose job was to model data and design proper databases. Using Excel is just amateur hour. It's fine for keeping track of my capital gains. For tracking 60 million people, not so much.

But then I hear the NHS and other government departments still use hierarchical databases from the 1960s, and have programmes written in Cobol. My local council still uses those so-called "dumb terminals", presumably attached to some creaking IBM mainframe from the 1970s. I guess their support staff are getting older and older.

Nobody is still using systems that old.

There are councils still using IBM mainframes, but they are modern machines with updated OS's and current support contracts.
It is easy to look at the IT systems used by other organisations and think "I could do that better", but the requirements often turn out to be more complex. Think: years of ever changing govt. regulations which apply in different ways to different people.

I once worked on a govt system which has a logic tree of about 1800 different parameters applied to each record, all due to ancient long forgotten regulations but the historical record had to be kept. Still it keeps us all in jobs I suppose.

In this particular case they were importing CSV files into the latest version of Excel, but someone - probably due to a lack of IT training had set things up to use the old .XLS file format which limits the number of rows.

It's a mystery to me why IT training seems to have gone so out of fashion.

[johnhemming]

In this particular case they were importing CSV files into the latest version of Excel, but someone - probably due to a lack of IT training had set things up to use the old .XLS file format which limits the number of rows.

When the system was originally developed it probably was within the reasonably expected requirement.

[nmdhqbc]

stand up maths
https://www.youtube.com/watch?v=zUp8pkoeMss