Compressing Data Files

[SteamyTea]

A while back, and on another topic, @saveasteading commented about making data files smaller.

I did a brief description about that happens, but thought I would spend a few minutes showing some real examples.

 

The data file I used was some electrical grid data from 2012 to 2025, averaged, minimumed and maximumed at the half hour level.

This created a large spreadsheet (I know, should be using a database) that was 227,954 rows by 75 columns, with 17,089,587 data points in it.

 

I saved the file in 3 different formats:

Comma Separated Value, .csv, a basic text file that is good for importing text based data.

Open Document Spreadsheet, .ods, this is the format that LibreOffice Calc uses by default.

Microsoft Excel Spreadsheet, .xlsx, as this is used by a lot of people, and probably by far the best spreadsheet package still.

 

I then separately compressed all three files, then compressed and encrypted to see what the difference in size was.

Finally I compressed and encrypted all three files into one .zip file.

 

As expected the .csv file was the smallest of the uncompressed files, same when compressed, with encrypting making no difference in size.

Compressing and encrypting all three files into one made no difference to overall size.

 

So what have I learnt.

No need to compress files individually, better of doing them all into one file, and encrypting makes no difference to size, so may as well encrypt.

 

Results below:

 

 

Edited April 20 by SteamyTea