Archiving data: storage prices in 2013 and 2020

▼ Back in 2013, I wrote a blog post about archiving. In it, I compared the costs per terabyte (and the weight per terabyte) of several ways to store data for archival purposes. When I read Beyond Time Machine: 5 Archiving over at The Eclectic Light Company blog, I realized that it’s time revisit this topic. This is the list of storage options with the price per gigabyte in euros, back in 2013 and now in 2020:    

	€/TB 2013	€/TB 2020	Sweet spot
DVD±R	45	67
BR-R		45
2.5" USB HDD	60	23	4 - 10 TB
3.5" internal HDD	35	32	4 - 8 TB
USB flash	370	195	128 - 256 GB
SD card	520	320	128 GB
Internal SSD		125	1 TB
USB SSD		170	1 TB

And just to see how ridiculous things can get, if you buy an Apple computer, SSD upgrades can cost as much as € 1000 per terabyte (€ 250 to go from 256 to 512 GB).

The sweet spot is the size where the cost per terabyte is the lowest. If you go smaller or larger, you pay more for the same amount of storage. I got the 2020 prices by looking at bol.com. Obviously, prices vary, sometimes significantly. I chose the lowest prices, except if only a couple of no-brand options were the cheapest. Note that inflation was around 7% between 2013 and 2020.

What’s interesting here:

• DVD±Rs have gone up in price
• Blu-ray recordable is now at about the per-TB price DVD±R used to be
• 3.5" HDDs haven’t really gone down in price per TB
• 2.5" USB HDDs are now about three times cheaper than they used to be
• 2.5" USB HDDs are now a good deal cheaper than 3.5" HDDs
• SSDs are still much more expensive than HDDs

2.5" USB HDDs are by far the cheapest way to archive your data. They’re also convenient: they’re small in size, and unlike 3.5" USB HDDs (which are extinct now, I think) they are bus powered so no issues with power supplies. Yes, copying terabytes worth of data to a USB HDD takes a lot of time. But you can do that overnight. With BD-Rs you need to put in a new one for every 25 to 100 GB.

I can still read most (but not all!) of the DVDs I burned 15 years ago without trouble, but I wouldn’t bet any money that a 15-year-old USB HDD will still work. You really have to keep your archived data on at least two of those and then replace them every three years or so, copying your data from the old one to a new one. (Or, more likely in my case: from my NAS to a new HDD.) But that’s a lot more doable than duplicating BD-Rs. Also, any computer can read USB HDDs, while for DVDs and blu-rays you need a drive, and those are much less common than they used to be, and that trend is sure to continue.

Permalink - posted 2020-04-19

The one perfect programming language

▼ There's an episode of the TV show Friends where Chrissie Hynde has a guest role. Phoebe feels threatened by her guitar playing, and asks her "how many chords do you know?" "All of them."

Wouldn't it be cool if you could give the same answer when someone asks "how many programming languages do you know?"

But maybe that's a bit ambitious. So if you have to choose, which program language or programming languages do you learn? I got started with BASIC, 6502 assembly, Forth and Pascal. Those are now all obsolete and/or too niche. These are other languages that I'm familiar with that are relevant today:

• C
• Javascript
• Python
• PHP

I'd say that either C or Javascript is the best choice as a first language to learn. Javascript has the advantage that you only need a web browser and a text editor to get started, and you can start doing fun and useful things immediately. However, Javascript's object orientation and heavy use of events makes it hard to fully understand for someone new to programming. So it's probably better to dip a toe in with Javascript and after a while start learning another language to get a better grasp of the more advanced fundamentals.

C is the opposite of Javascript. It's certainly not very beginner friendly, not in the least because it requires a compiler and a fair bit of setup before you can start doing anything. And then you get programs that run from the command line. It's much harder to do something fun or useful in C. However, what's great about C is that it's relatively simple and low level, which means that it's an excellent way to learn more about the way computers and data structures actually work. Because it's a simple language, it's a reasonable goal to learn the entire language. That's especially important when reading other people's code. Also, many other languages such as Java, Javascript and PHP are heavily influenced by C, so knowing C will help you understand other languages better.

If you want to be able to be productive as a programmer and you could only use one language, Python is probably the one. It's used for many different things and has some really nice features to help you start going quickly. But it also has many of its own quirks and complexity hides just below the surface, so like with Javascript, I would use Python as a "dipping your toe in" language and if you want to learn more, switch to something else. A big advantage of Python over C is that you don't need a compiler, but it still (mostly) lives on the command line.

PHP is the language that I've used the most over the last 20+ years. If that hadn't been the case, I'm not sure it would have been on this list. It's not held in very high regard in many circles, so if you want something that looks good on your CV, PHP is not a top choice. Then again, it works very well for web backends, and has an incredible amount of stuff built in, allowing you to be productive quickly. It's also close to C in many ways, so that helps if you already know C. But like Javascript and Python it's a dynamic language, so it takes a lot less work to get things done than in C.

Of course a lot depends on what you want to do. For stuff running in a browser, Javascript is the only choice. For low level stuff, C is the best choice, although Python could work in some cases, too. I think for web backends, PHP is the best fit, but Python can certainly also do that. For developing mobile apps, you need Swift or Objective C. For Android, Java or Kotlin. Mac apps are also generally in Objective C, with Swift (Apple's relatively new language) becoming more common. On Windows, a lot of stuff is written in C#. A lot of lower-level stuff, especially graphics, is done in C++. So these are all very useful languages, but I wouldn't recommend any of them as a first language.

So let's have a look at a simple program in each of those languages, and then see how fast they run that same program. For a given input, the program calculates what numbers that input is divisible by. (It's not optimized in any way and there is no error checking, so it's not an example of good code.)

C:

#include <stdio.h>
#include <stdlib.h>
 
int main(int argc, char *argv[])
{
  int n, i;
 
n = atoi(argv[1]);
 
if (n % 2 == 0)
  printf("%d\n", 2);
 
for (i = 3; i < n; i += 2)
  if (n % i == 0)
    printf("%d\n", i);
}

PHP:

<?php
  
$n = $argv[1];
 
if ($n % 2 == 0)
    printf("%d\n", 2);
 
for ($i = 3; $i < $n; $i += 2)
  if ($n % $i == 0)
    printf("%d\n", $i);

Javascript:

<script lanugage=javascript>
var ts1, ts2, n, i;
 
ts1 = new(Date);
 
n = 444666777;
 
if (n % 2 == 0)
  document.write(2 + "<br>\n");
 
for (i = 3; i < n; i += 2)
  if (n % i == 0)
    document.write(i + "<br>\n");
 
ts2 = new(Date);
document.write("Time: " + 
  (ts2.getTime() - ts1.getTime()) / 1000 + 
  " seconds<br>\n");
 
</script>

Python:

import sys
import math
 
n = int(sys.argv[1])
 
if (n % 2 == 0):
    print(2)
 
for i in range(3, n, 2):
    if (n % i == 0):
        print(i)

(For the Javascript version I hardcoded 444666777 as the input, for the others the input is read from the command line.)

Common wisdom is that compiled languages like C are faster than interpreted languages (the others). That turns out to be true, with the C version (compiled with -O3 optimizations) taking 0.7 seconds on my 2013 MacBook Pro with a 2.4 GHz Intel i5 CPU.

But interestingly, the Javascript is barely any slower at just over 1 second. This shows just how much effort the browser makers have poured into making Javascript faster.

The PHP version, on the other hand, takes more than 21 seconds. The Python version 50 seconds. Weirdly, 15 of those seconds were spent running system code. This is because running the Python program uses up 6 GB of memory on my 8 GB system, so the system has to do all kinds of things to make that work.

It turns out that having a for loop with the range function is problematic. It looks like range first creates the requested range of numbers in memory (all 222 million of them!) and then the for loop goes through them. But we can replace the for loop with a while loop:

import sys
import math
 
n = int(sys.argv[1])
 
if (n % 2 == 0):
    print(2)
 
i = 3
while (i < n):
    if (n % i == 0):
        print(i)
    i = i + 2;

This does the same thing, but in a way that's more like the for loops in the other languages. This version takes 36 seconds, and, more importantly, there are no issues with memory use.

C can do these calculations really fast because the overhead of pushing many small instructions to the CPU is small. Each instruction has more overhead in the other languages. With more complex operations, such as manipulating text strings, the C's advantage is a lot less because each operation in the program leads to a much larger number of instructions for the CPU, so the language overhead is a much smaller part of the running time. I haven't been able to think of a nice simple test program to see how big the difference is, though.

Permalink - posted 2020-04-01

Voorstel: maatregelen tegen het coronavirus

▼ Er zijn nu 22 landen met meer dan 100 gevallen van COVID-19 en acht landen met meer dan 1000. In China en Italië, de hardst geraakte landen, werd het virus in eerste instantie onderschat en daarna waren er zeer zware maatregelen nodig, in China op het draconische af. De ziekenhuizen daar zijn of waren zwaar overbelast, wat ongetwijfeld tot extra doden leidt of geleid heeft.

Het zou wel zo fijn zijn als wij het in Nederland en andere landen met een nog redelijk overzichtelijk aantal gevallen niet zo ver laten komen. Liever stevige maatregelen nu dan draconische over één of twee weken.

Eerste stevige maatregel: stoppen met forenzen in de trein. In de spits zitten de treinen hartstikke vol, wat overdracht van het virus goed faciliteert. Daarnaast worden zo infecties van het ene deel van het land naar het andere getransporteerd. Dus geen treinen meer in de spits. Buiten de spits kunnen er dan extra lange treinen ingezet worden zodat mensen op redelijke afstand van elkaar kunnen zitten en iedereen toch kan reizen als het echt moet.

Tweede maatregel: geen bezoekers uit risicogebieden, alleen mensen die terugkeren naar huis, en die twee weken in thuis-quarantaine.

Derde maatregel: in het hoger onderwijs geen colleges meer en geen andere activiteiten in de hogeschool of op de universiteit die ook ergens anders gedaan kunnen worden. Dus bijvoorbeeld: niet studeren in de universiteitsbibliotheek, maar nog wel de mogelijkheid boeken te lenen. Studenten in het hoger onderwijs zijn al zelfstandig dus dit hoeft niet een enorm probleem te zijn. Contact tussen studenten en docenten kan via het internet.

Vierde maatregel: iedereen die in één (of meer) van de volgende groepen valt zo veel mogelijk thuisblijven en niet naar werk of school:

• Mensen die ziek zijn, waaronder hoesten, niezen en koorts
• Mensen die extra risico lopen als ze besmet worden (dus al een aandoening hebben of verzwakt zijn)
• Mensen boven de 60
• Iedereen die samenwoont of op een andere manier contact heeft met iemand uit bovenstaande groepen

Vijfde maatregel: waar mogelijk thuiswerken.

Op deze manier kunnen de winkels en scholen open blijven en blijft reizen beperkt mogelijk en is de impact op de economie nog te overzien. (In tegenstelling tot wanneer de scholen dicht gaan en veel mensen niet kunnen werken omdat ze thuis moeten blijven met de kinderen.) Het virus zal zich zo ongetwijfeld verder blijven verspreiden, maar langzamer. De ziekenhuizen kunnen het dan hopelijk aan, en mensen die het virus krijgen en herstellen worden immuun zodat het virus in volgende ronden, bijvoorbeeld komende winter, minder kans krijgt.

Permalink - posted 2020-03-11