I once visited the Library of Congress on a day when their online library catalog was down. Like the British Library, the stacks are closed to the public, so you have to make requests and they'll deliver the books. Normally, the requests are made online, but that was impossible this day. The library staff found the paper tickets to make requests the old way, and I made a bunch of requests for books by call number based on what I had saved in Zotero. I'm glad that Zotero automatically grabbed the call numbers from the metadata. It was almost like being transferred back in time.

I wanted to get some journal articles that were not digitized. I didn't have the call numbers of the journals saved. I noticed that the Science and Business reading room had an old journal card catalog in the front. (It was still there the last time I visited about a month ago.) So I started pulling entire card catalog drawers to flip through at my desk. A librarian asked me where I got the journal card catalog from because they didn't know! (They also said to be sure to put back the drawers when I was done.)

I don't know how I would have figured out the call numbers for any books that I didn't have saved in Zotero. I guess they eliminated the other card catalogs.

I was still able to have a productive day despite the online catalog being down. I guess that's not as easy at the British Library.

The same happened to the Toronto Public Library around the same time, probably coincidence rather than connected. It has made me reflect on the breadth of non-traditional public services provided by libraries. I noticed because I was looking for practice questions for the Canadian citizenship test, and one of two non-profit sets is from the TPL (the other is from Richmond Public Library), and now it's no longer available anywhere.

It's not clear to me whether moving this role to the lowest level of government is a good idea or not: citizenship is one of the key responsibilities of the federal government, but it probably is better facilitated via new immigrant welcoming programmes which are usually community based. However, it seems pretty clear that if that is being done then we need to provide the resources to do so properly.

Also libraries are great.

I think we need a finer model than just centralized vs decentralized. You can have the federal government define standards and provide quality software options for the local administrations.

Ah, all I remember was printers not working at the time. I didn't know they lost stuff. That sucks.

I think we have been far too complacent as a society in tolerating these kinds of assaults on the backbone of our civilization. I would hope the perpetrators are being… dealt with.

They might be in slightly less trouble if they had provided dumps of their catalog for download in the past, like other libraries do.

The Library of Congress's offerings: https://www.loc.gov/cds/products/marcDist.php

Apparently no backup, along with private user book history stored on the same system as everything else.

With no backup, it's a matter of when you lose it all, not if.

No backup or the backup systems could be hit by the same problem.

A lot of “backups” are onto online storage, which is great when you accidentally delete a file. Not so much when you say lose control of a domain admin account.

Replication && untested backups != tested backups. Tested backups that exist in isolation from primary systems and sites are the only kind that matter.

(Also required: bare metal recovery environment and a DR runbook.)

I strongly suspect that most companies of that size and age do not have a tested run book to recover from bare metal.

problem is that it's an edge case and most execs are only going to be around for 3 years max...

> backup systems could be hit by the same problem

A backup system is not a backup system if it fails because of the same incident happening to the primary.

So I have a backup tape store that I keep offsite in the same town. Then a nuke hits.

You’ve always got a potential for a single event to knock out your backup system, the question is what level of events are you happy with.

My company is supposed to survive nuclear war, obviously many systems will be affected and may even be unrecoverable in a widespread nuclear attack, but that’s one extreme. At the other end a simple fire or natural disaster knocking out one cry would only cause limited loss of resilience and peak capacity. I suspect many smaller companies based in say New York have no plans on how to continue if the entire of NY was lost.

An online backup in another city would be more useful than an offline backup in the same building for some types of loss (fire for example), but not for other types (ransom ware)

Cyber attacks are nasty because unless you actually have restored your backups, they aren’t worth the tape they are printed on. They could have been corrupted ages ago. I’m sure everyone on HN has regular backup restore checks, but I doubt it’s ubiquitous in company’s with regimes in the 100-200m range. And while people may do some types of restore, how about restoring everything when you’ve lost everything but a few LTO tapes? How long would it take before you are back up. How long before you are hacked again?

Then there’s insider threats. If your head of IT has his kids bundled into a van, what damage could he do to your network and data, how long would it take for you to recover.

Some perspective:

In Boeing airplane design, the philosophy is the aircraft must be able to survive any single failure of a component or system. Everything that is flight critical is redundant.

But if the airplane flies into a cliff, or a missile hits it, there is no backup plan.

Undergoing a ransomware attack is commonplace, and so a backup plan for that scenario should be a minimum standard. Surviving a nuclear attack is an unreasonable requirement for most operations.

Definitely backups to write once, read many mediums are desirable. Even a disk array behind an API can effectively become write once, read many.

Off site encrypted backups that are audited occasionally would prevent a lot of pain. Sad to see this happen to a library.

With governments cutting funding to libraries across the globe (including the US where it's worse off), cybersecurity incidents like this will sadly continue to happen. I'm sure they want libraries to be extinct if they were created in the modern era.

These days, what could be the excuse for having an independent backup now and then? How do you implement a major database system for your business with not a thought to backup?

Moral of the story: saving on the cyber security doesn't pay off.

Moral: you can buy 20T drives for $300. No excuse for not copying your data onto them now and then, and storing it disconnected from a fragile password.

It's not exactly legal but you can download a libgen dump & Wikipedia (is legal, but less thorough), put it on such a drive, and then have a good proxy for all academic knowledge on your network — not bad.

The silver lining is an opportunity: With regular lending and other functions shut down, what could they do that would be too difficult or impossible otherwise?

I think nothing, since they don't know where to find a specific book.