Probably a LTO drive a few generations old and a SAS card. Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer.
From personal experience you are not going to be able to saturate even a horrifically obsolete LTO drive and SAS interface without specialized software or purely sequential data. Most of the cheap/free ways of running tape drives aren't optimized for parallel I/O and will horrifically shoe-shine (that's the rev-up and rev-down sound you hear when the tape drive isn't getting enough data), which isn't great for the tapes and massively increases backup time and drive usage.
AFAIK most commercial tape deployments nowadays are disk-to-disk-to-tape arrangements. All the actual data is serialized to an archive on disk first, and then that serialized archive is written to tape at full speed. This minimizes tape wear and ensures your very expensive tape drives are being used efficiently.
I am using a LTO-7 drive connected to a FreeBSD server.
FreeBSD always succeeds to write the tape continuously at about 300 MB per second, which is the maximum speed for LTO-7.
All the files send to the tape are grouped into large archive files and for the dd command that writes to tape I use a block size of 128 kB.
The tape commands from FreeBSD are more convenient than those from Linux, which have not seen much maintenance in recent years.
Obviously, you cannot reach tape speed when making the backup directly from a HDD or from a 1 Gb/s Ethernet.
You must write the backup to tape either from a fast SSD, or from 10 Gb/s Ethernet coming from a fast SSD at the other end, or from a RAM disk configured on the server, if you have enough memory.
To not wear unnecessarily the SSDs on my server where the tape drive is located, whenever I write the backup, I configure a large RAM disk on the server. The backup files coming through Ethernet to the server are written to the RAM disk on the server, then they are copied to the tape.
With this arrangement it is very easy to ensure that the tape drive is written at maximum speed without any hiccups.
I would like very much if someone would introduce a tape drive with an USB interface, using the USB-Attached-SCSI-Protocol.
This would have no importance for software, as the tape drives would continue to be seen as SCSI devices. The performance of USB is adequate and an USB tape drive could be cheaper, while also saving money for not needing a SCSI HBA.
Unfortunately, there are no tape drives for modern tape standards with USB and no hope that one will appear soon.
I use a tabletop tape drive. Internal tape drives are a little cheaper, but I think that the external drive is more reliable, because it is protected from dust when not active and it is well cooled (even if it is noisy) when it is active.
Previously I have used a SCSI HBA card for PCIe, together with an external SCSI cable having the appropriate connectors.
The server motherboard that I am using now has on-board SCSI, so I have a cable from motherboard to the case, having internal SCSI connectors at one end, to plug in the motherboard, and external connectors at the other end, in a cover for a the opening of an expansion card slot on the case.
Then I have an external SCSI cable to connect the tabletop drive.
With SCSI cables, you must pay attention to the connectors at each end, because there are many variants. On the motherboard I have internal high-density connectors, on the case I have external high-density connectors, while on the tape drive I have external lower-density connectors.
I have used 2 kinds of SCSI HBA cards, some with LSI controllers, e.g. LSI LSI00343 SAS 9300-8E Host Bus Adapter, and a similar card with a Microsemi controller (now Microchip). There were no differences, all were OK.
The SCSI HBA cards may have either external connectors or internal connectors or both kinds, so you must choose the appropriate card, depending on whether your drive is internal or external.
The only difficulty that I had in the past was that on some motherboards the computer did not boot with the SCSI card inside and I could not understand why.
I have even bought a second SCSI HBA card from a different vendor, wrongly believing that it is buggy.
The problem turned out to be not a bug, but a standard feature :-(
When booting in legacy mode, some add-in cards, including all SCSI cards and all GPUs, attempt to map their ROMs adding BIOS functions in the address space above 640 kB but under 1 MB, which is accessed in the Intel real-address mode.
The SCSI cards do this for the case when you will attach a HDD or a SSD and you would want to boot from it, which the MB BIOS does not know how to do.
On the computers that refused to boot, without any errror message, I also had a NVIDIA GPU. The sizes of the SCSI ROM with the NVIDIA ROM together exceeded the reserved address area mappable by the BIOS, so the BIOS failed to initialize the GPU. When either one of the 2 cards was present booting was OK, with both cards the computer remained stuck with a blank screen.
The solution for those cards to work together was to go in the BIOS at PCIe properties, select the slot where I intended to put the SCSI card and disable the mapping of the extension ROM.
However, not all BIOSes have these options. On one of my Supermicro MBs, the BIOS had the option, but it did not have any effect, due to a BIOS bug. While I have been very content with the hardware of my Supermicro motherboards, I have encountered a lot of bugs in their BIOSes.
Because I use the HBA only for tapes, disabling the ROM has no consequence, because I never want to boot from SCSI.
Very insightful comment on the data rate, if you can't feed the LTO properly, it may be stopping and starting all the time, which could have a detrimental effect to its mechanism. But this is probably easily fixed by spooling (using bacula terminology). You buffer like 100Gbytes, write them in one batch. Plus, some drives support variable speeds.
Also, in addition to SAS, a fibre channel card would also fit the bill, albeit probably a trifle more expensive. But if you go for low speeds (e.g. 8Gbps), those cards can be less than 100$.
For example, I collect all the files that I send to backup in archive files whose size is approximately 50 GB, then I copy to the tape the 50 GB files.
Doing like this, the write speed to a LTO-7 drive is always constant at 300 MB per second, which is the maximum possible for that standard (which is much higher than what a HDD can sustain, so the 50 GB files must come from a fast SSD or from a RAM disk).
I didn't know -- or perhaps remember -- that you had to match the feed against the speed, sort of like the way you had to be very careful about burning CD-Rs in the old days, but that makes perfect sense.
I have been fearfully realizing that I will soon need tape backup for my next project and this is helpful. Now I am wondering if a RAID 0 of multiple HDDs could provide the ~300 MB/s speed needed.
You do not have to match the speed, but you should.
Sending the data fast enough to the drive will reduce both the total time required for backup, when the drive is active, and it will also avoid starting and stopping a lot the drive during the transfer.
Both eliminating the start-stop cycles and reducing the total active time will increase the life of your tape drive.
On my server I have 128 GB of DRAM, which makes it extremely easy to ensure maximum speed without wearing a SSD.
When the backup starts, I create an 80-GB RAM disk, larger than the up to 60 GB archives in which I collect the files sent to backup.
With less memory one could either use smaller chunks or use a SSD instead of the RAM disk, but that seems wasteful.
Then the archive files are buffered in the RAM disk and written in one command, without pauses.
> Current generations cost too much and you aren't going to be able to keep up with the data rate that they prefer.
Maybe. Keeping up only really matters if your slowest data rate is going to be between 50MB/s and 120MB/s. If it's below that then you have a strong need for a buffer drive no matter what tape generation. And if it's above that then you shouldn't have any real trouble no matter what tape generation.
We switched from Google Maps to OpenMapTiles for the same reason. Running OpenMapTiles via Docker was pretty easy. We bought a license for the Tiles (Europe) and it was a one time fee only.
Additionally, if you choose to use it, a map style is copyright: the design of showing motorways in blue at 4px and cycle paths in green, for example.
For the tiles you see on OpenStreetMap.org, all three parts are under open licenses. (And there are decisions at all three levels, for example OSM's database contains the water features you see at OpenSeaMap.org, but doesn't render them.)
The data is open and can be downloaded from the donation-run OpenStreetMap Foundation, and as long as you provide credits you are free to do anything with the data (commercial or otherwise), but if someone makes a custom style and creates tiles (images) from them, then those images are copyrighted.
I actually saw it but forgot to add it to the list. To be honest nvAlt and Quiver are my two favorite and make me sad that I am on Windows and can't use them. They really don't have any good equivalent on other OS.
I'd like to add treesheets (http://strlen.com/treesheets/) to the list: it's an offline tree based editor so it naturally has an infinite number of levels of details you want it to have. The format is a (custom) single file so it's easy to backup and share. While all data is in a tree, it is possible to display it as a grid. I'm not using it currently because I don't really need it, but it's the first one on my to-try list.
I went through that same list while researching note taking apps a couple months ago. I downloaded and tried a bunch but none of them really hit the sweet spot for me.
I'm glad that Collate hits the same use cases for you as it did for me!
It's good to have the explanation for beginners, but for people that just want to get the software (and kind of know what they're doing) there should be a bigger button in addition to the text link. :)
I'm currently waiting for my Apple and Windows code signing certificates, which will mean I can ditch the while "How to install" pages and just have the binaries install when you click on the big button on the main downloads page. Sadly this seems to be taking a while (I've been waiting on my Windows certificate over 2 weeks now) but I'm hoping I'll get them soon!
I basically do the same. I work 4 days a week and I feel that I'm not really less productive. The fact that I have one day per week off, gives me a reason to procrastinate less and do more with the time I have.
I remember the time from an old job, were most employees did nearly nothing on Friday afternoons. They were burned out from the week and waited until the time was up to finally go home.
I work 4-day weeks for a few years now and I can say that I don't want to change this anytime soon. Having more time for friends and family makes me much more happy than a bit more money. And of course, I feel much more relaxed on Mondays since every weekend is a "long weekend".
I observed a similar productivity drop on Friday afternoons, although in my devops-y team it's mostly due to reluctance to deploy any changes right before the weekend.
That's the thing with personal pictures online. Most people can't imagine what is going to happen in the future. People on Hacker News well understand this issue, but ask your non-computer friends and family.
If you'd tell them, that you can take a picture of a random person and search that picture out of a pool of people and get their contact info (as a private company / random programmer with enough data), they won't believe you or tell you that something like this is only possible in movies.
Even today it's no problem to analyze the Facebook picture of some random person and calculate a chance of that person being an alcoholic in X years based on the number of party pictures they share.
Before sharing personal information online, people should not ask themselves what could be done with that data now, but what is possible with that data in the future.
If Facebook is going out of business (just as a thought), their data has a lot of value. Don't think that they will delete everything. It will go to the highest bidder!
That's a really hard battle to fight, even for those of us who understand the full implications. The entire population is not going to drop facebook overnight, it does deliver some sort of value to them despite it's heavy privacy price.
I'm beginning to believe that the answer is to simply devalue all that information. We can do it in some contexts on the technical side, for example, rotating passwords devalues stolen credentials, virtual credit cards, random MAC for wireless scanning, etc.
So how far can we push this? If you got into my email, but 90% of the messages there were generated by AI and 10% are legit, can you accurately profile me?
To get a little more on topic, how close do you think we are to 3D printers that can print a mask of your face? Once that happens, will the value of these "face recognisers" drop off a bit?
For your point about face recognition, this art project may be of interest to you: https://cvdazzle.com/
A very interesting idea that email privacy might realistically turn into a steganography problem - how do you make it so the real content is indistinguishable from fake content, and given that this is imperfect, make it so that all but the most sophisticated attackers wouldn't be able to tell that a hidden message even exists?
That sounds a lot like security by obscurity, one blog post later and the world can distinguish the two.
Cvdazzle is unlikey to be the answer, it's just today's CAPTCHA to slow down face recognition, tomorrow's recognition will cut through that
We need a more radical solution to devalue our private information. Consider by analogy cell phones: IMEI blacklists (in theory) fundamentally devalue stealing phones. What can we do that fundamentally makes assembling troves of personal information useless? Poisoning it with plausible garbage might be part of the answer. Also the answer /might/ be regulatory. HIPAA has teeth, we could add something else with teeth. Or perhaps an insurance-driven approach: if holding personal information was a big liability in the event of a leak, it would be prudent to keep as little as possible.
Not at all. People can opt-in and send their own photos, holding that data would be kosher. Holding photos of those who did not opt in, or those who explicitly opted out, would be where the liability comes in.
If we can penalize people for possessing some kinds of data (pirated software, movies, etc) in the name of defending someone else's copyright, or for not decrypting things, it can also be done in defence of personal privacy.
Or perhaps it would result in only the companies that need that data actually holding it, and doing so with care. I would love to see a big nasty liability poison pill attached to PII
I have hope that this will change. Because more and more people live a fake life on social media.
Some people go crazy and share everything, but more and more people share less[^1] and if they share something, it's only something that holds together their "clean image".
Sure, that's a good sign, but it's still undone by someone else who posts a photo or recording of you. If I recall correctly, facebook used to (maybe still does) solicit users to "report" anybody who is not using their real name on the site. When will facebook start soliciting/rewarding users for uploading photos of others? How much does that guy in your class value your personal privacy?
I think is more phobia about technology than an actual problem.
I get that this is a radical view on HN, but when topics like this come up, it's always slippery slopes and fantastic nightmare scenarios (like 3D printing your face). I feel like we're in the 1980's debating over how to handle traffic laws for our flying cars in 2010 just because someone built one.
That HN comment you linked to is _still_ just a nightmare scenario.
If you read the article, you would see that they are lying about what happened. Even your question, based on incorrect information, is implying a slippery slope.
Once the nightmare scenario arrives though, it's usually too late to do anything about it. Look how impossibly hard it is to stop the NSA from gathering all of the network traffic. I wouldn't doubt for a second that Kim Jung would use this against protesters (if North Koreans had social networking). Just this morning I heard on NPR about a group working on genetic modifications to make half-human, half-animal beings, apparently for organ harvest reasons. However you feel about that, nipping things in the bud is so much easier than trying to reverse history.
Right, same with every other "sky is falling" argument.
The PRISM dark fiber exploit has since been blocked by most tech companies, and honestly, you're more likely to be negatively affected by slipping in your shower or a car accident. Doesn't mean don't be aware, it just means you don't need to wear a tinfoil hat. Or worry about hypothetical nightmare scenarios.
What makes you think they are lying?
Our view on the downsides of technology might be biased because we are (often, unless you're a moderator or a victim) sheltered from them[1].
[1] http://www.theverge.com/2016/4/13/11387934/internet-moderato...
Edit: grammar
They lied about persecution of gays and sex workers. The article was about a small group created to raise awareness about face recognition by trying to identify porn actresses, before closing entirely. Hardly a nightmare made real.
I'm also not sure how YouTube mods from 10+ years ago are relevant in a discussion about tech advancement. If anything, the automation of child porn detection since is proof that technology keeps up and prevents nightmare scenarios.
You don't need to drop Facebook -- just turn privacy settings so that your face can't be identified publicly or beyond 1 degree of connection. People should know how to do at least that.
Your main profile picture is always public, as is your cover photo, so you will have to use a picture that doesn't identify you, which may be confusing to your friends.
> Even today it's no problem to analyze the Facebook picture of some random person and calculate a chance of that person being an alcoholic in X years based on the number of party pictures they share.
The problem is not that this calculation exists, but that people so easily misinterpret it. It doesn't mean "Person with these photos has a 65% chance of being an alcoholic in 5 years", what it REALLY means is "65% of the people with these photos became an alcoholic in 5 years".
That is a HUGE difference. For the remaining 35% there is perhaps some additional factor not included in the existence of the photos that guarantees there is a 0% chance of them ever becoming an alcoholic. When you try to apply general statistics about a large population to a single individual, it's not as simple as just saying "65% of the time this person will become an alcoholic". That's ridiculous anyway, because the person will only live one life, one time.
The calculation is useful from a marketing perspective. If I decide to advertise alcoholism treatments to these people in five years' time, I'll have a lot more success then if I'd just picked people at random to advertise to.
This distinction seems relevant only if you target one person, which seems pretty rare. In reality you're going to use this metric on a pool of people, in which case 65% of them would become an alcoholic in 5 years.
It becomes a problem when it's used for profiling, such as "Don't hire that person, there's a 65% chance they'll be an alcoholic" or "send a drone to kill that person, there's an 80% chance they're a terrorist"
I agree that it's a problem, but your distinction still seems generally pointless. If you have a company policy to not hire alcoholics and you reject 1,000 people based upon this metric then 65% would have been an alcoholic.
The thing is, under a certain age everyone is doing it. When this current teenage generation grows up and gets into leadership positions - especially politics, they are all going to have their entire upbringing exposed for the whole nation to see.
The few that haven't will be viewed with cynicism and will most likely not be accepted by the population.
The population will be voting for their leaders based on how they were brought up.
It is going to be quite an interesting social phenomenon to witness.
> When this current teenage generation grows up and gets into leadership positions - especially politics, they are all going to have their entire upbringing exposed for the whole nation to see.
But that whole generation will be in the same situation. Everyone will have had to deal with it; embarassing/compromising college-era party photos will just be a thing that nearly everyone has dealt with.
Decades ago, having had an abortion or a child out of wedlock was a mortal social sin. Today, it doesn't matter.
That's a good point. But the only generation that's going to be affected is the facebook generation.The current teenage gen will just stick to whatsapp/kik conversations or snapchat and won't advertise themselves like fb gen did.
In that case analyze the photos sent on snapchat or whatever network is cool today. The problem is not only photos posted publicly (though it's most visible there).
For me the trick seems to be to get people to treat the net as if it was downtown during high noon.
Just because you are sitting there typing in your undies do not mean that what you post has the same privacy.
Also the worrying bit is not so much the data, as the actions taken on their basis.
Denying someone a job because they have a few photos of them enjoying a colorful drink is just the tip of the iceberg.
What about the day a kill shot is approved because the computer says the face is above a certain likelihood of being some big bad terrorist? At 70% there is still a 1 in 4 chance of being wrong.
Employers routinely subject applicants to personality tests, IQ tests, fitness tests for jobs that have nothing to do with manual labor (essentially age/gender discrimination), background checks, credit checks even when the job doesn't involve handling money, and yes, looking them up on social media. Do people really want to work for these places? Yes, because they need to pay their rent, and don't have the nestegg of a well-paid Silicon Valley software developer, or recruiters chasing them to offer them work.
The sense of free will is an illusion of physics and consciousness; choices are made by quasi-deterministic electrochemical processes before our conscious minds are ever aware of them.
Two approaches to stave off the inevitable come to mind:
1. Do whatever it takes to be "behind the curtain" and one of the people who is in a position to do the exploiting, rather than being exploited. (It's up to you what you do with that power.)
2. The opposite of signal is noise. The same hackers who build these Big Brother tools will inevitably build the tools that flood Big Brother with noise. Disinformation campaigns will always be capable of re-training the surveillance machine.
It honestly may be more productive to operate under the mindset of "My private information that is acquirable by a cursory examination of me (this includes face, which is obviously publicly visible) will inevitably find its way online in a way outside my control. What do I do given this reality?"
There is a fine line between "somebody who knows me (even if not personally) can recognize me on the street" and "everybody can know everything I do out in public or online"
This is an unbelievably ridiculous argument, the government has data on almost entire population thanks to all sort of identification cards and other innumerable sources of data. Why should corporations and other people be denied the same access & privilege.
You are spreading pure FUD, out of ignorance and misplaced sense of paternalism. The reality is that most people correctly understand the trade-off between privacy and utility. That's why they share information on social networks. They are not paranoid running around fearing government conspiracy theories. They just want to enjoy their short lives.
>> So what you're saying is, just because our governments collect everything, everybody else should be allowed to do it too, or stop worrying?
Yes, Strong corporations and public rights, are necessary to ensure, that government does not becomes too powerful. And there is at least some balance of power.
>> The fight for privacy is not for us, it's for generations to come.
Please define "us"?
"us" or "people" or "generations" are rhetorical devices, with little basis in reality. The romantic notion of "people" quickly disappears when those people either organize into corporations or political parties. At certain level Corporations & Governments are people, or aggregate expression of will of the people. The economies of scale necessitates strong corporations which can protect "people" from government and vice versa.
> Yes, Strong corporations and public rights, are necessary to ensure, that government does not becomes too powerful. And there is at least some balance of power.
Are you suggesting here that corporations are a meaningful brake on state power?
Given that corporations are a legal fiction only made possible by the state institutions of law, currency, markets, international treaties, trade, and war, I don't see how this assertion can be supported; the origin, future, incentives, and fortunes of states and corporations have always been inextricably linked.
'Anarcho-capitalist' and libertarians solutions for non-state corporations have always appeared to me to be shallow hand-waving (speaking as a former anarcho-capitalist and hand-waver myself).
> Strong corporations and public rights, are necessary to ensure, that government does not becomes too powerful
Corporations are an exercise of government power; their strength isn't a safeguard against government power, it is government power.
So the real message here seems to be that exercises of government power that disproportionately favor a certain narrow elite are seen as necessary to prevent government exercising power in ways that benefit other groups.
By that i mean, that our decisions today, will effect everything that follows.
If we educate people that privacy is a privilege, they might not do things (sharing personal information) in the future, that they would otherwise do.
And i'm not talking about world-changing stuff here, it's the little things, like sharing very personal information on social media, like my party picture example from above.
The problem is, most people don't understand the implecations of their doing. And that's why we, who understand those issues, need to educate them.
Another thing is, social media is sold as this warm and cozy place where all your friends are. But in reality, you share your information with big corporations that want to make money (which is totally fine, they provide a service).
People just need to understand, that social media is not a private and social environemnt, it's a tool, provided by a company that has (in most ways) no interest in your personal privacy.
As long as everyone has access to the data that's fair. What happens in practice though is that only a few actors with large resources do, thereby giving them even more power.
>Why should corporations and other people be denied the same access & privilege.
Should we be offering that access and privilege to governments? Assuming that we should, your argument seems to be that it's "unfair" not to offer the same to businesses? Bizarre. Perhaps you can clarify.
>The reality is that most people correctly understand the trade-off between privacy and utility.
I would argue against that. However, even if "only" something like 20-30% of people don't understand the trade-off, that's a huge problem. Additionally, those who understand the trade-off that they are personally making in the present may not be giving thought to problems it will have for them personally in the future (the argument of the commenter you are responding to), or problems access to huge amounts of data causes in society right now or in the future.
>> Should we be offering that access and privilege to governments?
They already have it.
>> your argument seems to be that it's "unfair" not to offer the same to businesses? Bizarre.
In Sorrell vs IMS Health (2011) Supreme Court ruled 6-3 in favor IMS Health, that Commercial Speech was protected under First Amendment. This case is particularly relevant since IMS Health counsel successfully argued that Government could use Prescription data to market cheap generics to physicians however Pharmaceutical companies were prohibited from using the same information. The judges decided that this was unfair and amounted to restriction of commercial speech.
And few days ago the US Supreme Court in 6-2 decision in Spokeo inc. vs Robins ruled that there has to be a concrete evidence of harm/injury before data brokers can be sued.
So regarding my reasoning being Bizarre I think not.
Whether the government already has data is immaterial when the question is whether they "should" have it from a moral or philosophical standpoint, which is the context I'm assuming we're working within. The legality of their having it certainly has questionable constitutionality.
In Sorrell vs IMS Health, the issue does not seem to be whether government and private companies could use the same data to market pharmaceuticals, but whether pharmacies could sell prescription data without doctors' consent. "Fairness" to companies using the same data as government appears to have nothing to do with it. You missed a period at the end of your link, so I added it in mine below.
If we ignore that and assume your interpretation is correct, your conclusion that private individuals and organizations are privy to the exact same personal information that all government organizations may have is definitely not the conclusion that outcome would point towards and is still, indeed, a bizarre belief.
As to your final argument, who is talking about suing companies? If you're responding directly to something I said, the logic to connect it to your response is unclear at best.
You do realize the irony in your argument, considering its the Russian government which has probably the worst track record in recent times when it comes to a corrupt government abusing its power.
What you are essentially saying is that [1] this is bad but a government having a similar tool is fine.
I never said the government having it is fine. But the fact that it does doesn't mean you shouldn't be concerned about corporations and private people having it. And I certainly did not mention the Russian government.
The things is, can it run natively on a decent array of hardware? Unless it can reliably use Windows drivers I expect the hardware support to not be that great. After all, Linux & FreeBSD, which are much older and a lot more popular, still have problems sometime with hardware support.
You'll notice (from reading the youtube description) it's using the VirtualBox Direct3D driver (which was written with windows as a target, not reactos). Support for windows driver is one of the goals of the OS IIRC.
I think there is a reason, why crank or hub located motors are the norm now.