This makes saying things like "Fetch all artists where ..., for each artist fetch their first 3 albums where ..., and for each album fetch the top 10 tracks where ..." really difficult
AWS Athena: selling a buggy, old, stale copy of someone else's work (Presto / Trino) for high prices and getting away with it because you control the platform.
I think many users just see they can execute a query on huge data cheaply and incredibly quickly and are delighted. That's certainly my experience.
It's one of the backends available in Splink, our FOSS record linkage software and it's revolutionary how it allows users to execute large scale probabilistic record linkage ridiculously cheaply. It wasn't long ago you needed very expensive proprietary software plus a big on prem cluster, costing in the hundreds of thousands, to achieve this.
A lot of the magic for me is on the infrastructure side: how they can read/write large datasets from s3 so quickly, so the value isn't just in the SQL engine.
The biggest value for corporate users is that they get everything already included as part of their existing cloud agreement.
Adding a new vendor to the mix needs to involve procurement, the legal team, vendor negotiations, while using a new AWS feature is just a matter of using it, even if it's not as good as the original ISV's version and doesn't support the long term viability of the project.
other arguments aside .. Athena costs $5 per 1TB scanned and also supports predicates pushdown to S3 Select. I wouldn't call this expensive, at least in comparison to self hosted Presto.
At a certain scale it does become very expensive. It's easy math.
When your monthly Athena bill crosses whatever it would cost to have 5 or 10 EC2 machines it'll be cheaper to use Trino. At my previous workplace we moved from ~$40,000/month to ~$18,000/month by replacing Athena.
Athena is a very good tool to start with - unless you have super large scale you'll probably not outgrow it. But when you do there's Trino.
I do contribute to Trino - although I was merely a user when that cost reduction happened.
I'm not sure the math is so easy. Even knowing the direct cost savings in hindsight, engineers' time is expensive, and it's not obvious that the ongoing engineering cost of maintaining Trino on an EC2 cluster would be that far below $22k/month. Even if you get a net cost savings on an ongoing basis (which, granted, you probably do), you may have a long payback period for the initial engineering time spent evaluating solutions and getting the deployment spun up.
And that's all with the benefit of hindsight - it's hard to know a priori how much cheaper your own deployment will be compared to a managed service or how long it will take to implement. Of course, anecdotes like yours help with that, so thanks for sharing your experience!
Sure, I agree, above certain usage threshold hosted Trino becomes totally justified. But then, some engineering time to maintain the cluster has to be factored in as well.. for anything ad-hoc in nature, I would start with Athena by default.
Can’t you say the same thing for EC2 but with Linux instead of Presto? Personally I like Athena. The fact that it’s in the Amazon platform and managed is a plus for me.
EC2 involved substantial VM management and networking innovations that I respect. Ditto lambda and S3. I would not categorize any of these as OSS flips in nearly the way that I would categorize Athena as an OSS flip.
Steven here, I'm Ahana CEO (Managed Service for Presto) and member of Linux Foundation's Presto Foundation
Since you asked about an alternative to AWS Athena, we offer a free-forever in-VPC Managed Service for Presto on AWS, which keeps up with the latest Presto releases.
Note: the original Presto continues to run in production at Meta (fmr. Facebook), Uber, and recently ByteDance TikTok data platform talked about running 1M queries a day with tens of thousands of cores. Some reasons to stay with Presto:
- Reliability and scalability per above
- Cutting edge innovations only in later versions of Presto: multi-level caching (project RaptorX) to boost query performance by 10X+ and table scan improvements (project Aria) to name a few
- Only PrestoDB is hosted by Linux Foundation, giving confidence to community users that future releases will remain open.
Or possibly he is just passionate about a project that he believes in, just like others in the larger Presto / Trino community. Personally, I think both Ahana and Starburst have interesting takes on how to monetize the original Presto platform. And like many, many other data projects, there is room in the market for different platforms. Long gone are the days where everyone was just stuck with a single RDMBS to do everything!
But man is it a huge PITA (especially when doing programmatic code generation of the SQL) compared to LATERAL joins
Someone familiar with the CockroachDB query planner showed me that a window function like this is what Cockroach turns LATERAL joins into for instance:
demo@127.0.0.1:26257/movr> explain select * from abc, lateral (select * from xyz where x = a limit 2);
• filter
│ estimated row count: 1
│ filter: row_num <= 2
│
└── • window
│ estimated row count: 2
│
└── • hash join
│ estimated row count: 2
│ equality: (x) = (a)
│
├── • scan
│ estimated row count: 6 (100% of the table; stats collected 2 minutes ago)
│ table: xyz@xyz_pkey
│ spans: FULL SCAN
│
└── • scan
estimated row count: 1 (100% of the table; stats collected 3 minutes ago)
table: abc@abc_pkey
spans: FULL SCAN
It turns out that the dialect doesn't support LATERAL joins with a LIMIT in them. The below query only works if you remove the LIMIT clause.
https://i.stack.imgur.com/rdB1s.png
This makes saying things like "Fetch all artists where ..., for each artist fetch their first 3 albums where ..., and for each album fetch the top 10 tracks where ..." really difficult
Does Trino support this out of curiosity?