> you will be stuck reimplementing a sql engine in your app code.
Only if you're intent on keeping the s3-only architecture when it no longer makes sense, and the transition to SQL need need not be hard if you've been disciplined about your data structures from the get go.
I've been part of a transition from s3 only to s3/sql when business requirements changed, it only took us a week to set up a db, set up a workflow manager to insert necessary data into the db when something uploaded to s3, and ran some batch jobs for data that already existed. Total data in s3 was ~1pb.
IMO it's all about whether your problem space fits well with s3-only, and sometimes it does!
Only if you're intent on keeping the s3-only architecture when it no longer makes sense, and the transition to SQL need need not be hard if you've been disciplined about your data structures from the get go.
I've been part of a transition from s3 only to s3/sql when business requirements changed, it only took us a week to set up a db, set up a workflow manager to insert necessary data into the db when something uploaded to s3, and ran some batch jobs for data that already existed. Total data in s3 was ~1pb.
IMO it's all about whether your problem space fits well with s3-only, and sometimes it does!