What AWS stack will let you build something like this? I don't think I would trust a 3rd party with all of my log data for...I can't really see what the benefit of moving to cloud would be, a log file sits on a local disk or if you had to move it to the cloud you could use S3 or SQS even no?
Nothing on AWS will give you the UX that LogDNA is delivering. And a good experience working with logs can be a huge productivity and visibility boost to your team.
But the infrastructure components are certainly there on AWS.
CloudWatch Logs will ingest tons of data at minimal cost. It has simple API for searching for events, but it's extremely slow to return results if you have lots of data in a LogGroup. It has a simple API to get the latest events but it has a 10s delay and its a bit tricky to get all the events from all the LogStreams in the right order.
Kinesis will ingest tons of data at minimal cost, and let you stream it back in real time.
Lambda can subscribe to all the CloudWatch Log or Kinesis Events for parsing, filtering, and forwarding.
AWS ElasticSearch can ingest all the logs and give you a richer query language and visualization tools.
I've been building all this into Convox, an open source AWS toolkit. Take a look at this CloudFormation template for an example of how to configure a CloudWatch Log Group and Lambda filter in your own AWS account:
Mainly so you can avoid tailing or grepping from 1 server at a time, or setting up an elaborate log shipping method of your own. We'll do that for you. :)
Another benefit is if you have many services then viewing the logs in timestamped order can make debugging across services far easier. This is really quite difficult without a single service aggregating and indexing all your logs
It gets especially useful when you start to get unique ids across "jobs". Imagine a single flow of work hits 10 services, but you can search for job_id=123 and get all the logs for the job . Now its easier to see what went wrong and where in the flow.
