Yeah we can get on a call. My DB might not be in the same state by the time we do though. I'm needing to unblock my team, and will be continuing to prune data in order to do so.
The biggest improvement from what I've seen as I've been looking into this would be to denormalize the project id onto the spans table so that you don't have to do a join from traces to spans for all of the project level queries. For example, just loading the spans for a 1 hour window should be fast, but in order to find eligible spans for the current project, the query plan forces the db into a full table scan of the traces table before joining against spans which is in part due to the low selectivity of the project id column on the traces table.
The second thing would be to separate your graphql resolvers to allow for establishing read only connections so that you can utilize read replicas for read heavy workloads. Right now when we have high write volumes, we end up seeing a fair amount of table locking due to heavy analytics read queries attempting to be run on write heavy tables. This of course would be mitigated by making the read operations less heavy, but any way to offload any read heavy workloads from the writer would be helpful.
