Attachments, structures, and tables are all operated on quite similarly.
It would be ideal to begin moving them all to a shared "components" folder and MongoDB collection.
A few steps will be needed:
- Data storage
- Storage of data should be moved out of MongoDB and into S3 via Parquet files (potentially a more advanced storage - delta tables, iceberg - but that is largely unnecessary at this point). MongoDB will then be in charge of handling metadata and maintaining references to the S3 objects
- Unify MongoDB collections
- After moving unique storage information into S3, the components will hold metadata and an S3 reference. This will let us use a single "components" collection and key off of a field - "type" - to delineate between attachments, structures, and tables.
- Unify API server logic
- The API server will have a simpler structure of code by combining the attachments, structures, and tables domains into a single "components" domain
Attachments, structures, and tables are all operated on quite similarly.
It would be ideal to begin moving them all to a shared "components" folder and MongoDB collection.
A few steps will be needed: