Terraform, Lambda, and Frontend Changes#104
Conversation
This reverts commit 3f5ed96.
…n-fix Update conversion start lambda permissions
… the manager's dockerfile.
…roups / ECS one off tasks
…ntend/Manager setup task changes
…redis / Changed frontend database configurations / Working towards fixed static file s3 storage
…bdas & ecs / Moving geojson metadata initialization from submission to validation lambda
…reet Maps free tiles.
…ager Statistics endpoint bug fixes
…ager Statistics endpoint bug fixes
|
I can't claim to have fully understood every part of the PR (because I haven't internalised all of the structure of the code, only examined the diffs), but except as indicated it appears logical to me. I agree with Brian that we should do a dry install of the system using this setup to check before fully merging. |
|
@gt1074 As part of wrapping up this PR, please implement the S3 backend for storing Terraform state as is done for the UploadServer, as described here. You can use the same default bucket name ("unhjhc-wibl-tf-state"), but pick a different key for now, maybe something like "terraform/state/wibl-processing-server-deploy.tfstate". Eventually, we can use the same key so that the UploadServer can be deployed into the same stack, but for now we'll keep them separate. |
…oving unused subnet_efs Terraform module / manager statistics bug fixes.
|
Do you want me to have the user run a script nearly identical to the Terraform-bootstrap.sh script, meaning the upload server and the wibl-python section both use different buckets. Or do you want to just port the configurations, leading the Terraform module in wibl-python to save a copy of its state file in the same bucket as the upload server, just a different location. |
Just port the configuration over for now, but use a different default key name for the state file for wibl-python. We can merge them later (ideally, the upload server will be fully integrated into the wibl-python cloud stack). I just want to make them as amenable to merging as we can right now. |
…dn delivery / removing commented out code
…state storage module.
…ion-parameters.sh file.
There was a problem hiding this comment.
@gt1074 This file path should be listed in .gitignore. We should then check a template file into Git (e.g., default_auth.txt.proto), then copy the .proto to .txt before the build (probably in build.sh). Users should be instructed in the README to add their DCDB key to the .proto. Then no need for them to change or even look at the auth_file_name TF variable. As it stands, it would be too easy for someone (even us) to push a DCDB key to a public repo.
There was a problem hiding this comment.
@gt1074 On second thought, we'll need to add a step where the user copies the .proto to .tfvars. They shouldn't be adding secrets in the .proto file, since that will be checked into the repo. Duh. Was trying to avoid an extra step for users, but there's not a great way around it.
There was a problem hiding this comment.
@gt1074 Like with default_auth.txt we should .gitignore terraform.tfvars and add a terraform.tfvars.proto to the repo. Though I suppose then we'll need to add a step where the user copies the .proto to .tfvars. Having the build.sh script copy .proto to .tfvars for us doesn't help with accidentally committing secrets if we are having the user edit the .proto file, which would be checked into the repo.
| terraform_state_bucket ="unhjhc-wibl-tf-state" | ||
| terraform_state_key ="terraform/state/wibl-processing-server-deploy.tfstate" | ||
|
|
||
| incoming_bucket_name = "default-incoming-bucket" |
There was a problem hiding this comment.
@gt1074 The user also must be instructed to change all four of these bucket names as S3 has a more-or-less global namespace (unless they are using GovCloud or are in China).
There was a problem hiding this comment.
@gt1074 Actually, I see that as of March 12, the namespace for S3 is no longer global. However, we should still recommend that they change the bucket names.
| } | ||
|
|
||
| backend "s3" { | ||
| bucket = "unhjhc-wibl-tf-state" |
…t_auth) / Changing file names / Adding prototype files
…ocker version requirements / adjusting database naming
3 participants
Full implementation of the Terraform AWS suite. Transfers data pipeline from scripts to Terraform configuration. Initializes all necessary S3 buckets, Lambda functions, IAM privileges, ECS and EC2 modules, SNS modules, and private VPC configurations. All required attributes are configurable through terraform variables.
Changed Geojson metadata configuration. Geojson metadata is now configured in the validation lambda, fixing a bug where non validated files were left hanging in the staging bucket with no way to delete them.
Added AWS SSM parameter to handle the provider authentication string for submitting to DCDB using the submission lambda.
Submission and validation lambda now correctly update metadata to include upload and validation status, while showing whether a file is online, archived, or deleted. Metadata is now kept even if a file is deleted from S3.
Changed python consumers and socket manager to include payload and reconnection support.
Configured the frontend Geojson map to render tiles from AWS Location Services.
Configured frontend to now display status (upload/validation/conversion) and state (online/archived/deleted) of files.
Use AWS RDS instances instead of ECS container database instances for the frontend and manager.
Switched to AWS Elasticache for Redis implementation.