To Do

[DanRoscigno]

• Possible architecture: One CI file per "test environment." where a test environment is the StarRocks deployment method. Here are some test environments:
  • allin1
  • one FE and one BE
  • the Hudi demo
  • the Iceberg demo
  • (demo does not yet exist, and may require more resources than available in a GitHub runner) k8s operator
  • (demo does not exist) recommended config with 3 BEs
• We reference compose files in the starrocks/demo repo in the docs, so we need a system to grab the compose file from the other repo (and any needed conf files) and run that compose file. Probably in our CI workflow:
  • checkout the demo repo
  • launch the demo system
  • launch our golang container in the same Docker network as the demo system
  • run the tests
• Use the files in SQL/quickstart/basic (and in SHELL) in the file website/docs/shared-nothing.md and link to an example of SHELL and an example of SQL inclusion from the README and website/docs/setup.md
• Add verification to the test steps. At the moment they all check for failure, instead of checking for failure and verifying correct output. For example, the curl command that loads the crash data should have a single record rejected as the data contains a record with a problem. The test returned by curl should be checked for "NumberFilteredRows": 1,. This type of verification should be done to catch changes in behavior of the system.
• Decide on a strategy for test file layout based on how test results can be reported. It would be nice if a test failure opened an issue saying "The test of https://docs.starrocks.io/docs/quick_start/shared-nothing/ has failed at step STEP: JOINing to see impact of icy weather. Maybe this means a separate test file per URL?
• Move common code to a helper file:
  • connect
  • check return value in curl output
  • check return value in DESCRIBE
  • check return int value
  • check return string value
  • etc.