[HiTL] Command Dist Eval & Vision Pipeline Tweaks

[ethan-carlson]

Description

This PR includes a new script eval_command_distribution.py which compares the command distribution of sets of interaction and/or labeling HITs for analysis.

It also includes a few minor tweaks to the labeling -> annotation pipeline in order to specify a) a subset of the IGLU scenes to pass to the scene generator for the purposes of limiting the data distribution and b) the ability to repeat scenes a specified number of times (eg. 10 scenes with 10 HITs each) for the same purpose.

Type of change

Please check the options that are relevant.

• Bug fix (non-breaking change that fixes an issue)
• Proposes a change (non-breaking change that isn't necessarily a bug)
• Refactor
• New feature (non-breaking change that adds a new functionality)
• Breaking change (fix or feature that would break some existing functionality downstream)
• This is a unit test
• Documentation only change
• Datasets Release
• Models Release

Type of requested review

• I want a thorough review of the implementation.
• I want a high level review.
• I want a deep design review.

Before and After

Previously the labeling pipeline generated scenes randomly if using basic shapes, or sampled from the entire database of IGLU scenes if using IGLU. This changes allows for more fine grained control.

Testing

I have run substantial numbers of HITs using this change; it should be reliable.

Checklist:

• I have performed manual end-to-end testing of the feature in my environment.
• I have added Docstrings and comments to the code.
• I have made changes to existing documentation where needed.
• I have added tests that show that the PR is functional.
• New and existing unit tests pass locally with my changes.
• I have added relevant collaborators to review the PR before merge.
• [Polymetis only] I ran on hardware (1) all scripts in tests/scripts, (2) asv benchmarks.

[codecov-commenter]

Codecov Report

Merging #1367 (6ff7fcb) into main (5e4d64d) will decrease coverage by 0.09%.
The diff coverage is n/a.

❗ Current head 6ff7fcb differs from pull request most recent head bea6637. Consider uploading reports for the commit bea6637 to get more accurate results

@@            Coverage Diff             @@
##             main    #1367      +/-   ##
==========================================
- Coverage   34.13%   34.03%   -0.10%     
==========================================
  Files         358      357       -1     
  Lines       35541    35286     -255     
==========================================
- Hits        12131    12009     -122     
+ Misses      23410    23277     -133     

Impacted Files	Coverage Δ
lowlevel/minecraft/pyworld/fake_mobs.py	73.33% <0.00%> (-13.34%)	⬇️
lowlevel/minecraft/small_scenes_with_shapes.py	8.80% <0.00%> (-0.13%)	⬇️
agents/droidlet_agent.py

[facebook-github-bot]

Hi @ethan-carlson!

Thank you for your pull request.

We require contributors to sign our Contributor License Agreement, and yours needs attention.

You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!