Skip to content

Improve content length handling by using S3 as the source of truth and logging size mismatches #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MasloMaslane merged 1 commit into from
Apr 4, 2026
Merged

Improve content length handling by using S3 as the source of truth and logging size mismatches #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 23 additions & 10 deletions src/routes/ft/get_file.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,11 +141,19 @@ async fn ft_get_file_inner(state: &AppState, path: &str) -> Result<Response<Body
// (including Garage) complete in-progress GETs even if object is deleted.
let _ = guard.release().await;

let content_length = if compressed_size > 0 {
compressed_size as i64
} else {
s3_content_length.unwrap_or(0)
};
// Use S3 content-length as source of truth for Content-Length header.
// DB compressed_size can diverge from actual S3 object size (e.g. different gzip
// encodings across uploads with same content hash).
let content_length = s3_content_length.unwrap_or(compressed_size as i64);
if compressed_size > 0
&& let Some(s3_len) = s3_content_length
&& s3_len != compressed_size as i64
{
error!(
"compressed_size mismatch for hash {}: db={} s3={}",
hash, compressed_size, s3_len
);
}

debug!(
"GET {} returning Last-Modified: {} (unix: {})",
Expand Down Expand Up @@ -208,11 +216,16 @@ async fn serve_file(state: &AppState, path: &str) -> Result<Response<Body>> {

let _ = guard.release().await;

let content_length = if compressed_size > 0 {
compressed_size as i64
} else {
s3_content_length.unwrap_or(0)
};
let content_length = s3_content_length.unwrap_or(compressed_size as i64);
if compressed_size > 0
&& let Some(s3_len) = s3_content_length
&& s3_len != compressed_size as i64
{
error!(
"compressed_size mismatch for hash {}: db={} s3={}",
hash, compressed_size, s3_len
);
}

let body = Body::new(byte_stream.into_inner());
Ok(build_ft_file_response(
Expand Down
23 changes: 13 additions & 10 deletions src/routes/ft/head_file.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,17 +113,20 @@ async fn ft_head_file_inner(state: &AppState, path: &str) -> Result<Response<Bod
// Release lock — all metadata reads complete
let _ = guard.release().await;

// For legacy blobs without stored compressed_size, fall back to S3 head_object.
// This runs after lock release — a concurrent DELETE could remove the object,
// causing us to return Content-Length: 0. This is acceptable: the file is being
// deleted anyway, and HEAD has no body to truncate.
let content_length = if compressed_size > 0 {
compressed_size as i64
} else {
match state.s3storage.object_exists_with_size(&hash).await {
Ok(Some(size)) => size,
_ => 0,
// Always query S3 for the true object size. DB compressed_size can diverge from
// actual S3 object size (e.g. different gzip encodings across uploads with same
// content hash). Falls back to DB value if S3 HEAD fails.
let content_length = match state.s3storage.object_exists_with_size(&hash).await {
Ok(Some(s3_size)) => {
if compressed_size > 0 && s3_size != compressed_size as i64 {
error!(
"compressed_size mismatch for hash {}: db={} s3={}",
hash, compressed_size, s3_size
);
}
s3_size
}
_ => compressed_size as i64,
};

Ok(build_ft_file_response(
Expand Down
Loading