It seems that memray attach is not correctly tracking all new allocated memory.

[syc0000000]

I'm using memray attach inside a Kubernetes pod on a GPU machine to investigate a memory leak in a production service. My command is:

memray attach <pid> -o /path/pid.bin --compress-on-exit --native --trace-python-allocators --duration 1800 -f

This service is a pure CPU-based service launched via Python's multiprocessing module by a GPU service. Before the container started, the allocator was explicitly set to malloc. After collecting data for 1800 seconds, the resulting file is only 18.3 MB (compressed). When I generate a temporal flame graph with:

memray flamegraph pid.bin -o leak.html --temporal

the timeline appears incorrect:

The timeline appears to contain only a single valid data point, and the timeline itself is completely scrambled—even including timestamps from the future. Aside from the inaccurate timeline, the memory stack traces seem to be missing all native memory allocations entirely, and even Python allocations appear incomplete.

Even without the --temporal option, the statistics reported are clearly wrong—showing only a "Peak memory usage: 7.7 MB," whereas the actual memory usage(RSS) increased by at least 500 MB during that half-hour period.

Since this is a company project, I cannot share the original files. However, if you need more information, please let me know, and I can collect additional data. One noteworthy observation: when running memray attach, it takes about 5–20 seconds before the command returns, whereas on my local macOS machine, it returns almost instantly.

[godlygeek]

Without seeing a reproducer or the Memray report, I'm not sure that there's much I can do to help debug this. One potentially interesting thing to see would be what you get from:

memray parse pid.bin | egrep 'HEADER|MEMORY_RECORD' | head
memray parse pid.bin | egrep 'HEADER|MEMORY_RECORD' | tail

At least the "Resident size" line on that plot should be based entirely on those MEMORY_RECORD records. The time stamps on those lines (start_time and end_time on the header, time on the memory record) are milliseconds since epoch, so given a line like:

MEMORY_RECORD time=1765223769603 memory=32948224

that corresponds to

$ date -d "@1765223769.603"
Mon Dec  8 14:56:09 EST 2025

If you're only getting one memory record, or zero, something is definitely wrong; one should get written about once every 10ms. Likewise something is definitely wrong if you see a memory record with a time before the recorded start_time in the header, or after the recorded end_time (if the end time is non-zero, at least).

[godlygeek]

It's not easy to think of anything that could be going wrong here, honestly... There's a background thread periodically grabbing the current time and resident set size:

memray/src/memray/_memray/tracking_api.cpp

Lines 950 to 960 in 5756b60

	auto now = timeElapsed();
	size_t rss = getRSS();
	if (rss == 0) {
	std::cerr << "Failed to detect RSS, deactivating tracking" << std::endl;
	Tracker::deactivate();
	return false;
	}
	std::lock_guard<std::mutex> lock(*s_mutex);
	if (!d_writer->writeRecord(MemoryRecord{now, rss})) {
	std::cerr << "Failed to write output, deactivating tracking" << std::endl;

Interestingly, that fails if the RSS is computed as zero, without writing a record! So rss must have been non-zero right before that last record was written to the file.

The time is taken straight from the C++ standard library's clock:

memray/src/memray/_memray/tracking_api.cpp

Lines 906 to 912 in 5756b60

	uint64_t
	Tracker::BackgroundThread::timeElapsed()
	{
	std::chrono::milliseconds ms = std::chrono::duration_cast<std::chrono::milliseconds>(
	std::chrono::system_clock::now().time_since_epoch());
	return ms.count();
	}

The RSS is more involved (it's read from /proc/self/statm). Something could conceivably be going wrong there, but that wouldn't explain the weird time in the future, and it wouldn't explain how we could have gotten past the if (rss == 0) { check.

The RSS is then written directly to the file (as a protobuf-style varint). The time is massaged a little first - we subtract the start time in the header from it first.

memray/src/memray/_memray/record_writer.cpp

Lines 241 to 247 in 5756b60

	bool
	StreamingRecordWriter::writeRecord(const MemoryRecord& record)
	{
	auto token = static_cast<unsigned char>(RecordType::MEMORY_RECORD);
	return writeSimpleType(token) && writeVarint(record.rss)
	&& writeVarint(record.ms_since_epoch - d_stats.start_time) && d_sink->flush();
	}

For us to be able to get memory=0, it seems like either our varint encoding or decoding would need to be buggy, or for us to have somehow de-synchronized and begun to read things that aren't actually memory records as though they're memory records... If the latter were the case, I'd expect all of the other records being read at that point to also look nonsensical.

[godlygeek]

Huh. Actually, this seems weird:

>>> last_nonzero_memory = 1763315712
>>> last_time = 1770728849001
>>> start_time = 1768965348969
>>> time_delta = last_time - start_time
>>> print(last_nonzero_memory, time_delta, sep="\n")
1763315712
1763500032

Those numbers seem way too close for that to be a coincidence. Somehow I think we've wound up interpreting an rss as ms_since_epoch - start_time instead. But how... ☹️

[godlygeek]

Was this a one-time thing, or does it reproduce reliably for you?

[godlygeek]

Ooh, here's something else interesting:

$ start_time=1768965348969; last_valid_mem_record_time=1768965448698; end_time=1768967155818
$ for timestamp in $start_time $last_valid_mem_record_time $end_time; do date -d "@$((timestamp/1000))"; done
Tue Jan 20 22:15:48 EST 2026
Tue Jan 20 22:17:28 EST 2026
Tue Jan 20 22:45:55 EST 2026

You say you ran the report for 1800 seconds, which is 30 minutes. That matches the start_time and end_time in the header, but the last valid memory record has a timestamp that's less than 2 minutes into that tracking session! Something strange has definitely happened there...

[godlygeek]

I'm not really sure what else I can do here without a reproducer, tbh. Clearly something went badly wrong, but it's not obvious what. Maybe an off-by-one somewhere in the capture file growth management, or in the compression or decompression, or something like that... If this reproduces for you, the first thing I'd try is switching from --compress-at-exit to --no-compress to see if that changes anything. Hopefully not, but it would be good to confirm. If you can reproduce, it would be very helpful to try to isolate a minimal reproducible example that someone else can run. Or, if we can't get something that reproduces outside of your environment, we could also try giving you a copy of Memray built with tons of extra debugging to try to figure out where things are going awry. But until we can reproduce this, the odds of me being able to figure out exactly what went wrong are low, especially without access to the bad capture file.