Data Sharding

[Max-Meldrum]

The following are some aspects we need to address/implement to enable data parallelism:

• KeyHasher (e.g., Murmur3)
• Range vs. Hash Sharding
• KeyExtractor
• Keyed Streams (KeyedStream, KeyedArrowStream)
• Region & Shards

[segeljakt]

One question, should it be possible for operators to be generic over KeyedStream<K, T> and Stream<T>? For example, the map operator in:

let stream0: Stream<_> = ...;
let stream1: Stream<_> = stream.map(...);

let stream2: KeyedStream<_, _> = ...;
let stream3: KeyedStream<_, _> = stream.map(...);

If not, we need special operator implementations to handle each kind of stream. In Flink they use subtyping. Although Rust does not have it, we could maybe use traits to achieve something similar:

trait Stream<T> { ... }
trait KeyedStream<K, T>: Stream<T> { ... }

[Max-Meldrum]

One question, should it be possible for operators to be generic over KeyedStream<K, T> and Stream<T>? For example, the map operator in:

let stream0: Stream<_> = ...;
let stream1: Stream<_> = stream.map(...);

let stream2: KeyedStream<_, _> = ...;
let stream3: KeyedStream<_, _> = stream.map(...);

If not, we need special operator implementations to handle each kind of stream. In Flink they use subtyping. Although Rust does not have it, we could maybe use traits to achieve something similar:

trait Stream<T> { ... }
trait KeyedStream<K, T>: Stream<T> { ... }

Would be nice to have the same map on both. Yeah, may need to use traits in that way. Something to discuss tomorrow 😄