• You're correct network latency is a huge bottleneck, I should have mentioned it among the limitations on my problem statement. Then again network latency isn't the barrier for entry for consumer level distributed computing tasks.

    e.g. If we can provide the data over a network share, Dask, Ray etc. can provide better performance over single machine for data science related tasks by using the compute from the network for large data sets.

    But still these tools are accessible only certain sector of people(Developers, Scientific Computing etc.) and not truly consumer level distributed computed; Hence I mention it as a need-gap.

    But this might soon change with projects like gg  which addresses one exact use case from my problem statement - FFmpeg encoding over distributed compute. So if handbrake supports gg in future, All I need to do is link my AWS and get exponential improvement in performance in return.
  • Voted!
    Need karma! Please check submission guidelines.
    Paid!
    Why pay twice?