▲ ▼ Consumer level local distributed computing

posted by Abishek Muthian Ideator , 1338 days ago permalink , show insights

Current Smartphones, Tablets for most is more powerful(compute) than their old computers. At this point we should be able to distribute the computing load across all our capable devices. Distributed Computing is now limited to scientific research, data science, machine learning and I feel it's high time we use it for solving some consumer problems.

I use low power computers by choice for my climate goals, So I have several low power task specific computing devices. Consider able to compress 2TB folder consisting of disjoint files using the CPU of computer, smartphone, tablet etc. If Handbrake could use distributed computing then the encoding could be much faster. Network is an Achilles heel for distributed computing, but average network bandwidth in home networks have been growing steadily.

The manufacturers themselves are unlikely to create an interoperable environment, but perhaps applications which function in the user space could fulfill this need gap.

WHO NEEDS THIS

▲ ▼ bzlister , 1215 days ago permalink

I think you're underestimating network latency. If reading from the cache is like talking to a person face-to-face, then reading from disk is like sending a letter, and reading over the network is like sending a rocket to the moon :P

So for the example you gave, with 1 Gbps upload speed (pretty fast even by today's standards) then a 16Tb folder would take 16,000 seconds = almost 4 1/2 hours just to send to one other device. So unless the compression task would take over 9 hours and is completely parallel (no compression algorithm is) then its faster to just do it on one machine. You could send it over to more than one other device, but then the transmission time would increase (albeit non-linearly) and you'll see lower percentage gains in compute time due to Amdahl's Law.

That being said, solutions for what you're talking about do exist! See Hadoop/Apache Spark for a distributed, scalable, general-purpose mapreduce framework. I'm sure Azure/AWS have more modern solutions as well.

reply I WILL PAY FOR THAT
HOW MANY WILL PAY FOR THIS
▲ ▼ Abishek Muthian Ideator , 1215 days ago permalink

You're correct network latency is a huge bottleneck, I should have mentioned it among the limitations on my problem statement. Then again network latency isn't the barrier for entry for consumer level distributed computing tasks.

e.g. If we can provide the data over a network share, Dask, Ray etc. can provide better performance over single machine for data science related tasks by using the compute from the network for large data sets.

But still these tools are accessible only certain sector of people(Developers, Scientific Computing etc.) and not truly consumer level distributed computed; Hence I mention it as a need-gap.

But this might soon change with projects like gg which addresses one exact use case from my problem statement - FFmpeg encoding over distributed compute. So if handbrake supports gg in future, All I need to do is link my AWS and get exponential improvement in performance in return.

reply I WILL PAY FOR THAT
HOW MANY WILL PAY FOR THIS
▲ ▼ Abishek Muthian Ideator , 1338 days ago permalink

RentMyCPU - https://github.com/franklbt/RentMyCPU seems to run webassembly files across devices to achieve distributed computing and the associated commercial service seems to add incentives to those who lend their compute power.

I haven't tried this yet, But I'm eager to do so soon.

reply I WILL PAY FOR THAT
HOW MANY WILL PAY FOR THIS