Yeah, choosing an arbitrary time to start a task sounds like it might not help with getting distracted while doing it.
Is focus drift significant enough for you to be interested in writing down how focused you were and what other things you did recently and looking for correlations that can be tested as possible causes?
Here are some specific factors that could be interesting:
1. Consistent, sufficient sleep
2. Good food
3. Prioritizing harder tasks when you're more energetic, possibly the morning
4. Having a bunch of reminders about how you previously said that the task you're working on is important
5. Recording success or failure (and time spent) in a log where it will be remembered forever
6. A list of all the other things you want to get done after this task