First, you have to track what you're doing.
- Trello: Kanban board for TODO lists (TODO, Doing, Done)
- Pomodoro: A timer to break my day into 25-minute segments of INTENSE focus
I've lost data too many times. I have three rules of thumb:
- Automate everything
- Check in all source files into Github
- Save all derived files to Amazon S3
When I automate something, I write either a Bash, Groovy, or R script for it, depending on the task and domain. System-level things go into Bash. Java-y things go into Groovy, but call Java libraries for heavy lifting. Stats things go into R.
In the case of Groovy code, I have to "include" whatever Java libraries I want to call. So when I'm ready to call a script, I wrap it in a Gradle task which processes args and calls the Groovy script with the right classpath.
In addition to the Gradle -> Groovy -> Java stack of tools, I also write scripts in R for statistical processing.
When I want to call a Gradle or R script (or a sequence of them in combination), I use Jenkins. I define Jenkins jobs which call Gradle and R.
For persistence across jobs, I use MongoDB and Amazon S3.
My Jenkins "slave" machines (their term) need Gradle and R installed; and sometimes, I find that I need to run a bunch of similar tasks in parallel. That's why I did this, to bring up my Jenkins master and slaves (as well as a Nexus Java library artifact repository and a MongoDB instance) within Docker containers but still persist important data. My latest Docker scripts for this infrastructure are public on Github.