| S | M | T | W | T | F | S |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| 13 | 14 | 15 | 16 | 17 | 18 | 19 |
| 20 | 21 | 22 | 23 | 24 | 25 | 26 |
| 27 | 28 | 29 | 30 | 31 | ||
Computer Scientist, Data Munger, Doctor (PhD), Math Geek, Bargain Hunter, Baker, Budding Photographer, Crafter, Domestic Diva Wannabe and 
March 23, 2016
A Less Than Productive Evening
As I was driving the kids home from daycare yesterday it suddenly dawned on me that a code change I had made a few weeks ago may have introduced a bug into some of my less popular apps. It wasn’t until I was drawing the kids their bath that I had a chance to whip out my phone and confirm the bug’s existence. Drat. At least I was pretty sure it was an easy bug to fix.
Once the kids were in bed I opened up my laptop to get started. This was looking like a minor bug in an set of apps with an embarrassing low number of visitors last week. I decided to chance editing it directly in production rather than setting up a test environment first. I made a single change, before getting called away. I was gone for a minute. Not more than 30 seconds. I returned back down stairs to discover the internet was out. I opened my phone to test the change I had made. Instead of a small bug that only affected an obscure case, my app was now completely unusable. Serves me right for working in production.
I checked in with our internet service provider. They already knew about the outage, and expected to have the internet restored by 2 am. It was 8pm. Not a good. Not good at all.
The internet wasn’t technically down, but experiencing about a 60-75% packet loss. Packets were bouncing around all over our provider’s network before getting out to the internet backbone. It looks like our service provider were experience a major hardware problem and were trying to reroute traffic around it, but the rest of the network couldn’t handle the extra load. What that meant for us was that we could occasional connect for a minute or two. It was usually just enough for me to connect through the web ftp, open a file and make a single change. If I was lucky, I could also test the change on my laptop instead of my phone before I lost the connection again.
By the way, my work environment? I’m working with a web based ftp and IDE. The web based ftp wasn’t anticipating connection interruptions. If the connection timed out while trying to commit a change, the web based ftp showed the change as committed. I discovered this after debugging the same bug and changing the same line the same way multiple times.
It was a night of not fun discoveries. Thankfully, the internet was stable enough at 11:30 for me to return my app to it’s mostly correct state. This morning I took a few extra minutes to squash that bug.
Next task: setting up a better working environment.
Comments
Trackbacks
-
[…] last months testing frustrations I wrote a simple deploy script for my javascript files. Now I can work on a shared library and the […]
By: Sarah K Tyler | A Smarter Working Environment on May 9, 2016 at 8:25 pm
I use and recommend WinSCP to manage my remote files and did so throughout my PhD. It handles disconnects and packet loss ok. And definitely tells me whether the file has been successfully copied.
By: Brian on March 31, 2016 at 8:42 am
Thanks! That’s actually the one I used at Santa Cruz as well 🙂 Loved it.
What I really need is a good IDE. The Web based ftp I used (pydio) is an integrated IDE & FTP. I’ve been using it because of the IDE functionality, but I’ve been increasingly frustrated with it. Have any recommendations?
By: Sarah on March 31, 2016 at 9:24 am
My IDE is just Notepad++. The problem with IDEs is that all of my source files, headers, compilers, libraries etc are on a remote machine. There is nothing that I can do locally except edit files.
By: Brian on March 31, 2016 at 11:00 am
I have the same problem, hence my tendency to short cut the process and edit in production.
I created a mirrored test site, but I have to manually push the files between the sites and update paths. I’ve been working on ways to have the paths update automatically. That helps.
By: Sarah on March 31, 2016 at 11:11 am
Write a “deploy” script that does the push for you?
I keep my files in git, so development is in my “production” environment, but that work is not visible until I push it to the server.
By: Brian on March 31, 2016 at 11:38 am