This year, despite generally taking precautions and having two shots of Moderna, Christmas found me in quarantine in my home office, hiding away from my family. As far as Christmas traditions go, “physical isolation” has to be one of the more unusual ways to celebrate a holiday whose traditions frequently involve getting together in larger groups.
It was Christmas day and I was alone, but it didn’t really feel that way: my wife, daughter, and I spent Christmas lunch (and supper) talking via FaceTime.
Over the last two posts I explored the process of extracting a very large dataset (an XML export of Wikipedia) into various types of databases (an RDBMS1 and a Graph database) that I can run locally as a precaution against being marooned in the past should some portal, machine, or ancient artifact that send me careening backwards in history. While these two databases have satisfied my need to quickly look up information and find relational paths between said pieces of information, they still leave me needing to know exactly what I’m looking for.
In April, I put together a Python script that would load an XML export from Wikipedia into a local SQLite database because, once again, you never know when you’ll inadvertently become a time traveller who needs knowledge of future events to return home. (Look, it’s more interesting than just saying “because I find it amusing”, so just roll with me here.)
This script satisfied the typical time traveler’s use-case of “finding a bit of topical information quickly”, yes, but that is only one of the many use-cases that chronological explorers must concern themselves with before traipsing around through spacetime.
The code for this post can be found on my GitLab
While the last two years have realized a number of risks that we had all left unmitigated for far too long, one even greater risk has yet to be realized: the risk of unscheduled time travel. Yes, the grave concern that must be considered is that one moment you might be hanging out in the mall parking lot and the next you might find yourself delicately navigating 1955, aware that your only hope for return is tied to some esoteric piece of knowledge such as “when will the clock tower be struck by lightning”?
In the previous post, I talked about much of the reason I quit writing posts was because of the painful workflow surrounding the whole deployment process. To top it all off, when I finally tried to automate it, I found out that the Jekyll theme that I was using had used had many unpinned dependencies: rather than specifying the version of each library that should be retrieved, the author of the previous theme had chosen to simply request the latest version available at any given time.
Judging by my prolonged absence from posting new content, it could safely be assumed that I’ve spent the last two years encased in block of carbonite. If that idea doesn’t resonate with you, you might simply assume that life got busy. While one of those options is statistically more likely than the other, neither will take home any points for correctness. The truth of the matter is that I’ve avoided posting new content because posting new content is hard.
Just over two years ago, I wrote a little version of Snake for a 4x4x4 LED cube. Since then I’ve picked up several little Arduino devices, include a 6x6x6 LED cube, an Arduino Esplora, and the one I’m most excited about, an Arduboy. For every Arduino I bought, I had some grand plan. I never followed through on any of them though. It wasn’t because of the technical challenge of writing an executable less than 32k in size.
I have a love-hate relationship with the terrain from Minecraft. One the one hand, it’s the terrain that makes every world kind of memorable in it’s own way. It comes at cost though; the process of generating terrain can cripple a server.
That’s where Chunky comes in. Chunky lets you pregenerate a world so that our blocky wanderers don’t cause the server logs to fill with “CAN’T KEEP UP!”
Okay. Why?
The Worst Thing Ever I know the worst code that I’ve ever professionally written. It was an overly generic UI component. It managed to combine a total misunderstanding of MVC, heavy constructors, an overly broad interface that leaked abstraction like a sieve, empty methods, liberal use of Object, and was driven by unexceptional-exceptions. Oh, and it couldn’t be resized.
Not that any of that mattered. Despite all of the odds against it, it worked quite well from the user’s perspective.
Today I turn 31. Before the day is through, I’ll have ended up celebrating with family and coworkers. And Bash, because as it turns out, Bash can be a bit festive.
I find most default Bash prompts to be rather boring. Sure, they vary from platform to platform, but they all seem a little too lite on info that I care about. A few years ago I decided to liven it up a bit.
Lately I’ve been exploring the world of Clojure. I’m not sure whether it was the siren song of functional programming that started driving me, the simplicity of Lisp, or my curiosity about this language that was being promoted by the likes of Uncle Bob. Whatever the cause, I needed to explore this strange world head first. I needed to try out this language for myself.
I spent much of time just dipping my toes in and writing little toy programs here and there.
I wrote a song this morning. I feel really good about it.
I tend to record music, frown a little and then hide it far away from human ears. It isn’t that I don’t like it. I just don’t want anyone else to realize how amateur it is and that I, by extension, must be an amateur. A fake. An imposter.
So, I’m doing the imposter syndrome post? About time, I suppose.
My desk has a new ornament on it: an Arduino powered lamp. And why does that merit a blog post? Because I built it.
After combining the Rainbowduino kit with some 600-ish points of solder, I found myself ruler of a 16MHz processor, 30Kb of instruction storage, 2Kb of data storage, and 64 tricolor pixels.
And what does one do with 64 pixels, no inputs, and processing power reminiscent of a vintage entertainment console?
Roughly a year ago I posted what I expected to be the last entry in my Fiddler series. Each of the seven entries had focused on various ways that Fiddler could help improve our development life cycle. They represented the highlight reel.
All isn’t perfect though, and where there is a highlight reel, there tends to be outtakes as well. So, here goes: here’s a few ways you can use Fiddler to slow down your development process.
Just over a year ago we went over how to setup a reverse proxy that would require our users to authenticate. In our original proxy article, we generated some NGinx configuration that looked a bit like…
location / { proxy_pass http://jira; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } And it worked. And everyone was happy. Until the day the request came in that NGinx should add a fixed set of credentials to all upstream web requests.
A while back, I announced that I had moved my blog over to HTTPS. What about HTTP though? Do I serve content over HTTP still, or did I doom existing links to be forever broken?
As you could guess by the title of this post, neither of those was satisfactory. Instead, I added four lines to our NGinx configuration telling it to listen on port 80 and respond to every request with a permanent redirect pointing to the same path under https://joshuarogers.
From time to time I hear a certain question floating around through the office, along the lines of “Why won’t .NET let me pass my List<Thing> as a List<IThing>?”
The tone of the question usually suggests that someone is annoyed that .NET isn’t smart enough to make such a simple inference. The reality though is that the compiler is smart enough to know that it can’t.
You’re kidding, right? If Thing is an IThing, then all the Thing in a List<Thing> must also be IThing.
Back in May we looked at how to setup Access Controls with NGinx. I didn’t mention how I test it though. In order to test the different rules, the requests needed to originate from different IP ranges, which in turn meant that I would need to send the requests from different networks. As I find the idea of fidgeting with cables or hotspots to be more than slightly annoying, I opted to tunnel request out of my internal network using a SOCKS proxy.
My server had an unauthorized login attempt today. Really, it was more along the lines of 2,000 login attempts. Why is my server under attack? Simply because it is on the internet. See, this is actually a normal day for a small server. Last week, there were just over 12,000 attempts.
So, business as usual for a server looks something like walking through a zombie filled wasteland armed only with a cowbell and a belt made of bacon.
One of the things that I love about living in a Unix environment is that 99% of the time, remote administration only means a quick trip to the console. That’s 99% of the time where my desktop can look something like this:
Of course, 99% doesn’t quite make a whole. There’s the matter of the other 1% of the time where a GUI is required. Earlier on in my journey with Linux, I would have tried to setup a VNC server or to work around the need for a UI.
This weekend I spent some time migrating my Minecraft server a good 700 miles closer to my house. The new server is ~30 milliseconds closer. If you play games, you probably know how huge that can be.
So, how did I move data to the new server? FTP, right? No. Definitely no.
Why not FTP? FTP is plain-text Unless you’re using something like FTPS, you’ll be sending your server credentials across the wire in plain text, so anyone sitting between you and your new server has all the info they need to login.
A little while back I spent some time looking at an interesting issue with a colleage. He was trying to load a virtual machine from an .OVA but kept recieving error messages as he loaded it that the file was invalid. Somehow though, other people used the same file. This one could be fun to diagnose.
To start with, the premise of our argument has a problem: it wasn’t the same file but rather a copy of the file.
In the first six parts of our series, we examined some of the power that Fiddler has to offer us: we covered remote debugging, composing requests, breakpoints, autoresponders, plugins, and a bit of FiddlerScript. So what’s next? There’s still an incredible amount that Fiddler can do, but despite the “Land Before Time”-esque post names, this was never meant to be a comprehensive Fiddler feature tutorial. The series was just meant to cover some of Fiddler’s most powerful yet underused features; specifically features that have gotten me out of a bind a time or two.
Our tour of Fiddler has covered a lot of ground so far, and while there’s still more to cover, I’m going to have to let you in on a secret before we can continue. The UI is definitely powerful, but here’s the thing: it’s not the killer feature. The real killer feature is FiddlerScript.
Getting Started Fiddler already detects invalid requests and broken responses. What more could we possibly add, you might ask.
The more time I spend examining Fiddler, the more and more impressed I am. Not only is it feature rich out of the box, but it includes a number of mechanism to easily extend it. This week we’re going to focus on the easiest way to add new functionality: installing new plugins.
All of the following plugins are available at Telerik.com. I highly recommend installing them.
JavaScript Formatter The first plugin, and my personal favorite, was introduced to me a few years ago by Joey Robichaud.
Over the last few years I’ve grown rather accustomed to listening to my Pandora while working, even upgrading to PandoraOne so that I can enjoy my music without interruptions 1. However, I found myself avoiding Pandora earlier this year. Why? Pandora uses Flash for the web and Air for the desktop, presumably in order to annihilate the battery on my laptop. 2
The problem isn’t Pandora though, it’s Flash. What if I could use Pandora without Flash?
Last week we figured out how to use Fiddler’s breakpoints to let us edit traffic at will. While it is a powerful tool, if we are making the same edits each time, it can become as tedious as it is useful. And with that rather painful segue, we get to the AutoResponder.
Fiddler’s AutoResponder is represented by yet another tab hiding away in our detail view. When enabled, it can, among other things, automatically return a response for a given request.
One of the great things about using an API is that it implies that I can use functionality that I don’t have to write myself. I can write an app that live tweets driving directions between two random Yelp reviews without ever having to directly know how the tweeting or directing is taking place. As long as I stick to the API, I can pretty much accept that the API’s functionality should usually “just work”.
Build servers and source control are like the peanut-butter and jelly of devops; they just go together. That would make us suspect that it shouldn’t be that big of a deal to setup push notifications to trigger builds, right?
My GitLab instance shows that it will gladly trigger a build via web hook when it receives a new commit. My Bamboo shows that it will just as cheerfully scan for changes when it gets a web request.
It’s not Monday, so why am I posting something new? Just a quick announcement: I’ve migrated joshuarogers.net from http to https. http://joshuarogers.net now just returns a 302 Redirect to the appropriate https page.
Last week we started looking at the web proxy Fiddler, or the “Fiddler Web Debugger” as it unfortunately bills itself. While the name is accurate, I can’t help but feel that it is a bit of an understatement, much like calling the Hulk “somewhat tempermental”1, or MacGyver “good at assembling things”. Sure, Fiddler is a debugger, but it’s less of a simple tool and more the web equivalent of a swiss army knife that contains an entire Home Depot2.
If you’ve done any amount of web development on Windows, you’ve probably used a program called “Fiddler” a few times. Fiddler is a small proxy server that allows you to see all of the requests made by your system. Sadly, for most people, I’d guess that that is all that Fiddler is. It’s a shame because Fiddler offers up so much more power than that.
So, let’s fix this. For the next several weeks, we’re going to explore some of the under noticed functionality that Fiddler provides us.
Introduction The concept of dependencies seems rather straight forward. If “Thing A” depends on “Thing B”, then we can understand that we can’t possibly use “Thing A” until we have “Thing B”. It’s not even a development principle really, it’s more of an “even small kids realize this” kind of principle. It’s sort of a universal given, unless you happen to be Gallifreyan.
Taking that a step further, if you want me to make an application for you using some web services you’ve published, it might be prudent to give me access to those services, right?
If you’ve spent any time looking at the items that I post, you’ve probably gathered that I’m a developer, but that I have a rather strange habit of not talking about code. I talk about build automation, source control, compiler output, virtualization, proxy servers, and continuous integration, but I barely talk about matters of code. So if I love coding, why do I spend my time talking talking about DevOps rather than code?
The drunken scribblings of a mad man. Possibly some kind of ancient cuneiform. A random page out of the Middle Earth phone directory. Whatever it was, it was the server password, not that it matters now: the little Post-It note reminder had long since thrown itself into the waste paper bin. Somewhere a landfill sat, knowing our password and chuckling away at our misfortune.
It seems that our hero is in a bit of trouble, if only slightly self-inflicted.
If you’ve spent any amount of time running a Linux machine on a Windows network, you’ve probably had the need to share files between the two. This seems like it should be a trivial matter, but as often as not, it seems to produce bouts of sheer insanity. The problem usually lies in the realization that Windows uses SMB and that Linux tends toward transit over SSH (rsync, scp, sftp.)
The universe is most surprisingly finite, it seems. Technology even more so. A few posts ago, I lamented that n IP addresses always need to be split n+1 ways. If it is true with IP addresses, it’s twice as true with hardware. Given three servers, there will be demand for, at minimum, five. Certainly there must be a way for us to sneak out of this problem too, right?
We could always just install multiple sets of services on the same bare metal.
Alright Internet, we need to talk.
You’ve grown up quite a bit over the last several years, but I’ve never taken the time to have thetalk with you. You know the one. The safe passwords talk.
Your requirements for a minimum length, and a diversity of characters in passwords made many classes of brute force attack more difficult. You’ve gotten a bit carried away though, it seems. You’re actually making the internet a more untrustable place.
In the last few posts we’ve gone over how to build and secure a reverse proxy. While this is a great option if you want to add extra access controls, rewrite urls, or hide multiple servers behind an IP, sometimes it is just a bit of overkill. Sometimes all that is needed is to change the port that a service listens on.
Let’s take Atlassian Confluence for example. By default it serves up pages on port 8090.
Last week, we went over how to quickly build a reverse proxy using NGinx. While this solved our immediate problem of hiding n+1 servers behind our n ip addresses, it could still stand a bit of work. As it stands right now, we don’t have any access control capability.
The Setup Let’s assume we have two sites that need to be proxied: public.example.com and internal.example.com1 and 2.
upstream internal { server 10.
Our universe is comprised of a seemingly infinite number of rules ranging from little tidbits like magnetism and inertia, to the unchangable truths of buyer’s remorse and Steam downtime occurring on a weekend. Still, there is another rule that seems to be more faithful than gravity itself: given ‘n’ public facing IP addresses, you will receive ‘n+1’ requests for their allocation. Six IPs? I’ll see your six IPs and raise you seven servers.
If you’ve ever setup a web-app on a Linux server, chances are that you’ve setup a MySQL server at least once. On the off chance that you haven’t, we’ll do a quick crash course now:
apt-get install mysql-server It might seem like I’m being a bit sarcastic. I am, though not by much. I suspect that most of the instances that I’ve seem running have had no configuration past what comes stock.
The Setup Thirty-eight megabytes. Over time our application had somehow grown to 38 megabytes. The road to that point made sense: features needed to be added, and with features came new libraries each with a few dependencies. Still, somehow the path of good intentions had led us to an application that weighed in at a colossal 38 megabytes. The would have actually been a reasonable size if it weren’t for one slight detail: it was a web app.
Just a bit of a fun puzzle today. Take a moment to look at the following two classes. Given an architecture where an int is four bytes and a character is one, what is the size of an instance of each class?
struct Alpha { int x; char ch1; int y; char ch2; } a1; struct Beta { int x; int y; char ch_name[2]; } b2; The most obvious answer would be 4+1+4+1, giving us 10 bytes for the first class, and 4+4+1+1 giving us another 10 bytes with our second class.
Had an interesting question posed to me the other day from David Ruttka: Did I have a favorite way to list all of the directories that have been deleted from a folder in git? Admittedly, this took a bit of thought. No arcane git command came to mind. Nothing did. Google wasn’t that much help either. Not only did I not have a favorite way, I didn’t have a way at all!
If you run a Linux server alongside Windows servers long enough, you’ll eventually have the need (or request) to add that machine to a Windows domain. Thankfully, it’s a rather easy process. Assuming you run Ubuntu, you can simply run the following commands, substituting the domain and a domain admin in place of EXAMPLE.COM and jsmith, respectively.
sudo apt-get install likewise-open sudo domainjoin-cli join EXAMPLE.COM jsmith sudo lwconfig AssumeDefaultDomain true Out of the above lines, the first two are probably self-explanatory, but the third is likely a bit more opaque.
Just wanted to share a quick MCEdit filter that I put together. Given a selection, this replaces the top layer of the perimeter circle with glowstone. On our server we use this to mark off private land borders so that people don’t accidentally interfere with one another or set up camp on top of one another. Feel free to use it as you like.
from pymclevel.materials import alphaMaterials import math displayName = "Player Boundary" inputs = ( ) replacableblocks = [ alphaMaterials.
Software is constantly changing. Bugs are fixed, features are added, performance is increased, and finally binaries are built. With this constant cycle of change come people wanting to know exactly what the changes are. Between major versions, and minor versions, we normally have release notes to give us this information. However, between release builds the list of items is in constant flux with no let up in the people wanting to know what the changes are.
I’m a huge fan of Git. Used properly, Git’s branching system can be immensely powerful. In fact, it forms a core part of our engineering workflow at the office. By providing a healthy degree of isolation between different changes to the codebase, we can provide QA with a clean way to test various change sets in total isolation before we produce a release candidate.
This sounds rather nice until the realization that after we finally get a release candidate that we accept, somebody is left we the rather tedious job of removing all of the source branches from our repo.
Just a quick Ruby tip today. If you use the Paperclip gem for image attachment with Ruby, you might have noticed the option to use a hash as part of your URL. Using this provides a simple way of creating a secret URL for your images. Rather than an easily predictable URL, a section of the URL created by hashing a secret key with a row from the database. For instance, the following code you might get a URL like /avatars/square/c2794196caf44b11b0b7b70cfd12ab88.
In Linux you can make or restore an image of a drive with the dd command. For instance, creating a disc iso of a DVD could be done by specify the block device for the dvd-rom as the input file and a new file for the output file.
dd if=/dev/sr0 of=/home/jdoe/cdrom.iso Another frequent use would be to make a bit-for-bit copy of a disk as a backup.
dd if=/dev/sdb of=/dev/sdc What happens if you need to make a bit-for-bit copy of a drive to a remote machine though?
In my first year of college, there was a festival of sorts on campus. It was a day designed to provide the college with goodwill and easy photo ops for the pamphlets they would use to lure freshmen the next year. Though only a side-effect in the sight of those organizing the event, it managed to provide those of us who would normally spend the entire day in class with a chance to stop and imagine what it would be like to relax.
A fun idea came to me a while back: using x86 assembly, could I implement a basic if/else statement that runs arbitrary code without using any conditional instructions? If you care to try out the challenge, I’d recommend using a language that provides low level memory access and then taking a quick break from reading further.
Determining Inequality The first part of the problem that we need to consider is this: is there any way to tell that whether two values are equal without using the conditional operator?
For anyone that attended my talk “Moving from Manual to Automated Builds” at Tech Mixer U, I’m posted the slides with extra notes and our full demo ant script.
On any long running project of consequence, refactoring will happen. While usually this is a good thing, it can sometimes lead to the case where all references to a file are removed, but the file itself still sits in the codebase. Though not technically a problem in itself, this can cause a bit of unnecessarily bloat and confusion, not to mention unnecessary time spent updating these orphaned files to support future refactoring efforts.
Refresher A few months ago, I started writing about my personal requirementsfor simplifying the process of using testing feature branches at work. I’d made quite a bit of headway, but never got around to posting an update. Thankfully, DavidRuttkadecided to remind me that the article needed some resolution. 1
In order to get everyone to buy into the concept of feature branches, the process of being able to test a feature branch needed to provide next to no overhead.
If you really care about your sanity and the sanity of those around you, you probably start most projects by creating a build script of some description. 1 If you’ve compiled Java or .NET, or if you read the Ant primer then you’re probably familiar with writing an XML based build script. Adobe Flex application can be built in the same way, with just a few steps.
Prerequisites Basic Compilation We’ll start by creating a very basic build.
Over time, I’ve started noticing a trend: as humans we have a great aversion to pain. Okay, that doesn’t really make me a master of the human mind, but it’s still a starting point. Hearing this, an appropriate response would be suprise: how could it have possibly taken that long for me to come to that realization? I suppose the real issue isn’t that people don’t like discomfort, but that I’m starting to see the things that people are willing to sacrifice just so they don’t have to endure a little overhead now and then.
Assuming you read the last post, you should have a basic understanding of projects, targets and tasks. In this post, some different tasks, and try to make a slightly more practical ant script. Suppose our build requires a little bit more complexity than displaying “Hello World.” In this case, it has to meet the following requirements:
The file HelloWorld.java must be compiled HelloWorld.class should include debug symbols. HelloWorld.class must be zipped Let’s start by creating a basic project skeleton.
To me, some of the most fascinating problems are the ones that are only visible in hindsight. The angled measuring cup might be the best example of this. With a standard measuring cup, you have to bend over or lift the cup to see how much of a substance you’ve measured. On the other hand, the angled measuring cup can be read while standing upright. Yet when asked, most people didn’t realize the problem with a normal measuring cup until after having seen the one from Oxo.
My wife and I are paying off student loans. Obviously we didn’t just take on debt because we wanted to. Going into college, she wanted to be a teacher and I wanted to be a software engineer. The only practical way for either of us to get the job that we wanted was for us to get the required degrees. That meant student loans. However, we took them on with the belief that they would be an investment that would pay for themselves in time.
A few weeks ago, a few of us from work went to GiveCamp. Hampton Road Studios used footage from the weekend to make a promo trailer. The weekend was awesome. Take a look at the embedded promo trailer, or visit givecamp.org to learn more.
Give Camp Birmingham | Promo from Hampton Road Studios on Vimeo.
I usually listen to Pandora at work. I’d rather listen to my personal library, but Pandora makes for a good second place. There’s no question that it beats trying to store / sync untold gigabytes worth of audio on my work computer. Still, sometimes Pandora just doesn’t cut it: especially when I want to hear a certain song. So, I decided to use lunch to figure out how to get my music collection into iTunes at work.