The Story of How I Used Technical Financing to My Advantage.

In my last blog post, Intro to Technical Financing, I introduced 5 concepts that helped me come up with technical financing. In that blog post I spoke about how I was able to leverage technical financing to help me complete a 1 year job in 3 months. This blog post will explain how I did that by giving by utilizing the beneficial idea that is technical financing. To reintroduce the concepts and further explain them a bit, they are:

1. Interest rates for technical debt is not equal.
   • Like it sounds, some tech debt has different interest rates than others. This of course depends on the tech debt you’re taking out, but the fact simply is, interest rates vary.
2. There is such a thing as having technical capital.
   • This one is very self explanatory, if there is such a thing as technical debt, then there should be technical capital, technical interest rates, and technical financing as well.
3. You can refinance your tech debt.
4. You can intentionally take on tech debt to raise your immediate technical capital which can be paid back in full at a later time.
5. There are technical loans you can take that have a higher ROI than the interest payments. Therefore you can intentionally take on tech debt to raise your overall technical capital.

While this blog post is able to be read on its own without any prior knowledge, it is a part of our Lessons in Factorio series, so if this is your first time reading one of this series’ blog posts, I recommend starting from the first one and working your way through them. The first post can be found here: Why I Learned from Factorio: Lean Networking

Assessing Technical Capital vs Technical Cost

So let me start by telling you my awful situation that forced me to grow and come up with this crazy idea. I’m hoping to kinda draw you in with this cool concept and how I implemented it, then like some magician, I’ll show you how I did it behind the scenes afterwards. I’m planning on revealing all the tricks later, along with how I learned them and how they work. So, like skip ahead to those if you wanna actually learn something. Otherwise, sit back and enjoy my tale of how I delivered a 1 year project in 3 months.

The situation:

So I just started this job maybe like 3 months ago at this point. I’m barely getting comfortable with the environment, well as comfortable as you could. I literally had to take a break and think about my life decisions when I saw that there was like 10 subnets on VLAN 0 (this was Juniper). That’s a story for another day though.

Anyways, there’s talk about closing down the headquarters and moving everything to another state. My manager and I were laughing about it because that’ll obviously never happen. The entire data center was sitting behind the IT office. There’s no way in hell that stuff is moving, especially with the boatload of tech debt that was sitting in there that I’ve barely made the likeness of a dent in.

Then, it happens. We get a company-wide newsletter where the CEO said we’re moving out of HQ and closing it down by mid next month.

WHAT?! The data center is right behind me. How the heck are we supposed to do this?! I asked my manager about it and this was news to him too. He told me to chill out and that we’ll probably get more time.

Fast forward, we’ve already gone through the 7 stages of grief and completely failed at bargaining. We got 3 months to get all of our stuff out. I told my manager, there’s absolutely no way we’re doing all of this in 3 months. It’s just me and him and the data center. He says we just need to keep moving forward so I start moving along in my one-man project death march. (That’s a term for when you know you’re on a failed project, but everyone on the team has to move forward anyway knowing you’re all getting fired along the way). Whatever, here we go!

Taking Out a Technical Loan

It took me a whole week to cutover 1 server. That’s actually not bad all things considered. Unfortunately, I know how to do math and we had 3 months, which is 12 weeks. I had more than 12 servers. That’s a problem. I told my manager this and I was like bro, it takes 1 week to do it, right? He said, can’t I do it faster? And I was all, well yeah, if you want me to do it poorly. Then I realized, yeah! 80/20 rules. I could do about 80% of the work in 20% of the time. I was doing 1 week’s worth of work because I wanted a 99% success rate on the cutover. But, I bet I could do it in like 1 day or less if he was cool with like 60-80% success rate.

His response: You can’t make an omelet without breaking a few eggs, lol.

Making Sure My Tech Loans Were Low-Interest

I did a lot of things awfully on purpose just to get things done faster. The magic part though was that I documented everything I did poorly and what I needed to fix later. It was like a tech debt IOU (this was also an apology to future-me).

One example of this: I needed 2 data centers (DC). I only had 1 set of firewalls. Not usually advised but I took out a small technical loan to speed up our DC project. I broke High Availability (HA) and used one half for the new DC lol.

This allowed me to start moving stuff over right away. There was some interest but not much. I didn’t suffer any outages as a result of my risk-taking. Building HA while the DC was live was a bit scary, and it took me some time to do it, but the ROI was definitely worth.

We did a lot of stuff like that. We ended up closing our old data center a few weeks ahead of schedule, which was good because the L1 cleanup took forever. Once we were done, we had some time to focus on just clearing out our tech debt. Here’s a list of some of our other shenanigans:

We skipped steps that we knew we should do, but could get away with avoiding doing.

• Documented them for later.
• Used NAT to band-aid a lot of stuff.
• During discovery for the cutover, we uncovered more issues we should resolve.
• We ignored them lol.
• Didn’t wait for another internet circuit. Data center was running of a single internet circuit only.

Technical Refinancing

We spent the next 9 months documenting everything as we should and cleaning out all of our tech debt IOU’s. It was a lot easier to do it without the business breathing down our necks, and they didn’t care because they got their DC already.

This was a long story. I don’t want my article to get too long, but this is a very key strategy that is worth learning. A quick real life example of technical refinancing in the network engineering world:

Sometimes there are legacy systems that are so old and the interest payments are so bad, it makes sense to simply stop paying any of the interest payments and let the tech debt pile up. The way we eventually eliminated this tech debt was to refinance it by replacing the system with something brand new built from scratch, and nuking the old one from orbit, thus eliminating all that piled up debt.

• DHCP was funky – just a complete mess. It had one VLAN, multiple subnets, needed a MAC address based DHCP reservation to move people between subnets.
  • Easier to just create a new subnet with its own VLAN and move everyone over instead of dealing with that mess and spend time doing other things instead of fixing all of that jumbled up mess.

Interest Rates For Technical Debt Is Not Equal, Factorio Edition

Like I said above, tech debt has different interest rates. There is some tech debt that has an ROI (Return on Investment) that is greater than what the interest payment would be because there were very little shortcuts taken in the first place, therefore netting a profit making paying off the full tech debt not valuable in the long run. In our Factorio playthrough these were the conditions that I had determined taking the tech debt made sense:

• Low interest rate tech debt.
• Red science doesn’t really need to be built to a high scale. A few assemblers go a really long way with red science.
• I learned I could ignore doing red science well, and I wouldn’t really need to pay for the repercussions of doing it poorly until much later in the game.

Technical Financing, Factorio Edition

Okay so here is the Factorio example, if you don’t care about Factorio, then you can skip straight to the conclusion at the bottom, but you will miss out on a few very cool examples of Technical Financing in use that you could implement in your real world applications!

T Minus 8 Hours

So my friend really wanted to get the There is No Spoon achievement in Factorio.

Getting this achievement is quite the accomplishment because it requires you launching the rocket (beating the game) in 8 hours. I think my first play through took me at least 60 hours of playtime to beat the game. I had done a practice run alone and it took me 25 hours to beat the game. Even with another person or two helping, we’ll need to take some drastic measures to get significantly better results.

I’ve picked up and tested a lot of technical financing ideas in a lot of previous runs, but this run being especially challenging really forced me to pull out all the tricks out of my hat and really think about how to do them. Spoiler: it took us 11 hours and we got the achievement where you beat it within 15 hours. We’ve since ran a post-mortem and are planning on doing another run to shoot for the 8 hour mark. I’m hoping telling you about this run and how I leveraged my tech financing tricks will be fun and you get to learn along the way.

Prep work

We broke out the run into four 2-hour sections. Since this is also my partner and we’re business nerds, we referred to each section as a quarter as if it were a business lol.

• Quarter 1 – 2 hours in.
  • Blue science and red belts.
• Quarter 2 – 4 hours in.
  • Logistics and solar + capacitors.
• Quarter 3 – 6 hours in.
  • First rocket part built.
• Quarter 4 – 8 hours in.
  • Rocket Launched.

Even looking at this list, it seems ridiculously demanding. This means we need yellow science within 4 hours. I’m not sure how we’re supposed to do it that quickly, but that’s about halfway and that’s what needs to happen if we’re going to launch within 8 hours.

We also recruited another person (Adamantium) to help us with this run. He’s played Factorio quite a bit as well. Since we had a new engineer working in our factory, we also needed to onboard him the same way we would onboard a new engineer at work.

Onboarding

In hindsight, we should have done this prior to the run, but at this point, we’ve spent more time talking about this run and planning it than doing the actual run. We figured we’d at least get some experience under our belt and we’d just try again if we don’t make it on this run. The likelihood of us doing it on the first try seemed low and it would be more fun to play more Factorio rather than less Factorio, anyway.

Anyway, to onboarding. So we had to spend some time setting some ground rules with Adamantium in our Factory. We showed him a few of our standards so that he could build in our environment without ruining everything and disrupting our flow. Later on, we realized we needed to assign tasks to him as well for him to handle so that he wouldn’t feel like he’s stepping on our toes. I didn’t think this would be necessary because I’ve been playing Factorio long enough with my business partner (NotaFish), we already have a rehearsed plan and communication strategy. Staying in sync and on track isn’t usually a problem for us, but adding a third person into the mix made us adapt our strategy a bit.

Some of the standards we had were:

• General building design
  • Transportation of materials goes east/west.
  • Building of components goes north/south.
  • Try to leave extra room for scalability.
    • More than you think you need, because it’s never enough.
• Trains
  • Naming convention for stops.
  • Always 1 engine followed by 3 wagons. Unidirectional.
  • Unidirectional rails. One for each direction.
  • Loading/unloading always looks the same.
  • Only 1 material per wagon.
    • In most cases, 1 material per train.
  • All the rails connect to the same rail network.

Starting the Run – Quarter 1

Interestingly, we actually chose to ignore our own standards for red science. The cost of creating the materials at that stage of the game is so high, it’s faster to ignore the standards and scalability in favor of just getting it done with as few belts as possible. Once you get green science going, the constraint moves away from how many belts you have, to how quickly you can place them down.

During this stage, the new engineer was very hesitant to do things without explicit direction. He spent a lot of this time observing how we did things and helping where it was obvious that we needed help. He asked a lot of questions on what to do. This wasn’t because he was inexperienced with Factorio. He’s actually quite the veteran. He just wasn’t able to operate in our Factory because he wasn’t sure which actions would help vs harm.

Other than that, Q1 isn’t really notable to me. It’s fairly routine.

Quarter 2

Adamantium was surprised to see how we built stuff. He’s used to creating perfect ratios of machines so that there’s no leftover material built.

My strategy involves working backwards from a science pack. I start by building a small amount, usually 1 machine for each subcomponent. I place them in a way that allows me to place more in the future. Once everything starts trickling through, I scale everything up a little bit based on intuition, then I scale out the last product to an amount I desire. From there, I just build more of whatever sub-material is missing to fill in the gaps.

Adamantium asked, “How do you know how much of each material you need?” I just told him that “I just look for whatever I don’t have enough of right now, and I build more.”

It’s not as neat and orderly as he prefers, but he couldn’t deny that it gets results a lot more quickly.

Quarter 3

Once we got construction robots and the network going, we started running into a ton of problems.

Not Enough Copper and Iron

Adamantium kept asking how much more we needed. The response was “Whatever amount you’re thinking of, it’s not enough lol.” He didn’t quite believe it, but he later was floored every time I used it all. The issue of not enough copper shifted from not smelting enough, to not mining enough, to not being able to belt enough. We started having to run additional belts, but some of our earlier choices to leave less room for the main bus in favor of getting stuff done really bit us here.

Not Enough Power

Once you get construction robots, the new constraint of growing your factory shifts away from your APM to a new area. I thought it back to either how many construction robots we have and our robotics network in general, or maybe how much materials we had in storage. I was horribly wrong, it was power .

Yup, I don’t know why I consistently forget that Roboports take a boatload of power. My dumb self made the problem even worse by deploying electric furnaces, which also take a boatload of power. So here we are, using 2 boatloads of power and we only had about a quarter boatload of generation. Our factory slowed to a crawl.

Adamantium disagreed with us on how to solve this problem. He wanted to build more steam engines. NotaFish and I wanted to build more solar panels. In hindsight we should have done both, but the direction we ended up doing was scaling out solar panel and capacitor production (which takes more power), then having construction robots place them both (which also takes more power). It was pretty slow going, especially since it needed power to do this. We were in a Catch-22 situation.

It took a while to break the feedback loop by slowing making and placing enough solar panels, but we eventually did. It probably took a good hour or 2 and really set us really behind schedule.

The Main Bus Was Getting Crowded

This was serious pain to deal with. It would have taken more effort than I wanted to to clear stuff out of the way so that I could make room for the growing main bus. Especially since we’re running out of time, I didn’t want to take even more time cleaning up my past mistakes.

I made some minimal changes here to improve it, but for the most part, I just let this problem fester and I started building around it.

Finally Advancing Towards the Rocket

Refinancing Red Science

After we got all the production stopping events behind us, we finally got to focus on researching technologies so we can work towards building the rocket again. If I remember right, it was at this point I refactored red science.

I had done it terribly in the beginning and I couldn’t scale it out any further. It really wasn’t a problem until this point in the game when we were making so much of the other sciences that this became a bottleneck. This allowed me to scale it out further. This also let me move it out of the way to make room for the main bus.

It’s super easy to do at this point in the game since I had the construction bots just clear it out and rebuild it into nice clean rows.

Quarter 4

Pretty smooth sailing here. At this point in the game, we had researched all the tools we needed to do everything well. We started building the rocket components and scaling them out.

Adamantium was stunned to see how much we scaled them out. We needed to move if we were going to finish any time soon, and that meant continually scaling up production of each bottleneck until the rocket was launched. He kept asking “Do we have enough copper/iron?” and my answer was always, “We always need more.” He didn’t believe that my answer was correct until he saw how our circuit production just ate however many resources we gave it. And yeah, this happened twice. I think he wasn’t expecting our factory to also consume a ridiculous amount of resources.

It was really exciting, but not really much to discuss here. We just built stuff, built it big, then built it bigger. We eventually launched the rocket around the 11 hour mark. We did this in one day, so I was more than ready to end my shift at the factory and get some dinner.

Coming Out Ahead

Putting it in very simple terms, I was able to use Technical Financing in a multitude of ways to make my overall tasks way simpler, be it in Network Engineering, or Factorio. I believe you’ll be able to use these tools as well to make your job/life simpler if you are able to master these concepts I’ve introduced. Hopefully technical financing can help you as much as it has helped me! You’ll be able to read about how I used these concepts in our Factorio playthrough in another blog post coming soon!

Note that I also applied the tech financing techniques in my Factorio run to get the time down to where I did. You might be able to spot them if you read between the lines. Just like I will with my data center migration story, I’m planning on writing future blog posts about each technique, how I applied them to both the DC and this Factorio run, and how these techniques work. You can find all the posts in the Lessons In Factorio series here!

Thank you for checking this out. I hope you learned something new or enjoyed reading this. If you had any comments, questions, or just wanted to share your thoughts on this article, you can contact me at blog@e-mayhem.com

e-Mayhem helps companies successfully deliver business projects. We also help companies avoid losses associated with IT disruptions and security threats. You can learn more about our services at e-mayhem.com or by emailing sales@e-mayhem.com