Full Disclosure: This is a REALLY long post.

Life for me has always been about the journey, and not the destination. I am fortunate to have experienced many unexpected twists and turns, some good, some bad, but never boring. If you just want to learn about the BSidesLV Pros vs Joes CTF, skip down to that section.

My participation in this year’s PvJ CTF isn’t the end of my infosec journey - it’s just a part of it. But I would never have predicted how I would arrive at this point. Read below to find out why.

A Little History for Context

I started my infosec adventure in July 2020 when I signed up for the CyberMentor’s Practical Ethical Hacking course, which I completed six months later in January 2021. This led to additional courses, and my first jeopardy style Capture The Flag (CTF) competition at BSides Vancouver 2021. I posted a message on the CTF Discord channel, asking if anyone wanted to team up with a complete n00b. Miraculously, two strangers (Gx00 and malwaremama) foolishly agreed to join me - and we ended up doing quite well (thank you to Gx00 who kept a screen cap of our final score - 11th place - but 95% of the points were scored by him LOL). By the way, remember those two names, there’s a reason I mentioned them.

I was hooked - and started to do more CTFs. I then discovered ctftime and in 2022 created a team, SilenceOfTheLANs. I only participated in 6 CTFs that year, but was happy to solve a few challenges here or there. It was mostly a solo affair. I just thought it was kinda cool to see your team name on a leaderboard.

Jeopardy-style CTFs were ok, but I was always more interested in the attack-defense format. I am an avid gamer, and prefer multi-player games. Similarly, practicing pen-testing skills against static boxes were ok, but wouldn’t it be cool to practice on servers managed by co-competitors? Unfortunately, attack-defense CTFs are difficult to manage, and thus are seldom hosted except for a few in-person cons, which were not common during the COVID pandemic. I had stumbled upon the BSidesLV Pros vs Joes website in 2022, but was not planning to attend any cons that year, and more importantly, I did not feel I had the requisite skill sets to be a productive participant. But wouldn’t it be cool to be able to participate one day?

Fast Forward to May 2023…

During this time, Gx00 and I had kept in touch on Discord. We recently reconnected after BSides Vancouver had concluded and spoke about upcoming conferences. He mentioned he participated in last year’s BSidesLV PvJ - and his team won! He was also invited to join the PvJ Staff, and will be a team captain this year. He encouraged me to submit an application, and assured me “I would be fine” (see meme below for my reaction). The application form included a long list of questions where you rank your depth and range of skills. Sort of like D&D attributes. I was brutally honest with my responses, submitting 1-2 out of 5 (higher = stronger) for most skills. The applications are anonymized before team captains choose their players in rounds, similar to a hockey draft. There is no guarantee my application would be chosen. Great. Memories of being picked last in gym class came rushing back to me.

A short time after, I received an email from Dichotomy that the PvJ staff have received my application, and the draft will be taking place in the near future. I was immediately excited (and anxious). However, when I looked at the email header, I noticed two email addresses in particular - Gx00, which I had expected, but malwaremama as well! I immediately reached out to malwaremama on Discord - we had not chatted in over 2 years - and indeed, this was the same person! Gx00 and malwaremama had participated in last year’s PvJ on different teams, and at some point during the planning process for this year’s PvJ had agreed to be co-captains this year. They did not realize the connection until now. At this point, I did not care if I was drafted to a team or not, I was just incredulous of how the stars had aligned and we were reconnected.

But wait - there’s more. A couple of weeks later, I receive an email confirming I had been drafted to a team - you guessed it - to Gx00 & malwaremama’s team !?!? You can’t make this stuff up.

This was followed by another email from the co-captains, calling for a team meeting.

But before we go there…

What is a Pros V Joes CTF?

From the Pros vs Joes website:

The Pros V Joes CTF is an event where the average Joe can have a chance to defend along with Professionals in the field, to learn from them while having fun. The game consists of live combat, with each team of Joes defending a network from a Red Cell of professional hackers. Each team of Joes will be lead by a Pro Captain (PvJ Staff) and Pro co-Captain. These fine folks will help train and prepare their Joes, supporting them throughout the two days of carnage and mayhem.

This is not a typical attack-defense CTF, where teams accrue points by maintaining services on their servers, while taking down services on competing teams' servers. There is an additional element of a red team, who have had advance access to your servers during the preceding weeks, to penetrate and implement multiple methods of persistence. These are seasoned red teamers who have deeply pwned your servers.

Day 1: This is meant to simulate an incident response call to a horribly configured IT infrastructure, where you cannot eradicate the APT, but just try to maintain as many services as possible. You begin with a small number of mixed OSs, including various flavours of Windows, Linux, a DNS and a firewall server. Services have included file servers, PBX, mail servers, and Jiri tickets (!!). As the day proceeds, the PvJ Staff will add additional servers. And of course, be prepared to be trolled by red team. Constantly.

Day 2: You begin the day where you left Day 1 off, except in addition to dealing with tomfoolery from red team, you are now able to attack and plant “beacons” on competing teams' servers, to score additional points. Additional servers go live during the day as well. The final hour is designated as “scorched earth”, where anything goes (primarily red team blowing everything up).

There are rules, of course, primarily to keep the game fun and educational. Sensible things like you can’t take down the network infrastructure, or block all ports/packets to certain addresses because the game infrastructure relies on these for scoring etc. But that doesn’t exclude the possibility to be clever to outsmart red team, or how the beacons work, etc. [intentionally vague].

There is a scoreboard, where you can see how all the teams are doing in terms of maintaining services for each box. It is updated every 3 minutes, as the scorebot spiders out to interrogate the status of each server and corresponding services.

Preparing for PvJ

I was most anxious about preparing for the PvJ. We all suffered from impostor syndrome and I just wanted to be able to contribute to the team. Our first team meeting was six weeks prior to the event. During that time, I stopped gaming (gasp!) and dedicated most of my free time to prepare. I was not able to find a lot of information to help with this phase, so I thought I would summarize my experiences here:

Our co-captains held weekly team meetings

• they started the first meeting with an explanation of the PvJ gameplay, as summarized above, followed by an overview of their experiences and lessons learned from previous CTFs
• second order of business was coming up with a team name (since we were all suffering from impostor syndrome, we naturally settled on The Impostars), as well as as team logo (see below)
• as the weeks went by, we began to feel more comfortable around each other and I think we started to gel as a team; we also started to get a sense of each other’s interests or capabilities at a very high level, enough that we were able to think about appropriate roles for ourselves/each-other
• I was relieved that my team-mates seemed quite chill, and our personalities mixed well - this is important since we will be spending the better part of two full days together - you may as well have fun together (oh right - this is supposed to be a fun event! LOL)
• We played a few rounds of Backdoors and Breaches. Think of it as Dungeons and Dragons but in the setting of Incident Response. I found this useful for two reasons. I had just completed a certificate program for Cyber Risk Management and Compliance; I was able to play-test the tools and terms I had learned in those courses. More importantly, I think it helped the team become more comfortable with each other
• Our last meeting was the day before most of us were to fly to Las Vegas - by this time, I was pretty excited for the event, had accepted the fact that we will be constantly pwned by red team so no amount of preparation will completely eradicate them, which meant I was pretty relaxed but excited to meet the team and start the event

We had a team Slack channel

• this helped facilitate lots of chatter in between our weekly team meetings, which was useful to clarify rules, talk strategy, get to know each other, etc.
• many of us, at multiple times, asked if we could create sub-channels for different topics - the persistent response from our captains was that on game day, it will be too chaotic to keep track of multiple channels - all communication should stay in one channel. Previous attempts at using different communication tools ie. Discord etc, was not useful, or even detrimental. It is hard to argue against previous experience. However, our captains were always willing to try/do something if we felt strongly about it.
• there were discussions on various strategies including pre-built scripts, pre-installed applications, servers, update files etc - the bottom line was, we can think about these things, but game day is so chaotic that you have little time to do any of that
• we were reminded that we should try to stick to our roles - although we might jump between roles particularly to something we are comfortable with, which is human nature
• I recall being told multiple times, that despite our preparation, red team has pwned all the servers, and we will NOT be able to eradicate them - it won’t be possible - and that’s not the goal of the game - I found this very comforting
• on game day - they were correct - it was so chaotic - thank goodness we only had one channel for all communications

We had a shared google drive

• the drive contained things like the game rules, packing list, other info
• the most important file - ONE spreadsheet - a single repository for all the things
• this included a sheet for our introductions, skill sets, and proposed roles (including Windows Admins, Linux Admins, DNS, Firewall, etc)
• more importantly, it summarized procedures for Linux Hardening, Windows Hardening, and machine lists with passwords etc
• our captains reiterated that game day will be chaotic, and fast paced, so it is imperative that we only use one single repository for all our machine information, descriptions, updates and passwords

What I did to prepare

• Although I spend most of my computer time with a GUI, I started my computer life on an Apple //+ and a command line, and have always preferred a CLI - thus, I usually prefer to learn about or work in an Linux environment. My philosophy in life is learn something new, so when I started to learn about pentesting, I purposely did more Active Directory and Windows privilege escalation versus Linux, since I thought I would learn more. To make a long story short, I volunteered to join the Windows Admin team. I have zero experience as a Windows admin, except taking care of my personal Windows machines at home. However, I have spent many hours enumerating, gaining footholds, and attempting priv esc, against Windows boxes. So I thought this would be a good chance to learn the other side of the craft
• I reached out to the other Windoze admins on the team (TheGwar) and started chatting with them - fortunately they seemed like chill, like-minded computer g33ks, just like me (albeit much more experienced)
• I reviewed my previous course material for AD enumeration, pentesting, and priv esc
• I updated the AD and Windows sections of my pentesting notes (I use Cherrytree and try to document anything I learn, otherwise I will forget)
• I used both of those resources to begin to think like a blue teamer, which I never had to do before
• Our team spreadsheet had a “Windows Hardening” section - I used this as a framework for my preparation and researched/updated this section over the ensuing weeks
• I did all my preparation on the laptop I had planned to bring to PvJ. My main desktop has a lovely dual-27" monitor setup, with a full size mechanical keyboard and ergonomic gaming mouse. I am prone to neck pain if I ignore my posture while working at a computer - so I thought I should iron out the kinks (pun intended) before the main event. I ended up bringing a second LCD monitor (14") for additional screen real estate, as well as portable laptop stands to elevate the screen closer to eye level. Since the keyboard and trackpad will now be raised a couple of inches, which won’t be comfortable, I planned to use a portable mechanical keyboard and a mouse as well.
• It doesn’t sound like much, but with a full time job and other life obligations, this took up almost all of my spare time, leading up to game day

By the final week or two before BSidesLV, I was pretty relaxed. Our captains had drilled into our psyche that red team cannot be eradicated, our job is to maintain as many services as possible, and try to stick to our gameplan but it won’t be possible to complete all tasks. It will be CHAOTIC.

I was ready to go!

Game Day!

The Night Before: I was able to have dinner and a beverage (or few) with Gx00 and TheGwar the night before. It was really nice to meet both of them in person. Someone was able to take a pic of the game area before the madness begins.

On the morning of Day 1, team members started to trickle in around 8:00 am, to get some breakfast and begin setting up our gear. We had 12 members, squished around a large table - there was barely room for one laptop directly in front if you; although many of us brought second monitors, none of us were able to use them. I took advantage of the calm to take a team pic - notice everyone is smiling? Spoiler Alert: We are still friends after the CTF.

We organized the table by “role”.

• The Windows team were at one end, the Linux folks on the other end, along with our team captains.
• Our Firewall expert (NinjaWhiskey) was over there too. We didn’t have much of a chance to chat over the two days but I understand he took an amazing job of our firewall, especially after red team nerfed it.
• Our DNS expert (ZTK) sat on our end - it was really cool to hear him give real time reports of odd network traffic or odd packets. He was able to get our DNS up and running multiple times despite efforts by red team.
• FatherStalin sat beside me, he was our Jack of all Trades - he was constantly trolling red team, by yelling at them, phoning them, or by continuously kicking them off our Linux boxes but leaving messages for them. On Day 2 he planted our beacons on the other teams' servers. It was amazing to hear him work.
• It was a long table - and once the action started, it was non-stop, so we did not have much of an opportunity to chatter with our team mates at the other end of the table.
• Here’s another team pic, partway through Day 1 - we’re still kinda happy…!

Other observations during Day 1 and Day 2

• Our captains were absolutely correct - once the starting gun went off, the action was non-stop
• We started with 6-8 servers of various flavours of OS - the Windows Admins (TheGwar, t0nedef, ip3c4c) had our hands full with a DC (domain controller), two win10 clients, and a mysterious windows server of some kind (winsrv0)
• Our strategy was to keep to the basics ie. passwords, services, users - manual enumeration and very basic scripts/apps - there was no time for anything complicated or fancy - because red team was able to reverse any actions we did, or used additional persistence methods to regain access to the machines
• Furthermore, they continuously added machines to our roster - by the end of Day 2, we had 37 machines to manage
• The concept of a single team Slack Channel, and a single data repository, also worked very well - I could not imagine having to search through multiple communication channels or data files in the midst of all this action
• Our team did a good job of maintaining services - for most of Day 1 we were mostly green, and in the lead

But things changed on the afternoon of Day 1

• Maybe we trolled red team too much? Were we too far in the lead? One thing I learned, is red team is a vengeful team. Multiple attempts to take down our linux boxes (kudos to our Linux Admins)and DNS server were thwarted - as well as attacks against our firewall, until one attack completely nerfed it for so long we were dead last - and never able to recover from that position, despite getting most of our services running again
• we were also to gain bonus points by planting multiple beacons on another teams servers, and were able to keep most of red team’s beacons out - but it wasn’t enough…

Red Team owns EVERYTHING.

• Any illusion of “control”, was just an illusion - they were trolling us constantly
• The Windows Admins were constantly banging our heads - often laughing, because sometimes that’s all we could do - we did really well in removing malware and some admin accounts, but others would automagically reappear - I felt like I was in a CIA psych experiment, where red team will constantly remind you they are in control - from hidden cron jobs that added admin accounts every few minutes (see pic below - Billy Mays haunts me to this day), to purposely rebooting the machine when you get close to removing malware (yes, they have RDP access and watch what you are doing), or changing your background on your next reboot… the trolling was endless (but hilarious)
• I have been intentionally vague during this write up, not for opsec, but mainly to not spoil any fun if you plan to do a PvJ in the future - because you always remember your first time
• they review a lot of what they did afterwards, but I think I’ll just share some memorable screen caps here

EOF

If you could tell by now, I have had an amazing journey thus far. For me, this started 3 years ago when I signed up for my first penetration testing course, followed by (virtually) meeting Gx00 and malwaremama online for my first CTF 2 years ago, and culminating in meeting them in person, to participate in my first Pros V Joes CTF.

Just a few points before I sign off:

• I highly recommend the PvJ CTF - I learned so much in preparation for, and during the event - This is more than an attack-defense CTF, this is a simulation of an incident response where you are parachuted into a horribly configured environment
• Therefore, you are competing against red team - think of them as an APT - they are your adversary - the points don’t matter, everyone gets a prize after
• Although preparation is important, KISS (keep it simple stupid) is the key - depending how you l33t you are - because there is no time to be fancy
• I’ve met some really cool people through this event and hope we maintain contact
• Thank you to Dichotomy, the PvJ Staff, Gold Team, red team, my co-captains Gx00 and malwaremama, and my team mates for an awesome experience!
• Maybe I’ll see you next year?

Addendum: Thank you to Gx00 & NinjaWhiskey for reviewing/edits.