This post is about a proof-of-concept augmented reality (AR) board game I built. It’s really rough, and not even really a game, but it demonstrates how to use AR markers to manipulate digital artifacts in a way that could form the basis of a multiplayer game. But before getting into that, some back story (if you want to see the video of the app in action, start scrolling).

I’m somewhat of a board game fan - and by somewhat I mean more than the average person but not nearly as passionate as my other board game friends. One such friend hosts an event called “BGR” (Board Game Retreat). It fits nicely with my other yearly adventure called “GBR” (Gold Box Reunion - a completely separate event with college friends). At BRG, there’s a well-refined meta-game system. Points are awarded for winning a game, and the amount of points is a function of the number of players and how long the game took (and yes, there’s a corresponding function for cooperative games). This presents interesting within game dynamics. If you’re not winning you start playing the meta-game to decide who you want to lose to.

Anyways, BGR is where I get exposed to all the latest and greatest games, and many old ones I’d simply never ran into before. One such game is called “Unlock” and it’s a fantastic cooperative game that involves a deck of cards that you use to solve puzzles that allow you to reveal new cards. As you play a story unfolds. The game has a free companion app that allows you to enter codes you decipher using the cards, and the app then instructs you to reveal new cards. It also keeps track of time, penalties if you enter an incorrect code, and provides hints if needed. For added fun - my buddy bought two Unlock decks so that we could turn a cooperative game into a two team competitive game.

On the ride back from BGR, a friend and I started talking about why there aren’t more technology-enabled board games. There are some - but there aren’t that many, and the technological component is often handling the tedious aspects of game play like keeping score, or keeping time, or doing calculations. While useful, there’re surely entirely new game play mechanics that can be created when you add technology to board games.

My friend and I started thinking about the ways smartphones could be integrated into an Unlock style game. A key aspect of our discussion was around augmented reality (AR). First off, a quick overview of AR - it is a way of overlaying digital information on top of the real world. For smartphones, this generally consists of having digital text, images, or 3D models incorporated into the smartphone’s video feed. There are two main ways to do this - using special markers or going ‘markerless.’ Markers are specific images that the smartphone camera can easily recognize (they usually look like a 2D bar codes) that the smartphone can then render content over. A markerless system is much more like how our brain works. The smartphone analyzes the camera feed and recognizes and tracks objects just like we do with our eyes, and then renders appropriate content. You can think of this as the smartphone creating an internal 3D model of the world using the camera, then adding content to that 3D world which is then displayed on the smartphone screen. For this post, I’ll be focusing on maker-based systems. Markerless systems (Google’s ARCore and Apples ARKit) are impressive, but they don’t quite allow the same level of interactivity between players, yet, but they the future and will soon surpass marker based systems.

We started thinking about all the ways that AR could be added to a social gaming experience. An important point to keep in mind - we’re talking about a game that explicitly involves the physical interaction or proximity of the players. AR gaming done poorly (IMHO) is just adding an AR board to a computer game. Imagine chess where an AR board is added between two players and they tap their screens to move the pieces. There’s no reason this couldn’t be played with both players in different countries, and it would end up being an awkward game that would have been better without the extra technology. Instead, imagine a game where the relative position of each player factors into the game mechanics, where sound is an integral component (better than stereo if you have more than two players!), where the AR board enables puzzles or strategies that wouldn’t be possible using physical game pieces. Here’s an incomplete list we came up with of game play mechanics that AR enables:

One marker could have an AR laser beam attached to it. Other markers could have mirrors or lenses or prisms. The markers could have 3D models above them that must be arranged into a larger whole to unlock the next level. One marker (or perhaps two) could serve as reference markers that would then display a larger scene. For example, it could be the inside of a room or house with the ceiling removed. Other markers could then be moved around as the people inside the room. Markers could have different images on the top and bottom so that flipping them over would be like pressing a button. I won’t go into the math, but the phones could calculate the location of every other phone, so sound could be played based on a player’s location relative to the board, or relative to other players. The final level could be setup such that solving it puts one phone in the perfect position for a group shot that could be shared via social media. In fact, social media could be integrated throughout. Phones could record critical scenes, or scenes likely to have lots of cheering, and a highlights video automatically created that everyone gets a copy of. A smartphone itself could serve as a marker, but with a code that changes dynamically. This could yield all sorts of interesting game play mechanics. Players could take a selfie at the start of the game. The selfies could then be integrated into the game somehow. Markers could be left in completely different rooms. Players could move between rooms. The phones tracking a marker would confirm who was in the room. Players could be given secret information via their smartphone. Players could see entirely different things (e.g. one player can see ghosts, others can’t). Players could have hidden information, and collusion between players can be enforced by their screen going blank if they get too close together.

So after that long car ride with my friend, I decided I just couldn’t help myself and I needed to create a working demo. I implemented a bare bones, not at all pretty, and only mostly functional implementation of #1 above. The video below gives a demonstration of the “game” in action. It’s a simple laser beam that bounces off walls. The laser and the walls are pegged to AR markers. I used the Vuforia AR library because it includes marker-based AR, it’s free for playing around with, and it’s nicely integrated with [Unity](https://unity3d.com/. If you’ve ever done any 3D game design you’ve definitely heard of Unity, if you’re at all curious, it’s surprisingly easy and very powerful, I’ve loved using it - and it’s an excuse to learn C#.

There are some technical issues to point out. Marker based AR can have tracking issues. Since it’s only using a single camera, there is no depth perception. The algorithm is recognizing a set of key features in the image, then using the size and skew of the point set to estimate the marker’s distance and orientation relative to the camera. There’s always noise in the result, and in the case of shooting a laser beam, small errors can translate into large offsets and an unpleasant jitter to the virtual overlays. In the video, when the shapes are grey, that means each marker is being tracked independently. The small errors are largely uncorrelated and so the laser beam has a tendency to jitter all over, and even under or over shoot the reflectors. However, I pulled a geometric trick to reduce the problem. Three points define a plane, and the game has 3 markers. So when all three are being tracked (and the option is enabled) the shapes turn blue and I define a hidden ‘board’ that all the shapes are pegged to. So the markers define the plane and the laser and reflectors are forced onto the plane and orientated correctly. You’ll notice in the video that when the shapes are blue, tilting one of the makers has little effect (it would have no effect if I could tilt it without changing the center point’s location, but alas, I can’t do that when all three markers are on a table). This reduces some of the noise but doesn’t eliminate it, and importantly it eliminates a degree of freedom - the laser will never be “up” or “down” relative to the reflectors - it will always hit the reflectors exactly half way between top and bottom.

An important omission in this proof of concept is the presence of another player. I just didn’t make the time to implement that, but the extension to two or more players in conceptually simple, though with a not-so-simple implementation. Two or more players will still recognize the markers, but the random location estimation errors they encounter will be different and likely uncorrelated, so they could end up seeing different things. However, if the players are all networked together (which they would need to be for a game of course) then they could actually share information to both reduce the jitter AND come to a consensus on the position and orientation of each marker. If my phone and your phone are talking, then each can supply their best guess of the position and orientation of each marker relative to every other marker. Together, we can make a model that is more accurate than either model alone, on average. I think a Kalman filter might be a good starting point for such an estimation, but I haven’t gone too deep on this yet. As an added bonus, if multiple players are present, they can fill in each other’s gaps. If I see markers A, B, and C but you only see markers A and B, then I can give you a good estimate of where C is and you can render it even if it’s too far away or at too weird an angle for your phone to track it itself. Thus the more phones are present, the easier and more accurate the tracking for everyone.

You can check out my code on GitHub for the simple example above. And if you’re interested in a really nerdy book on the meta-concept of game design, check out Katie Salen and Eric Zimmerman’s “Rules of Play, Game Design Fundamentals”. It actually seems very applicable to a whole host of non-game concepts, maybe even life in general.