A Smooth Ride in Game Development with AI Assistance
I'm currently working on a game inspired by tabletop adventure games, filled with twists, turns, and surprises. Random events play a key role in the gameplay, making weighted random numbers crucial for game mechanics. The algorithm needs to handle various scenarios for different game objects, such as controlling the probability of encountering monsters or items based on a character's level or location.
I started by using Bing to develop an algorithm that would generate weighted random numbers based on the position of items within a list. Surprisingly, Bing came up with a decent implementation on the first try. However, as with any code, it's essential to validate it, so I asked Bing to write some unit tests.
The unit tests revealed some issues in the algorithm, and through a series of iterations with Bing, we managed to resolve these problems. But as the code structure became more complex, Bing's ability to reason about it began to degrade. Interestingly, Bing repeatedly forgot to write comments throughout the code, even when asked multiple times, making it resemble a real-life software developer!
I then turned to ChatGPT using the GPT-4 model and found it slightly better than Bing at understanding the complex algorithm. However, both AI assistants still struggled to provide deep insights as the code structure grew more intricate.
Despite these challenges, I managed to produce a solid weighted random number routine, which I've made available as a NuGet package.
DocFX Struggles: A Stark Contrast to Game Development
While AI was helpful in game development, it wasn't as effective when working with DocFX, a tool I've been using for years to build static websites for my open-source projects.
One persistent issue in DocFX has been the homepage displaying the same content in both the top and left navigation. Although this problem has come up numerous times on StackOverflow and the DocFX repo, neither ChatGPT nor Bing could offer any help.
Surprisingly, I spent far more time dealing with this small DocFX behavior than building the complex algorithm with unit tests for my game. Ultimately, it was my human ability to debug software and understand large, complicated systems that allowed me to resolve the issue. This experience taught me that if AI assistants don't have a clear understanding of the problem or domain, their help may be limited.
Is it Still a Good Idea to Share Work Publicly?
Considering the power of AI assistants in parsing publicly available information, there may be concerns that their presence will have a chilling effect on discourse, especially for specialized domains. I thinks it is completely sensible to wonder, “will my code and thought leadership be used against me by a robot and put me out of a job?”
While AI assistants can be incredibly helpful in boosting productivity, they struggled with the proprietary knowledge around DocFx which is very well documented with lots of samples. That work might have been faster if I had just rolled up my sleaves and thoroughly examined all the options myself!
The next post will cover an experiment I did with the OpenXml SDK – a nasty bit of software indeed with a huge amount of material online going back over a decade.
Final Thoughts
AI assistants, like ChatGPT and Bing, can be fantastic tools for boosting productivity and guiding developers through complex tasks. However, they still have limitations, especially when it comes to specific domains or understanding intricate code structures. Developers need to have a strong foundation in their domain and problem-solving abilities to effectively utilize AI assistance. AI may not replace your job anytime soon, but it can certainly enhance your productivity if used correctly. The stark contrast between the ease of developing a complex algorithm for my game and the struggle with a minor issue in DocFX serves as a prime example of AI's potential and limitations. As AI technology continues to evolve, it's crucial for developers to understand how to effectively leverage these tools in their work while recognizing their constraints.
The next day I gave it a much harder challenge…
--Doug Ware, April 17, 2023