Current Opinions

I made this site to share what I am learning but I am doing the work to help myself understand the brave new world by experiencing the tools first-hand and experimenting. What I think about it all keeps changing. So, I am putting today’s opinions down on paper to see how well they hold up over time. These are my current opinions, in no particular order:

• You can’t use generative AI to make systems in domains you don’t understand without being or having access to human subject matter experts or becoming a subject matter expert. If you try to use it to build anything but the most trivial software but don’t already know how to build software, you are going to have a bad time.

• Systems based on generative AI should be subject to much more stringent liability standards compared to traditional software. This would be a sensible policy change that would let the markets control how fast it changes society by disincentivizing aggressive rollouts by CEOs (who publicly admit that their products might be dangerous).

• Based on my observations so far, the biggest opportunities to improve software development using LLMs/Chat AI (better/faster/cheaper) are in the front (product design and specification) and back (operations) of the process, not in the actual construction (coding). In other words, they are great for knowledge management of all kinds. The tools make it easy to capture, transform, and analyze information. The transformation ability is especially valuable for ensuring quality because different views help identify misunderstandings and errors.

• I believe Sam Altman (CEO of OpenAI) is right when he says that major increases in parameter counts won’t make as big a difference going forward. I think major advances going forward will come from distributed models (a mix of traditional application services, LLMs, and specialized ML) acting in concert.

• The emerging computer vision AI models and the ability to create three-dimensional objects and scenes easily will have as big or bigger impact on society as chat AI.

• The professionals most easily replaced by these tools are people in support roles who are not subject matter experts. Business Analysts in Agile shops who go to meetings, take notes, and make user stories have essentially no purpose, and the AI can already do a better job. Those who connect the dots and are good at extracting information from people and who actively add to the conversation can’t be replaced by today’s AI.

• The most useful and most accessible chat AI tool at the moment is Bing. At $20 a month, the GPT Plus subscription is a good value. At $10 a month, GitHub Copilot is not too expensive, but I don't recommend it if price is no object. Tools such as Jetbrains Resharper do a much better job, and I spend more time using Bing or ChatGPT than I do directly interacting with the Copilot tools.

• Chat assistants are very good for:

  • Extracting information from meetings and conversations
  • Reformatting content. For example, I often use the AI to create JSON, which is a format easily read by programs, from Markdown, which is a format easily read by people.
  • Creating and validating user stories in Gherkin format
  • Explaining code and writing documentation
  • Writing unit tests
  • Validating that code and unit tests are in accordance with user stories
  • Comparing alternative implementations
  • Writing small routines such as Python scripts for cells in Jupyter notebooks and algorithms that do not require complex APIs. Other examples include data model objects in C# and other languages.

• Chat assistants are not very good for:

  • Refactoring and restructuring code. It can’t take a moderately complicated top-to-bottom routine and break it into smaller pieces reliably, and although it does a good job explaining how the code could be improved and decomposed, attempts at applying these suggestions using the chat bot were frustrating.
    • Building complete systems by themselves beyond the most trivial scenarios.

--Doug Ware, April 21, 2023