innovation – Andrew E. Scott

How to deal with the AI pace of change

This post focuses on one of the points covered in the Far Phase board training session on Generative AI, and complements the previous posts on AI risks concerning hallucinations (making mistakes) and intellectual property.

There seems to be a new AI tool every week, and that every month a big AI firm is announcing new features. It isn’t surprising that I am hearing decision-makers are worried about making a bet on AI that is quickly obsoleted.

For organisations that have previously created Data & AI Governance boards / risk committees / ethics panels, there may also be a worry that such governance processes are asking for a level of evidence about new AI opportunities that is time-consuming to gather and create a barrier to progress. Perhaps this results in leaders who fear that their organisation is falling behind similar organisations who are announcing big deals and making bold moves.

How to chart a way forward?

Pace of Change

It is rare that a technology category moves as quickly as Generative AI (GenAI) is moving, as shown by the following two charts.

OpenAI’s ChatGPT itself is one of the most quickly adopted online services ever. It reportedly had 100 million users within two months of launching at the end of November 2022. While it wasn’t the first publicly available GenAI service, as image and software development tools using GenAI technology had been released previously, its speed of adoption shows how quickly tools in this category can become mainstream.

The above graph from the Stanford AI Index 2025 report shows the rate of improvement of select AI benchmarks as they progress towards or exceed average human performance, over the course of the past 13 years. As you get closer to the present day, you can see the rate of improvement getting faster (lines with steeper slopes), at the same time tougher problems are being tackled by AI. For instance, the steep black line that appears from 2023 and exceeds average human performance after just a year is for a benchmark relating to correctly answering PhD-level science questions.

The speed of adoption and speed of technology improvement is also reflected in the way that AI firms release major new AI models and tools on a frequent basis. This trend is clear when looking at the various releases of ChatGPT, as a proxy for the speed of the whole industry. It is typical for a model to be replaced with something significantly better within six months, and the speed of change has not been slowing down.

6 month time period	ChatGPT releases	Count of releases
2H2022	Original ChatGPT (GPT-3.5)	1
1H2023	GPT-4	1
2H2023		0
1H2024	GPT-4o	1
2H2024	GPT-4o mini, o1-preview, o1-mini, o1, o1-pro	5
1H2025	o3-mini, o3-mini-high, GPT-4.5, GPT-4.1, GPT-4.1 mini, o3, o4-mini, o4-mini-high, o3-pro	9

Keeping up

This pace of change is a universal challenge. Academics in the AI domain are publishing articles about AI technology that is out of date by the time the article is published. (Similarly, this post is quite likely to be out of date in six months!) Businesses also struggle to know whether to jump on the latest AI technology or hold out a few months for something better.

As a specific example, in May 2024, ASIC (Australian Security & Investments Commission) shared a report about their own trial of using GenAI technology. The report was dated March 2024, and referred to a 5 week trial that ran over January and February that year, where AWS experts worked with ASIC to use GenAI to summarise some public submissions, and learn how the quality compared with people doing the same work. The conclusion was that GenAI wasn’t as good as people, based on using Meta’s Llama2 model. However, Llama2 was already obsolete by the time the report was shared, as Llama3 had been launched in April 2024.

The frankly ridiculous speed of change in this technology area poses a challenge for IT/technology governance. The traditional approach to procuring technology used by large firms is a RFI/RFT process, spending 12-18 months implementing and integrating it, and spending millions of dollars . This results in wasted money when the technology is obsolete before the ink on the requirements document is dry. How do executive leaders and board directors ensure this doesn’t happen at their organisations?

At some point (perhaps in a few years), things will likely slow down, but organisations that choose to wait for this are also choosing to forgo the benefits of GenAI in the meantime and may be paying a significant opportunity cost. There is currently a bunch of FOMO-driven marketing from AI firms that play on this, and Gartner’s hype cycle shows many GenAI technologies are clustered around the “peak of inflated expectations”. While it is fair to say that organisations should avoid doing *nothing*, that’s different to saying they should be adopting everything.

Effective governance

The use of a Data & AI Governance board or AI & Data Risk group to govern GenAI is not going to help with this problem. Instead, the smart play is to use governance to drive the organisation to learn at a speed close to the speed of change. Specifically, use governance around an innovation framework to identify/prove GenAI opportunities and to centralise the learnings.

An innovation framework in this context is a clear policy statement that outlines guardrails for ad-hoc experimentation with GenAI tools. It clarifies things like the acceptable cost and duration for an experiment, what types of personal/confidential data (if any) can be used with which types of tools, what the approval/registration process is for this, and how the activity and learnings from it will be tracked.

Such a framework allows people across the organisation to test out GenAI tools that can make their work life better, and build up organisational knowledge. Just as there is no single supplier for all IT applications and systems used by an organisation, e.g. across finance, HR, logistics, CRM, collaboration, devices, etc., it is unlikely that there will be a single supplier for all GenAI tools. While there is a level of risk in giving people latitude to work with tools that haven’t gone through rigorous screening processes, the innovation framework should ensure that any such risk is proportionate to the value derived from learning about the best of breed GenAI tools available in key business areas.

Without a way for people in an organisation to officially use GenAI tools to help with their jobs, a risk is that they will use such tools unofficially. The IT industry is well aware of “shadow IT”, where teams within an organisation use cloud services paid on a credit card, independent of IT procurement or controls. With many GenAI tools being offered for free, the problem of “shadow AI” is particularly widespread. A recent global survey by Melbourne Business School found that 70% of employees are using free, public AI tools, yet 66% used AI tools without knowing if it was allowed, and 44% used AI tools while aware that it was against organisational policies. With GenAI tools easily accessible from personal devices, it is difficult to eliminate it through simply blocking it on work devices.

Organisations that are looking to take advantage of GenAI tools will typically have a GenAI policy and training program. (Note that generic AI literacy programs are not sufficient for GenAI training, and specialised GenAI training should cover topics like prompt-writing, dealing with hallucinations, and GenAI-specific legal risks.) An innovation framework can be incorporated into a GenAI policy and related training rather than being a general framework for innovation and experiments. However, more organisations should be putting in place AI policies, as a recent Gallup survey of US-based employees found that only 30% worked at places with a formal AI policy.

As well as the ASIC example above, many organisations are running quick GenAI experiments. Reported in MIT Sloan Management Review, Colgate-Palmolive has enabled their organisation to do a wide range of GenAI experiments safely. They have curated a set of GenAI tools from both OpenAI and Google in an “AI Hub” that is set up not to leak confidential data, and provided access to employees once they complete a GenAI training module. Surveys are used to measure how the tools create business value, with thousands of employees reporting increases in quality and creativity of their work.

Another example is Thoughtworks, who shared the results of a structured, low cost GenAI experiment run over 10 weeks to test whether GitHub Copilot could help their software development teams. While they found an overall productivity improvement in their case of 15%, more importantly they built up knowledge on where Copilot was helpful and where it was not, and how it could integrate into the wider developer workflows. By sharing what they learned, the rest of the organisation benefits.

Recommendations

Board directors and executive leaders might ask:

How are both the risks of GenAI technology obsolescence and being slow to adopt best-of-breed GenAI tools captured within the organisation’s risk management framework?
How is the organisation planning to minimise the use of “shadow AI” and the risks from employees using GenAI tools on personal devices for work purposes?
Does the organisation have an agreed innovation framework or AI policy that enables GenAI tool experiments while accepting an appropriate amount of risk?

In conclusion

Generative AI tools are improving a rate of change and with broad impact that is unique. It is common for a tool to be overtaken by one with significantly better performance within six months. Traditional RFI/RFT processes are not intended to support an organisation making implementation decisions about new tools this quickly. In addition, shadow AI poses risks to an organisation if it does not offer its people GenAI tools that are comparable with best of breed options.

To tackle this, organisations should ensure that they are building up organisational knowledge at the same rate GenAI tools are evolving. This way, when clear business value from a new tool (or a tool upgrade) is identified, it can be rolled-out to all relevant parts of the organisation. Putting in place an innovation framework, possibly as part of an AI Policy, will help ensure experiments can be carried out safely and at low cost by the people who would like to use the latest GenAI tools.

Board directors and senior leaders should ensure that their organisation is properly considering the risks of these issues and has a plan to address them.

Diversification is the silver bullet

This post originally appeared over on Medium.

So, they say there are no silver bullets, but for dealing with uncertainty, diversification is as close as you can get. Instead of betting that the future will turn out one way, spread your bets across a diverse portfolio of likely possibilities.

I haven’t buried the lede, so there is going to be no surprise twist here, but I wanted to tease this out to show how widely applicable this concept is.

Company boards are made up of directors that need to make decisions about the future of the company. No-one knows the future for certain, so the background and experience of the decision-makers is critical for how good their decisions are. Instead of having every decision-maker with the same background and experience, having a spread of backgrounds and experiences improves the quality of the board. There are several pieces of research showing this, but one example reported in Forbes shows that, compared with individuals, a gender-diverse team makes better decisions 73% of the time, and teams that also have age and geographic diversity are better 87% of the time.

There is a danger that this seems completely obvious. Let’s just pause for a little and consider that it’s actually a little counter-intuitive. There is a proverb that has been around since the 16th century that too many cooks spoil the broth. Certainly, for a complicated task, an individual with deep expertise can often accomplish it better than a team. In fact, to underline that point, the same study reported in Forbes from before noted that diverse teams are more likely to struggle to put their decisions into action.

The difference is between complicated and uncertain. A complicated task can be made easier through the application of appropriate tools, skills and experience. Applying these things to an uncertain task doesn’t make it less uncertain. Producing a 7 day weather forecast is complicated. Getting it completely correct is uncertain.

This same logic applies in the world of venture capital. A VC firm will raise a fund to invest in a portfolio of startups, rather than just one. However, the same VC firm will typically seek out startups that each aim to win in a single market or technology area. Knowing which startup will become a unicorn is uncertain, so is best approached in a portfolio fashion. A VC fund of $50M could include something like a dozen startups, and there’s a well-known rule of thumb that only a third of startup investments will return more than their initial investment. (On the other hand, executing a startup is a complicated endeavour, and it benefits from simplifying and focussing where possible.)

Sometimes a VC firm develops an investment thesis for their fund, such as where they believe a particular technology or market should be the focus for their investments. Here’s a collection of over a dozen different VC investment theses, but a stark example is when Kleiner Perkins announced in 2008 that they would form a fund to invest in iPhone (and later iPad) app companiesdue to their belief in the potential of Apple’s iPhone. However, such VC firms still consider the winners in that space to be uncertain, and hence diversify their investments across multiple possible winners, e.g. Kleiner Perkins ended up investing in 25 companies. Similarly, investors in such a focussed fund (known as limited partners) are likely to diversify their investments across multiple VC funds, in order to mitigate the uncertainty that a particular investment thesis is wrong. An example here is Yale’s endowment fund, which historically invested in funds across VCs such as Andreessen Horowitz, Benchmark and Greylock Partners.

When it comes to corporate innovation, the lessons are the same. A particular corporate should take a portfolio approach to emerging opportunities. There might be some hypotheses that a corporation has developed about the opportunity sizes in particular markets, products or technologies. However, no one knows for sure how the future will turn out, so spreading risk across multiple opportunities is prudent.

The recent book from Geoffrey Moore called Zone to Win argues that a company can incubate only one major new business at a time, or risk spreading executive attention and corporate resources too thin. While there are examples of large companies like Amazon, Baidu, Apple, Google and Microsoft who are able to incubate multiple such initiatives at once, there are few companies at this scale.

However, when the expenditure and resources required to progress a new initiative are relatively small, and the likelihood of success of such an initiative is still very uncertain, it makes a lot of sense to spread the company’s investment across a number of these initiatives. Diversification may involve a range of possible time horizons, market segments, product areas, or technology domains. Spread the risk to increase the chance of overall success.

Whether it is the uncertainty relating to having relevant experience for board-level decisions, knowing which startups will hit home runs, or picking the right opportunities to explore within a corporate innovation function, the silver bullet is the same. Diversify across a variety of good options.

Patience is a virtue

This post is essentially a reposting of an article that I published on Medium a couple of months ago. I am giving the Medium platform a go, for topics that are more aligned with my professional life, but I don’t want to risk that the content disappears if Medium disappears. So, I’ll likely repost everything here a little afterwards.

I was speaking to an industry colleague in the innovation space, and commented to them that in corporate innovation, it was important to have patience. They blinked and restated what they thought I meant, that it was important to be tenacious. This revealed a surprising fact for me: that it wasn’t universally understood that patience is a virtue.

In the world of innovation, startups are often revered. The innovation that has come out of the international system of VC-backed tech startups is unarguable. Accordingly, in the land of corporate innovation in particular, it makes sense to seek to learn from the startup ecosystem, and apply their proven approaches into a corporate setting. Tools like design thinking, lean canvas, and the daily stand-up are examples of this.

However, innovation in a corporate environment requires a different approach to innovation in a startup, and not all of the startup lessons translate directly. Mark Searle from UC Berkley has recently made some insightful comments about that. I will add another — that the startup lesson about the the virtue of tenacity doesn’t translate directly either.

Before I go on, I’ll share some quick definitions so we’re all on the same page. Tenacity is the unwillingness to give up, even in the face of defeat. Patience is the acceptance that true success will take a while.

In my experience, it is the latter that better supports a culture of innovation within a corporate environment. That said, good innovators are not complacent, they do not accept the status quo, and they are driven to create a better world.

The reason that patience is a virtue in corporate innovation is due to corporate efficiency. Corporates are often set up so that the same idea isn’t funded in multiple places. In fact, there is usually a natural place for a particular idea to be explored, whether it’s in the IT group, marketing, or product development. If an idea fails, and most ideas do fail, it is unlikely that the same place will fund a similar idea again immediately. Effectively, a failed idea becomes taboo for a period of time.

How does this relate to patience? Well, getting the timing of an idea right is often a key part of success. However, since having an idea “too late” is a terrible outcome, people naturally err on the side of being “too early”. When a too-early idea fails, a successful corporate innovator will take the lessons from the failure, wait until the conditions are right, and then resurrect the idea. This time, the timing is likely to be better and the execution better informed. It requires an acceptance that true success can take a while, and often doesn’t come the first time.

Tenacity can be poisonous in this environment, with the unfortunate innovator continuing to push an idea within a company even after it has failed and become taboo. The reputation of both the idea and innovator can be harmed, and neither may end up working at the company in the future, depriving the company of real value.

However, in the startup ecosystem, tenacity is valued by the VCs who back startups run by tenacious people. A VC fund doesn’t live or die by the performance of a single startup, but VCs maximise their chances through knowing a startup will keep trying to find product-market fit while they keep funding it. They can then shift follow-on funding rounds towards startups that are performing better, and let the other startups run out of cash.

Many successful people from the startup ecosystem make their way into corporate innovation. They won’t have seen much patience within a startup; startups are all about urgency. Perhaps when they see patience, they associate it with lack of drive. However, corporate innovators have as much drive as innovators anywhere, and if one idea is paused, they will be progressing one of several other ideas. Corporate innovators often have many irons in the fire.

If you’re coming from a startup world into the corporate one, try to practice your patience. Sometimes the best strategy for helping an idea work out in the long term is to put it on ice for a while. When you thaw it out later, you may be surprised at how important your patience was for its success.

And remember, there’s no such thing as a bad idea

That is the cue – “remember, there’s no such thing as a bad idea” – for beginning the sport of suggesting ideas to my fellow brainstormers. However, instead of spurring me to reckless idea generation, it always stops me in my tracks while I re-evaluate the brainstorm facilitator. There is clearly such thing as a bad idea.

Playing in traffic while blindfolded.

Taste testing the contents of the laundry cupboard.

Stripping during a speech to parliament.

Assaulting an armed police officer.

It’s not hard to brainstorm them. So, why begin an exercise with people whose opinions you value by telling them such utter nonsense?

There are good intentions behind it, I admit. Even bad ideas may have the germ of a good idea hidden within them, and maybe one of the other brainstormers can bring that forth. Encouraging people to speak their ideas without thinking about their worth can improve the pace of the brainstorm session. Disruptive ideas can come from those outside of a field, because traditionally such ideas would have been considered “bad” by those inside the field.

On the other hand, perhaps merely being accepting of bad ideas is not going far enough. I’ve found that I can generate many more ideas of much greater variety if I focus on just generating bad ones.

Suggesting ideas in a language you don’t speak.

Brainstorming with just one person in the room.

Miming ideas to the other brainstormers.

Providing the same ideas as from the last brainstorm.

Overall, it is recognised that constraints enable creativity. The restricted forms of the haiku, sonnet or even limerick are able to result in enjoyable poetry. So, it’s understandable that coming up with “any idea, whether good or bad” will result in less creative ideas than coming up with “only bad ideas”.

Still, I don’t know why “only bad ideas” seems to work better for me than “only good ideas”. Maybe it’s simply that there are more bad ideas than good ones? Unfortunately, I can’t see a brainstorm session achieve a useful outcome if everyone involved is aiming for the worst ideas.

So, I’ve had an idea for how to harness the power of bad ideas in brainstorming. At the start of the session, the facilitator gives each brainstormer a note with either Good or Bad on it – which they keep secret from the other brainstormers – and this states the type of ideas they need to suggest. Maybe just a third of the brainstormers are given Bad, since their ideas will otherwise likely outnumber the Good ones.

This should help with improving the volume and diversity of ideas in brainstorms. In this case, the brainstorm facilitator will need to cue the start of the session with something like “Remember, I want to hear your ideas, even if they are bad.”

Tell me if this works for you, since I’m not sure if my idea for better brainstorming is a good or bad one.

Thinking of changes to traditional brainstorming.

Putting those thoughts out in public.

Lessons from NYT on innovation

Whatever the circumstances that led someone at The New York Times to leak their report on Innovation, I am thankful. Published (internally) in March, it is the fruits of a six month long deep-dive into the business of journalism within a company that has been a leader in that industry for over a century, and provides an intimate and honest study into how an incumbent can be disrupted. It is 97 pages long, and worth reading for anyone who is interested in innovation or the future of media.

The report was leaked in full in May, and I’ve been reading bits of it in my spare time. Just recently I completed it, and felt it was worth summarising some of the lessons that are highlighted by the people at the Times. As it is with such things, my summary is going to be subjective and – by nature – highly selective, so if this piques your interest, I encourage you to read the whole thing.

(My summary ended up being longer than I’d originally intended, so apologies in advance.)

Organisational Division

Because of the principle of editorial independence, the Times has clear boundaries between the journalists in the newsroom and those who operate “the business” part of the newspaper, which has been traditionally about selling advertising. This separation is even known as “church and state” within the organisation, and affects everything from who is allowed to meet with whom (even during brown-bag lunch style meetings) to the language used to communicate concepts. This has worked well in the past, allowing the journalism to be kept at the highest quality, without fear of being compromised by commercial considerations.

However, the part of the organisation that has been developing new software tools and reader applications is within “the business” (not being journalists), and has hence been disconnected from the newsroom. Hence new software is not developed to support the changing style of journalism, and where it is, it is done as one-off projects. Other media organisations are utilising developers more strategically, resulting in better tools for the journalists and a better experience for the readers.

Lesson: Technology capability needs to be at the heart of an innovation organisation, rather than kept at arms-length.

Changing Customers

For a very long time, the main customer of the Times has been advertisers. However, print media is facing a future where advertisers will not pay enough to keep the organisation running. Online advertising pays less than print advertising, and mobile advertising even less again. Coupled with declining circulation due to increased digital readership, the advertising business looks pretty sick. But there’s a new type of customer for the digital editions that is growing in importance: the reader.

While advertising revenues had the potential to severely compromise journalism, it’s not so clear that the same threat exists from reader revenues. In theory there is a good alignment: high quality journalism results in more readers. But if consideration of attracting readers is explicitly kept away from the newsroom as part of the “church and state” division, readers may end up being attracted by other media organisations. In fact, this is what is happening at the Times, with declines in most online reader metrics, and none increasing.

In the print world, it was enough to produce a high quality newspaper and it would attract readers. However, in the digital world this strategy is not currently working. Digital readers don’t select a publication and then read the stories in it, they discover individual articles from a variety of sources and then select whether to read them or not. The authors of articles need to take a bigger role in ensuring those articles are discovered.

Lesson: When customers radically change, the business needs to radically change too (many true-isms may be true no longer).

Experimentation

The rules for success in digital are different from those of traditional print journalism, although no-one really knows what they are yet. That said, the Times newsroom has an ingrained dislike of risk-taking. Again this made sense for a newsroom that didn’t want to print an incorrect story, and so everything had to be checked before it went public. However, this culture inhibits innovation if applied outside of the news itself.

Not only does it a culture of avoiding risks prevent them from experimenting and slow the ability to launch new things, but smart people within the organisation risk getting good at the wrong things. A great quote from the report: “When it takes 20 months to build one thing, your skill set becomes less about innovation and more about navigating bureaucracy.”

Also, the newsroom lacks a dedicated strategy and operations team, so doesn’t know how well readers are responding to experiments, or what is working well for competitors. Given that competitors are no longer only other daily newspapers, it’s not enough to just read the morning’s papers to get insight into the competition. BuzzFeed reformatted stories from the Times and managed to get greater reader numbers than the Times was able to for the same stories.

Lesson: If experimentation is being avoided due to risk, then business risks are not being managed effectively.

Acquiring Talent

It turns out that people experienced in traditional journalism don’t automatically have all the skills to meet the requirements of digital readers. However, the Times has a bias for hiring and promoting people in digital roles based on their achievements as journalists. While this likely worked in the past to create a high quality newspaper, it isn’t working in digital. In general, the New York Times appears to be a print newspaper first, and a digital business second. The daily tempo of article submission and review is oriented around a daily publication to be read in the mornings, rather than supporting the release of stories digitally when they are ready to be published. Performance metrics are still oriented around the number of front page stories published – a measure declining in importance as digital readers cease to discover articles via the home page.

The lack of appreciation for the digital world and digital people in general has resulted in the departure of a number of skilled employees, according to the report. Hiring digital talent is also difficult to justify to management given that demand has pushed salaries higher for skilled people even if those people are relatively young. What could be a virtuous circle, with talent attracting talent, is working in the opposite direction with what appears to be a cultural bias against the very talent that would help the Times.

Lesson: An organisation pays for the talent either by paying market rates for capable people or paying the cost in lost opportunities.

Final words

When I first came across the NYT Innovation report, I expected to read about another example of the innovators’ dilemma, where rational business decisions kept them from moving into a new market. Instead, the report is the tale of how the organisation structure, culture and processes that made The New York Times great in the past are actively inhibiting its success in the present. Some of these seem to have become sacred cows and it is difficult for the organisation to get rid of them. It will require courage – and a dedication to innovation – to change the organisation into one that is able to compete effectively.

Wolfram’s Folly?

Back in the 80s, I read Robert Heinlein’s sci-fi novel Friday, where the main character did an amazing trick with a computer. She discovered a correlation between a number of seemingly unrelated factors and the incidence of the plague – information that allows her to avoid the next plague outbreak. I thought it would be pretty cool when computers reached the point that this sort of thing could be done.

I suspect Heinlein and Stephen Wolfram had a similar idea. Wolfram has just launched a web site called Wolfram Alpha that provides a way for non-sci-fi-characters to discover strange and interesting facts. It is sort of a cross between a cloud-based version of Mathematica and the CIA World Factbook. You can ask what is “2+2” or the “Population of Australia / New Zealand”. They state they have “10+ trillion pieces of data” in their database already.

But the most interesting thing about it for me is that it can answer questions that have never been asked before. Unlike Wikipedia or Google, which offer up information that people have already written down somewhere, Wolfram Alpha computes answers from its data. For example, I asked it for the “next solar eclipse in Melbourne” and got back the answer Friday, July 13, 2018 (along with a heap of other information and charts). Such information is not easily obtainable from Google.

However, while it is clear how ambitious and innovative this project is, it’s not clear to me when people would typically use it. Why would someone use it to, say, find weather information rather than going to a weather website, or movie details rather than going to a movie website. Given that Wolfram Alpha has to gather and “curate” the data in their database, specialist websites are likely to have an advantage in timeliness or breadth of their data. This is indicated in a TechCrunch article that shows they are sometimes using 2006 statistical data when 2009 data is available.

Even if ordinary people won’t regularly use it, perhaps it could get used for specialist projects or assignments. However, another issue is the black box nature of Wolfram Alpha. While Wikipedia considers itself a “tertiary source” and Google is more of a catalogue than a source, Wolfram Alpha may be the only source of a particular piece of information, given that it computed it. So, how would this data be referenced? Can it be considered a trusted source? Will specialist projects or assignments be able to use it if it isn’t? And if it can’t be used by them, then by who?

Given that Wolfram Alpha is so cool, I hope it doesn’t prove to be a folly. I enjoyed reading Stephen Wolfram’s A New Kind of Science, which he ended up provided online for free as a bit of a philanthropic service. I really wonder if that could be possible for this new project.