Despite the flood of buzzwords and marketing hype around artificial intelligence (AI), many of these products aren’t capable of delivering on the miraculous things they promise. Case in point is IBM’s Watson for Oncology.

In a recent article from STAT, the health and medicine site examines how it believes IBM Watson for Oncology failed to live up to its promise to revolutionize medicine. While the article’s language definitely makes it feel like a hit piece against Watson, the findings bring up an important lesson for business and tech leaders: Don’t believe the promises made about AI–yet.

IBM has marketed Watson for Oncology as a tool that will digest massive amounts of data and help doctors pursue evidence-based treatment for the patients. The idea is that it would quickly surface research and treatment options, and increase the amount of time that doctors could spend with their patients.

SEE: Special report: How to implement AI and machine learning (free PDF)

However, STAT argued in the piece that this isn’t the case. IBM has not provided any scientific articles demonstrating the effectiveness of Watson for Oncology, and the tool often provides redundant information or recommends treatments that are biased toward American doctors’ preferences, being that it is trained by doctors from Memorial Sloan Kettering Cancer Center in New York, the report said. There also aren’t any third-party studies on its effectiveness, STAT said.

Dr. Taewoo Kang, a South Korean cancer specialist who has used the product and was interviewed by STAT, said that “Watson for Oncology is in their toddler stage,” and it needs help to grow healthy.

Some of STAT’s biggest contentions are that Watson for Oncology capabilities aren’t fully understood by the public and that it relies on a team of human doctors to train it. One of Watson’s trainers, Dr. Mark Kris, told STAT that it is a “struggle” to update the system.

STAT also argued that Watson for Oncology simply recommends treatment methods that doctors were already aware of, and doesn’t present any new insights, which it said IBM claimed the system could do. For example, Dr. Sujal Shah told STAT that Watson provided some additional background for a specific case, but recommended the same treatment he had already decided on.

In regards to potential bias, the report noted that some doctors in Denmark dropped Watson for Oncology as they only agreed with the system in 33% of cases. That rate, though, can jump to 96% in other countries.

However, the democratization of cancer data can also be hugely beneficial for hospitals without as many cancer experts. UB Songdo Hospital in Mongolia follows Watson’s recommendations nearly 100% of the time, the report said, which allows their doctors, who are mostly generalists and not cancer specialists, to utilize expert advice in their treatment plans.

Still, being that Watson for Oncology is in its infancy, the product doesn’t provide explanations for why it makes some of the recommendations that it does. Dr. Taewoo Kang told STAT that it once recommended a drug called taxane for a patient whose cancer had not yet spread to the lymph nodes, but that drug is typically used in patients whose cancer had spread to the lymph nodes.

Regardless of how one feels about Watson for Oncology in particular, the big takeaway from this example is that businesses interested in AI shouldn’t buy into the marketing hype around the technology. The entire AI industry is in its infancy, and many marketing campaigns are over-promising what the technology can deliver.

Instead, company leaders should seek out real-life case studies of the AI tools they are considering adopting. Does it fulfill the promises of its marketing? If not, is there another novel way that it can be used in the organization. Business leaders must ask hard questions of the products in order to justify an investment in them.

It also comes down to how early an organization wants to begin its journey with AI, especially if their vertical is one that stands to see significant change from the technology. As Dr. Uhn Lee told STAT, “If that trend, that change is inevitable, then why don’t we just start early?”