Ever stop to think about how your neighborhood bakery manages to stand tall among the highly successful latte chains? They’ve likely found a formula for success beyond the perfect recipes for melt-in-your-mouth chocolate chip cookies and bread that is as crisp on the outside as it is soft on the inside.
That formula may come in the form of a credo: We emphasize quality over quantity.
You’ve almost certainly heard that adage before. Maybe you’ve even said it. It makes perfect sense at a fundamental level; many people prefer quality goods over mass-produced options. Most folks, anyway.
When it comes to data that fuels generative AI, whose meteoric rise has some people heralding it as the ignition switch for the fourth Industrial Revolution, both quantity and quality are critical.
Using large, clean and diverse datasets will help you get the most optimal GenAI solutions over time. And GenAI models — like AI models before them — require high-quality data to generate great results.
Jump to:
Quantity? Check! Quality? Eh
GenAI models are trained on data, so naturally the quality of the data directly impacts the outputs the model can generate. Garbage in, garbage out, right? This philosophy is salient as you look to build and run GenAI models on premises with your organization’s data.
Operating a GenAI model in-house helps you get the performance you require without sacrificing security and privacy. You’re essentially bringing the AI to your data, rather than offloading your data to someone else.
This approach resonates with organizations, at least in theory. Many are showing interest in building or training GenAI models on premises for many reasons, including: Performance (55%), cost (35%), control over AI model (30%) and governance (30%), according to a Dell Technologies internal survey of IT decision makers.
As it happens, many enterprises have the data quantity part down, tapping into the vast amounts of business, process and operational data sitting in siloed repositories. The quality part? Not so much.
In fact, a lot more work is required before IT can build the GenAI model they require to create useful applications with the technology. How do you get started? Follow this framework to identify and prepare data for use in on-premises GenAI models.
The data preparation playbook
Define requirements
What are you trying to accomplish with your GenAI model? What kind of data do you need for your generative AI applications? What are the specific features and attributes that you need to capture? How will you identify and eradicate biases before, during and after the model is created? Failure to think through the answers to these questions can result in lost time and capital.
Collect
It’s incumbent upon you as an IT leader to make sure your data architects and engineers identify and collect the data necessary to train generative AI systems — and make sure it’s both accurate and diverse.
Clean
“Cleaning” data means handling missing values, correcting errors, removing duplicates and addressing outliers. Use undersampling to remove data points from majority groups and oversampling to duplicate data points from minority groups. Both techniques will help you aggregate a more balanced dataset.
Preprocess
Next you’ll preprocess the data, which may include tokenizing texts, resizing images or extracting audio features. This step will make it suitable for training.
Label
You’ll manually assign labels to each data point, underscoring what the data represents. Although time consuming, labeling is essential for training a high-quality GenAI model.
Organize
Organize the data for training your generative AI model, which includes splitting the data into training, validation and test sets. This is no trivial task, as many organizations struggle with organizing their data, according to Dell Technologies Global CTO John Roese. “A lot of people are behind the curve on getting their data organized,” Roese said during a recent GenAI discussion.
Model training
With high-quality and well-organized data, you can begin training. This is where the model learns to generate new data consistent with the patterns present in the training data.
Model evaluation
After training, you should evaluate the generative model’s performance using the validation and test datasets. Assess text, images or other outputs to ensure they meet your desired criteria.
Monitor
Regularly monitor the model for errors, inconsistencies and data outliers. Monitoring your data quality helps ensure that your GenAI models are always using the best possible data — and should help you avoid bias creep. Pro tip: Explore bias detection tools to examine your data as you work.
The bottom line
The playbook above may seem like a cookie-cutter approach for preparing data for your GenAI models but remember that every business is unique so your outputs, including biases and outliers, will vary. The key is identifying the right to use and making sure it’s properly prepared to feed your model.
“If you don’t have control of your data … all bets are off in doing an effective AI offering because it’s driven by the data, it’s trained on the data — even fine-tuning a model requires you to have an understanding of your data,” Dell’s Roese said.
Harking back to the neighborhood bakery analogy: Just as the baker honed his or her craft using proven recipes, your model is only as good as the data you use to feed it.
What are you going to feed your GenAI model to nurture and grow it?
Learn how Dell Generative AI Solutions help you bring AI to your data.