On this episode of the AI For All Podcast, Adam Kamor, co-founder and Head of Engineering at Tonic.ai, joins Ryan Chacon and Neil Sahota to discuss synthetic data for machine learning models. They talk about structured vs unstructured data, the limits of synthetic data, synthetic data examples and use cases, when not to use synthetic data, data risks and privacy, prompt engineering with synthetic data, industries using synthetic data, differential privacy, computer vision, and digital twins.

About Adam Kamor

Adam Kamor, PhD, is Co-Founder and Head of Engineering of Tonic.ai. Since completing his PhD in Physics at Georgia Tech, Adam has committed himself to enabling the work of others through the programs he develops. In his roles at Microsoft and Kabbage, he handled UI design and led the development of new features to anticipate customer needs. At Tableau, he played a role in developing the platform’s analytics/calculation capabilities. As a founder of Tonic.ai, he is leading the development of data generation solutions that are transforming the work of fellow developers, analysts, and data engineers alike.

Interested in connecting with Adam? Reach out on LinkedIn!

About Tonic.ai

Tonic.ai is the fake data company. They mimic your production data to create de-identified, realistic, and safe data for your test environments.

Key Questions and Topics from This Episode:

(00:54) Introduction to Adam Kamor and Tonic.ai
(01:11) What is synthetic data?
(03:45) Structured vs unstructured data
(06:54) Synthetic data examples
(09:58) Limits of synthetic data
(11:47) When not to use synthetic data
(13:05) Synthetic data use cases
(15:22) Potential of synthetic data
(17:39) Data risks and privacy
(19:30) Prompt engineering with synthetic data
(21:11) What industries use synthetic data?
(23:08) What is differential privacy?
(24:22) Synthetic data, computer vision, and digital twins
(27:13) Learn more about Tonic.ai

Transcript

- [Ryan] Welcome everybody to another episode of the AI For All Podcast. I'm Ryan Chacon. With me is my co-host, Neil Sahota, AI Advisor to the UN, the Godfather of AI for Good. Did I get that right, Neil?

- [Neil] Hey, you're never gonna let me live that down, Ryan. Hey, everybody.

- [Ryan] With me also is Nikolai, our producer.

- [Nikolai] Hello.

- [Ryan] On today's episode, we're gonna be talking about using AI to generate training data for ML models. What are the benefits of fake data? The limitations, what are the challenges of using generative AI in data generation solutions? To discuss this, we have Adam Kamor, the co-founder and Head of Engineering at Tonic.ai.

They are a company that is pioneering data de-identification, synthesis, and subsetting of modern or for modern data pipelines. Adam, it's great to have you here.

- [Adam] Hi, glad to be here. Thanks for the invite.

- [Ryan] Let's go ahead and dive in. I want to talk about the idea of using AI to train AI. Why do that? How effective is it? How does that work? Just talk us through that at a high level.

- [Adam] Sure. So let me take a quick step back and explain what Tonic is doing in this space. So at a really high level, Tonic is the fake data company. That's how we, that's our trademark name. We generate fake data that can be used for many different things. We started off generating fake application databases so that engineers and testers could test their applications before, merging to production or going to prod or what have you.

And then from there, we started building tools for data scientists and machine learning engineers. And that's I think primarily what we're going to discuss today. So what we do in that area is we can take an existing dataset, this is a dataset that you would be training a model on, keep it simple, you're training a logistic regression. Okay? So you have this data, but maybe the data is insufficient in the sense that it's imbalanced, right? You have a lot of one group of people or a lot of one group and not a lot of the other group. It doesn't necessarily have to be people, right?

So you have a lot of one, not a lot of the other. A good example of this would be like algorithms to detect rare disease types or algorithms for detecting fraud. Where there's, most transactions are not fraudulent and only a handful are fraudulent, right? So if you're training a model, you're going to have tons of cases where there's no fraud and very few where there is fraud and that can hurt the efficacy of your model.

So what we do for data scientists and machine learning engineers is we will take your dataset, we will train a generative model on it. And this is typically going to be a structured dataset, right? This is all, let's all get kind of LLMs and ChatGPT and OpenAI out of our heads for a second.

This is on structured data, not unstructured data. So we'll take your structured dataset. We will train a generative model on it. And then with that model, you can generate as many synthetic rows of data as you like, and specifically, you can generate synthetic examples of those kind of rare or minority cases.

And then you can feed those synthetic cases back into the original dataset to rebalance things. And then you can go train that logistic regression or your XGBoost or whatever model you're planning to train on that data. And oftentimes, not always, you can actually improve the classification scores of your algorithms by augmenting with synthetic data.

- [Ryan] Can you explain structured versus unstructured data just so our audience, anybody who might not be familiar with what you mean there?

- [Adam] I'm going to do my best. Neil, we might have to tag you in if I get something wrong, or I leave something crucial out, but I'll do my best. I'm not a data scientist. I'm on the engineering side, but I can, I think I can explain it. Structured data means the, well, A) it has a structure, but that's a silly definition, right?

But it means you know the meaning of each data element, right? Like an example of a structured dataset would be a CSV file or a database table. There are, data comes in rows, but each row has fields. Each of those fields is essentially a column. Like the first column in the row might be the first name.

The second column could be the last name. The third column could be the phone number, right? But whenever you go, whenever you get a given row and you go to that third column, you know it's going to be a phone number. The data has structure, right? You could see like JSON or XML as somewhat structured data as well or maybe semi structured, right?

Because there's no requirement that a JSON document always take the same form and has the same JSON schema, but if it does, then it also has structure, right? But then you have unstructured data like a doctor's note, for example, or maybe you work for an insurance company, and you're a claims adjuster, and you go out to the site to view the crash or the accident, and you got to take notes on all the damage, right? And that's like free form text. Like I'm going to say, oh, it was 98 degrees outside. I did this, and I did that. There's damage here, there's damage there. Then you wander off for a bit and talk about something not useful.

And then you go back to talking about the damage. You don't know what's going to happen. All you know is that there's data here, and you don't really know what it is. And that is challenging. It's challenging to work with unstructured data, but there's actually also a lot of benefits to working with unstructured data and a lot of challenges in working with structured data.

- [Neil] That was pretty good, Adam. I think a pretty good definition. Structured data is labeled. You fill out a contact form, name, email, phone number, everything is labeled. Unstructured data, just like Adam's talking about, think about social media, throughout a post, nothing is labeled, right? You just free form text as Adam had mentioned.

That's the key thing there. I am curious, Adam, though, that you actually alluded to synthetic data, synthetic rows, that's normally what I hear when you make up the datasets to represent what real datasets would look like. Why do you like the moniker fake data?

- [Adam] We could call ourselves the synthetic data company. I think if you were to poll marketers, I think the fake data company has a better ring to it, potentially. But I think it's also for historical reasons. We, the company started out doing mostly de-identification of application databases and data warehouses, typically for like engineering and test teams, and there's not a lot of synthetic data in that use case. So we became the fake data company, and it was only later that we started getting into this very mathematical, very statistical synthetic data offerings for data scientists and machine learning engineers.

- [Neil] You gave us some examples like fraud, which we all hope the banks don't have a whole lot of data on that. Where else do you see people looking for fake data?

- [Adam] I'll speak about this one at a high level. You might have data on, you know demographical information of the United States. You know it's 50% men, it's 50% women. It breaks down, populations are in these states, these races, these religions, et cetera. Like this information is well known, right?

You can get it from the census. And then let's say you're a company that's collecting data on consumers and consumer practices and behaviors. The data that you have might not be representative of the U.S. population, right? You might only have good data on certain demographics, and you only have a little bit of data on other demographics.

So you might want to create synthetic data to rebalance your consumer dataset such that it's more reflective of the U.S. population when doing various types of analysis. I think that's a good example that we've come across recently. There are other examples as well.

Let me think. I'll talk about this one at a really high level. The example that I've been working with in my head, even though I didn't really make it clear to everyone, is that this is data that has column wise relationships, right? I, if you live in this state, your salary's higher than if you live in that state, right?

That would be like a column wise relationship because you have your city column and your income column. But there's also data relationships that are more longitudinal in nature, where there's a time component to it. A good example of that would be if we all, logged into our credit card account, and we look at our transactions over time, right?

Like my transactional story or history is going to tell a different story than Neil's or Ryan's or Nikolai's. I might only make five transactions a week, but on average they're higher, but they're spread out by this amount, and normally I start my day getting coffee, and I end my day getting gasoline, whatever, right?

But there's like some kind of like time component to it as well. And that type of like longitudinal data, we see a lot of that. Are, the techniques for generating synthetic data in order to improve like model efficacy, there are open source techniques for doing this on data where you only care about column relationships. Like SMOTE is a very good example.

I never remember the acronym for it, but it's a cool paper that everyone should check out. It's spelled S M O T E. All capitals. It's an acronym. The paper is available on arXiv, and I think it's synthetic minority oversampling something or other, but now I'm forgetting the acronym. And that works really great on these column wise relationships.

But when you start getting into the longitudinal components where rows have dependencies between each other like credit card transactions, then you also can run into problems. Actually, the example I was giving earlier about consumer purchases, that project or rather Tonic in that case was used to actually generate synthetic transaction histories for users that the company did not have a lot of data on.

- [Ryan] What are some of the limits to fake data or synthetic data?

- [Adam] Yeah, that's a great question. There are limits. There are limits to what is tractable to actually train on. I'm going to answer the question in a few ways. First, as you want to train on more columns, you need more rows of data. That's your first limit, right? If you have a dataset with hundreds of columns, you need many rows of data to train on so that our models can pick up relationships.

Good rule of thumb that we found is for every column of data you have, you need a thousand rows of data. Of course, if that column is a numerical column, you could probably get away with less rows. If it's a categorical column, a very high dimension, then you'll probably need more rows to train on.

But a thousand is a good rule of thumb. So that's one limit. But another limit is just how many columns of data you have irregardless or regardless of how many rows. You might have an infinite number of rows, but your, our models are not going to be able to work on datasets with thousands or tens of thousands of columns in them.

It just becomes intractable, and our models would just never really complete their training. The dimensionality of the problem becomes too high pretty quickly. So there are definitely limitations on the complexity of the dataset. Anything on the order of a hundred columns we can typically be successful at. When you start going above that, you run into trouble though.

- [Nikolai] You know, if you were to just, what if you, could you generate to your heart's content endlessly, but you would just get garbage as a result?

- [Adam] Things might not converge. So, you know, okay, you're like, okay, well, I'm just going to train it for 500 epochs, and we're just going to see what happens, right? That won't be enough time if there's too many columns of data. And then, yeah, the model is going to give you results, but they're going to be like, I don't know, you might just use, you might as well just use like random functions in Python or something.

- [Neil] What's the rule of thumb to know, then, when can you not use synthetic data? Often if you're doing something that's first of a kind, you probably don't even understand the data well enough to be able to generate synthetic data from it. For example, NASA looking for life on other planets, we're sticking to, they're sticking to carbon life because we understand, but we know that silicon life could exist, but you have no way of developing those kind of models for an AI system to analyze, yeah.

- [Adam] That's right. Yeah, if you have, we do get a lot of inbound requests from potential customers for, what we call it internally, data from scratch. And that is a time when synthetic data is typically not going to work. People are expecting a tool that, you describe to the tool the relationships you want between columns and potentially between rows, and the tool will give you something.

And that's like a perfectly easy thing to build, but people typically underestimate the needed complexity in their data to test what they're trying to test, and how quickly things get out of control, and you just can't, you end up realizing you can't really encode into this imaginary tool all of the things that are actually important to you because there's just too many, and there's also many things that you don't even know you don't know about, right?

And you end up missing it, and it gets, you know.

- [Ryan] If I'm listening to this as an organization, potentially trying to understand and what's, not just what synthetic data is, but the value it can provide to my business. What do I as an organization need to be ready with or what makes them an ideal candidate to bring in synthetic data or ready for synthetic data to do what we're describing it does?

- [Adam] There's typically two use cases our customers come to us for when they need synthetic data for their data science initiatives. The first is the one that we've already talked about, which is data augmentation. If you're training models, and you're finding that your models are not as effective as you wish they were because of a lack of data, then Tonic can help. But there's a second use case that we haven't actually touched on, and it's my fault that we haven't, and that's the privacy use case. Tonic started because companies were becoming less and less willing to use their production databases in their lower environments for developing and testing.

Companies were already either starting to move away from that, or they wanted to, but they weren't sure how. So Tonic comes along and we say, hey, we'll take your production database, we'll give you a fake version of it, it's as good as the original, but all the data is fake. And then people can use this data in their lower environments.

It's like you have production data, but you don't. But you have all the benefits of using the production data, right? And none of the downsides of like compliance, regulation, et cetera, or at least not in the same way. We're starting to see like that same trend happening on data science teams and machine learning teams where engineers won't have access to production data when they're doing their model development.

So another use case that we've found is synthesizing data for the purposes of privacy. And in fact, in our tool, when we go to synthesize your data, we give you a report once we train our models that tell you like, okay, how good does the data look. But we also attempt to answer the question, how private is it for you?

And I say attempt because that's a very difficult question to answer. It's much harder to answer questions about privacy than it is about like how good does the data look? But anyways, customers come to us needing private data for their data science teams to use in lower environments for model development.

And that's another use case that we've, that we see somewhat often. So I would say that'll be another reason an organization could come talk to Tonic.

- [Ryan] Let me ask then, when it comes to just the future of synthetic data, where is this going? Like in your mind, like what are, where does this, how does it evolve? What does this start to enable that maybe it's not enabling now? What are you most excited about this playing a role in kind of just the growth of the AI space in general?

- [Adam] What I am most excited for at the moment is a move into unstructured data, which wasn't super on my radar six months ago, but I think we can all guess what's happened in the past six months that's made unstructured data very interesting. And of course, for anyone that doesn't know, I'm talking about ChatGPT, large language models, OpenAI, etc.

I just dropped all the buzzwords at once. I, we are, we will make a move into unstructured data, and I am personally excited for that because I think there's a lot, A) there's a lot of value that organizations are going to be able to get out of large language models. Like I truly believe that. But they also are going to have a lot of challenges around using large language models in a safe way that kind of safeguards their customer's privacy and also they're like organizational secrets and proprietary information. So I, we will enter that space, or we are currently entering that space, I should say, and that's what I'm personally most excited for at the moment.

- [Neil] So for law firms, healthcare companies, things where they're worried about client information, patient information, getting sucked in somewhere with generative AI like ChatGPT, you can anonymize it or make the fake data up so these aren't even real people and they can do this without that kind of worry.

- [Adam] That would be the goal. It's like, okay, I'm a healthcare organization. I have troves of data. I would just love to feed it to a large language model. People in my organization can ask questions of it and get value out of that data that is currently like hard to analyze because it's all unstructured, right?

But how do you do that in a safe way? I think that is a very interesting question and one that we're thinking a lot about right now at Tonic.

- [Ryan] Yeah, it's interesting, just like when we're talking about, we talked to a number of other guests, bringing AI into businesses and interacting with customers, collecting data, and how you're handling that privacy. When it comes to like just generative AI and at a high level, what are some of the data risks and challenges that you see with that?

- [Adam] One way that you can get useful information out of a large language model, via like a chat prompt or whatever, is by providing the prompt with context. Like you seed context into your prompt. You give it the information that you want to ask a question about and then you say, okay, now I'm going to ask you a question about that information. But you've sent it all this information, and if that model is running on a server that you don't control by a company that isn't you, then you are likely unwilling to send them super sensitive information about your clients unless there are like contractual obligations in place, if it's healthcare, possibly BAAs signed, et cetera, right?

And a lot of companies will go down that path and that's fine. But there's also dangers on the inference side. And by that I mean, I believe that many organizations will begin like fine tuning their own large language models. They'll start with these foundational open source models that are coming out and getting better every day, and they'll train additional data on top of these large language models. And you're like, okay, that's all being done on their servers, by their people. That's great. But then you have the danger on the inference side. Okay, I do this. I have now my company's large language model, and we open it up to other people in the organization or maybe even the customers to ask questions to or to do whatever.

If you trained on sensitive information, then the model is free to regurgitate that during inference. And that's also a danger. So I see issues both on the prompt side, you know what, by providing context and also on the fine tuning and training side when you begin the inference process afterwards.

- [Neil] So I'd be curious about that. You're alluding to like prompt engineering. Given the fact that you're creating the synthetic data, some of these things, could that be a feeder into helping some of these companies also develop better prompts because you're essentially figuring out some of the parameters, right?

- [Adam] People are prompt engineering. They're coming up with prompts that they can replay with different sets of parameters in order to get answers from their LLM. And I think you're saying then to apply the synthesis of the de-identification on the parameters themselves, I think is what you're saying.

- [Neil] Yeah, because the more parameters you give, the better output you're actually going to generate. And I think that's where a lot of people struggle. Like a lot of people are like, hey, I'm looking for a job. What's the best way to network? It's pretty broad and vague. Where if you came and said like what's the best way to network on LinkedIn to find a job as a senior financial analyst in the banking industry.

- [Adam] In the banking industry in Atlanta, Georgia, and by the way, my name's Adam. I'm a man. I'm this, I'm that, yeah, exactly. Yeah. Yeah, that's interesting. That is a, that might be a nice way to integrate Tonic's de-identification technology with prompts, is just to do it via the parameters.

For those that are listening that are in the know, think of your parameterized SQL queries, right? And the body of the query doesn't change, but the parameters that you insert in do. Yeah, I like that idea. That's interesting.

- [Neil] You can always throw some royalties my way if you pursue it.

- [Adam] Yeah, and there's a recording of it. I couldn't get out of it if I wanted to.

- [Ryan] So, Adam, let me ask you this. I know we've mentioned some industries. We talked about banking, we talked about healthcare, we talked about a number of others just sprinkled in here throughout. When, are you seeing synthetic data really take off in other, in any other certain kind of industries that we haven't mentioned that would be interesting just to talk about the applications and how it's being applied?

- [Adam] The industries that we see most interested in our tools, and I'm not saying just synthetic data, but just like all of our offerings, healthcare and finance are certainly disproportionate. And it makes sense, right? Like regulated industries with sensitive data, they need tools like what Tonic offers.

We've also seen a lot of disproportionate interest and kind of use from like education technology, is a, is one that like, when we started the company, we knew healthcare and finance were going to be interested. That was on our radar from day one because everyone would, should know that I think. Education technology, like a lot of our early customers were edtech companies, and that was a surprise to us, but they have sensitive information, it's about kids, there are regulations and rules around it, and people get in trouble when that kind of data gets leaked and et cetera. So definitely edtech.

- [Ryan] Have you seen anything on like the government side by chance? Anything on more of that kind of industry?

- [Adam] There is certainly interest from the government, yeah. They certainly deal with a lot of sensitive information. I think that, for example the, this isn't super related to Tonic, but the Census Bureau, when they did the last census a few years ago, there was a big deal made about the, an army of statisticians applying differential privacy to various results from the census in order to make available to researchers.

And I enjoyed reading all about that because differential privacy is one of the techniques that we employ at Tonic to aid in de-identification.

- [Ryan] What is that exactly? Can you explain that?

- [Adam] I can, but not well, but I'll do my best. It's very complicated. I'll do my best though. It is a mathematical framework for kind of giving you an idea of what's the worst thing that can happen if an attacker gains access to your de-identified data.

And what it gives you really is like plausible deniability, so that an attacker doesn't know with certainty a piece of information about someone they don't know anything about. But they know it with some probability and because that probability isn't one, you get some plausible deniability.

And there's a lot of mathematical properties of this framework that make it very interesting to use in practice. For example, there's this notion of composability where as long as all of the transformations applied to the dataset are differentially private and the output's differentially private, and once an output's differentially private, it's going to remain differentially private.

Like things of that nature. And it is a nice framework to use, and it's what we have typically employed at Tonic.

- [Nikolai] Another interesting use case I've seen is computer vision. So not just doing synthetic data on text or text data but doing it on other modalities like video. So you could have a recording of a, going down a street from cameras that are on a car, and then you could synthetically put that street in different weather conditions.

- [Neil] It's actually an interesting question because I was actually talking to some people in Tampa this morning, and they're worried about hurricane season. They've already been surprised by a couple of tropical storms. And so I was actually talking about something that NIST is doing and that we have a ton of data now on like hurricanes.

We can actually use AI to simulate different hurricane conditions. And this is now actually experimenting with building codes and building materials to see if we can actually make buildings more hurricane resistant or at least build new buildings that are that way. They can simulate like a Category 3, Category 5, Category 7, like they haven't even experienced Category 7.

So it's fascinating to see these, some practical use in terms of natural disaster prevention and recovery using some of the synthetic data.

- [Adam] Is it a physical simulation they're doing where like they're, and by physical, I mean, I know it's being done on a computer, but they're like doing something with like finite element analysis type deal on different building materials, with different forces being applied by different wind speeds, or is it something different?

- [Neil] Little bit different than that. They basically create a digital twin of the city. So you have all that stuff, but then you actually use generative design. So you're actually having an AI system come in and say, okay, given the parameters around different hurricanes and things like that, I want to figure out an optimal design.

So it's generating millions upon millions of design options and even fabricating materials like we've never actually made before but could and then testing those designs against different types of hurricanes, category speeds, different weather conditions, like all that kind of stuff, time of year, to figure out then which of those designs would be the best in a hurricane situation. I've heard people say can we do the same thing for earthquakes. We actually don't have enough data on earthquakes to figure some of that stuff out, right?

If you talk about like we know some things that are better in terms of a moving Earth situation but earthquakes, the Richter scale goes up exponentially. The difference between an 8.7 and a 9.1 is significantly different. So there's, we don't have any, we don't have enough good data to understand enough about how earthquakes operate, the damage it causes to actually even generate fake data around that to figure out what would happen.

- [Ryan] As we wrap up here, Adam, I wanted to just finish off with just having you give our audience some insights into where they can learn more about what you all have going on. Who's the ideal kind of people out there that are listening to this that should reach out and engage to talk more about what they're doing, potentially follow up on this conversation, that kind of thing.

- [Adam] To learn more about the company, you can go to tonic.ai. To reach out to me or to someone at Tonic, you're, you can send an email to [email protected], or you can reach out to us on Twitter. And for who should reach out? Like the people that we can help are those that, it's two types of people, right?

There's like the data science and machine learning teams that need synthetic data either for the purposes of data augmentation or privacy. And then there's the engineering and testing teams that find themselves continually frustrated by the lack of quality data in their development and staging environments and would like production-like data in order to do their jobs easily and more better.

If you're any of those people, reach out, and I'm happy to chat.

- [Ryan] Nikolai, Neil, any last questions or any comments?

- [Neil] I would say great discussion and Adam, I think you're really on the cutting edge here because I think the need for synthetic data or fake data is going to rapidly grow. So kudos man for being the pioneer on that.

- [Adam] Oh, thank you. I, mostly luck, I think, but I do appreciate the kudos. So thank you.