On this episode of the AI For All Podcast, Manasi Vartak, CEO of Verta, joins Ryan Chacon to discuss AI governance, AI regulation, and AI bias. They talk about generative AI and its risks, reducing AI bias, responsible AI, and how regulation will impact AI adoption.

About Manasi Vartak

Manasi Vartak is the founder and CEO of Verta, the Menlo Park, California-based provider of the Verta Operational AI platform and Verta Model Catalog. Manasi invented experiment management and tracking while at MIT CSAIL when she created ModelDB, the first open-source model management system deployed at Fortune 500 companies and the progenitor of MLflow.

After earning her PhD from MIT, Vartak went on to data science positions at Twitter, where she worked on deep learning for content recommendation as part of the feed-ranking team, and Google, where she worked on dynamic ad-targeting, before founding Verta.

Interested in connecting with Manasi? Reach out on LinkedIn!

About Verta

Emerging from the AI innovations at MIT, Twitter, NVIDIA, Google, and Facebook, Verta, based in Silicon Valley, specializes in Model Catalog, Model Lifecycle Management, and AI portfolio management. Since its inception in 2018, it has served Fortune 500 companies and digital pioneers, and was recognized as a 'Gartner Cool Vendor AI Core Technologies' in 2022.

Key Questions and Topics from This Episode:

(01:13) Introduction to Manasi Vartak and Verta
(01:44) What is generative AI?
(02:52) Current state and future of generative AI
(04:05) Generative AI risks
(05:32) AI bias
(06:54) Reducing AI bias
(10:58) What makes generative AI possible?
(12:34) AI governance and responsible AI
(15:30) Who is responsible for making AI responsible?
(18:41) AI regulation
(23:52) How will regulation impact AI adoption?
(27:51) Will we struggle to govern AI in the future?
(28:37) Learn more about Verta

Transcript

- [Ryan] Welcome everybody to the AI For All Podcast. I'm Ryan Chacon. This week, our regular co-host Neil Sahota is out, but our producer Nikolai is subbing in.

- [Nikolai] Hello.

- [Ryan] Today's episode, great conversation we have planned for you. So we're going to be talking about generative AI and AI governance. Generative AI obviously seems to be that spark that has really lit the flame around AI regulation and discussions connected to it.

The EU itself is looking to pass an AI act by the end of this year. With powerful large language models like GPT-4 really captivating the world, the rewards and risks are becoming more clear for policymakers who are now responsible for figuring out how to take action and what needs to be regulated. So to discuss this, we have Manasi Vartak, the Founder and CEO of Verta.

They are an AI startup that is focused on building software to help enterprise data science machine learning teams rapidly develop and deploy ML models. To give you a little background, Manasi invented experiment management and tracking while at MIT. She also created ModelDB, which is the first open source model management system deployed at Fortune 500 companies.

Manasi, it's great to have you.

- [Manasi] Thanks so much for having me. Gen AI has taken the world by storm and excited to chat with you folks about it.

- [Ryan] Let's go ahead and start there. Why don't you go ahead and explain to our audience for those who may be new, I know a lot of our audience knows what this is, but some of our audience is new to generative AI and what it is truly when they hear it, they may have interacted with it, but not know that's what exactly it is.

How would you describe it to somebody who's new to the AI space?

- [Manasi] So traditional AI, call it machine learning, data science, statistics, is all about predicting known quantities. So think about predicting whether someone will like a tweet or retweet it, someone will click on an ad, or figuring out what kind of products to recommend on an e commerce website. With generative AI, you are creating content from scratch.

So this is not about predicting whether someone will like your tweet or not. It's about writing a tweet from scratch. So you, the range of outputs it can produce is essentially infinite. And that makes it super, super powerful.

- [Ryan] Yeah. Also, I think for some, super, super scary on what it can do and what the output's going to be or what's influencing the output. I know we talked about AI bias and stuff before but let me ask you, how do you view when you look at what generative AI is or is supposed to be, where are we today with it and versus maybe where we potentially are trying to get to?

- [Manasi] We're in super early innings of gen AI. The GPT models, they were, the good ones were released late last year, and it's only six months in. Those models were also trained on specific tasks, they're mainly language tasks. And so how do we fine tune them for different kinds of tasks? I think in two or three years, it's going to look very different with very vertical specific models, foundation models, sort of generative models, and this could be used in images, this could be used in texts and robotics, you name it. So just like deep learning was a sea change, this is very similar.

- [Ryan] In your opinion, how risky is generative AI? Obviously we want to hope that people have the best intentions, but obviously there is, there are things that when it comes to technology in general that people are hesitant about either adopting or using but at the same time it's something that is exciting. So from your perspective, what are the risks or how risky is generative AI from in, I guess it may be more in theory, but compared to what really is the things we need to focus on from if they would be a concern?

- [Manasi] There's kind of two camps with generative AI risk right now. One is like the existential threat to humanity camp, which is AGI is going to take over the world. I don't subscribe to that camp. I think we're very far from it. There is concrete risks that AI poses today that are more around misleading users of generative AI technology, whether with misinformation, deep fakes, all of that. There is the risk of hallucinations where these models make up or fabricate information very frequently. That will again mislead folks. If you consider a mental health or therapy bot that is making up information or guiding the user in a way that's just not good for their health.

Those kinds of things are much more immediate and need to be fixed before we think about existential risk. I think that is way far out. And as you mentioned earlier, bias. With all ML, there is bias because we're training on data that's biased. And now that bias is amplified with generative AI because its outputs are so varied.

You don't know where the bias is going to crop up.

- [Ryan] How do those biases get into the data? What contributes to or influences those biases? Is it the team that is responsible for collecting and pulling those models together? Or what is it that when people hear, oh, this tool has a lot of bias in it, what really drives that and how is that kind of handled now?

- [Manasi] The bias exists in the data. It's not really the data collection. There's things you can do to minimize bias. But to give an example, if you look at software engineering resumes, historically, this profession has had a lot of men in it, and so the resumes are from men. And so your data is biased towards those kinds of resumes that turns into the kinds of activities that someone might have participated in, the way that the language that they are using, the projects that they're doing.

And so the way you counterbalance it is by having a dataset that has both different kinds of resumes in there. So it's not like the person who's collecting the data is purposefully doing something malicious. The data that we're using has been created by our society, which does have biases.

And so the data is going to reflect those, and it's up to the data scientists or the people who are assessing the data to try to minimize these.

- [Ryan] How do you think as we move forward as an industry, we're able to start to weed out those biases or fix and make those corrections when humans are still going to be involved on some level. So what does that future look like in your mind to get to a point where it's more easily, more trusted and in a better spot where those biases are minimized to the fullest extent possible?

- [Manasi] For sure. And I think it's, the solution is partly technology. It's also partly processes and having humans in the loop. So for instance, there are well known techniques to detect whether models are fair or whether there is bias. If different groups are being treated equally. And so the number one thing is to actually do the analysis.

This is a technical solution, but at least you can find out if there is bias in your data or in your models before you release it. So to me, that's a must have. You do it at the data stage, you do it at the model stage. There's some mitigation techniques. If bias is identified, there are certain mitigation techniques that can be used to minimize it.

At some point, the business also needs to step in and talk about what bias is important, to what is ethical for them and where they're going to draw the boundaries. There are certain things that are prohibited by law and people are not going to do that, but there's also how can we be a good citizen and there's things beyond what law recommends. How do people want to integrate that into the models?

And then the third thing I would say is there needs to be a human line of defense where if it is a type of application where the bias has a significant impact on the user or could have a significant impact, these model results need to go through rigorous review for folks to actually look at. Oh, is this response biased? When it's a yes, no, and this is where traditional ML is way easier, but if it's a generative AI piece of text, a human needs to review it and say, oh, that actually sounds like someone is being sexist or racist. That might be hard for a system to detect.

- [Ryan] I'm curious, and there might not be an answer to this today, obviously, but what do you think, when you get to that last line of defense, which we're saying is probably the human element, is there a way to really remove or like really minimize the bias with people because we see it with social media, we see it with the news, there's people involved in the filtering of the content that we digest and they are, to some level, potentially instilling their own views and values into what they say is approved and what is rejected, which does influence things.

It influences society, it influences people's opinions, it influences so many different things. And as AI grows and generative AI tools continue to evolve, this is only going to have a bigger and bigger impact potentially on individuals, businesses that adopt these tools, society, and so forth.

- [Manasi] I think this is where it depends on the application of AI and that might not be a great answer, but that's where I think it is at. There are certain cases where, and I think bias, at least for AI practitioners, means something very particular. Are different groups being treated in the same way?

So it's mainly, I would say, it's fairness more than it's bias. And all the techniques are really about being fair to different groups of individuals. The AI can be used to push users towards one kind of thinking or another. I might argue that's similar to what media does today anyway. And so we're, just like we call media biased because they're picking sides, the content generated by these bots is also going to be biased to one viewpoint or another. So maybe fairness might be a better sort of term, at least that's what the AI practitioners are thinking about, recognizing that you can introduce bias into systems whenever there's humans involved.

- [Nikolai] Before we leave the topic of generative AI, I was curious, what would you say were like, because generative AI has clearly blown up in popularity. And only very recently. So what are those, what were the key sort of innovations or discoveries or technologies that really unlocked these, like GPT-4, these large language models, all this generative AI that we see.

- [Manasi] The base architecture for GPT models, when I say architecture, model architecture, has actually been around for a while. It's called the Transformer architecture, so generative pre-trained transformers, GPT. The things that were different, and GPT-3 is not the first GPT, right, there was a GPT-2 and so on.

What they did was they increased both the data size and the model size very significantly in order to let the model learn different patterns, behaviors, and ultimately be able to predict the next word. And so what changed was being able to train these massive models with billions of parameters or more.

And that means you need the GPUs. You need the technology to run them. You also need the data. The, GPT has been trained at internet scale. Now Google and others have done that even previously for search and YouTube and all of that. But this time around, the model was much, much bigger, and it allowed it to develop the capacity to solve a lot of problems without explicit training for them.

- [Ryan] Another topic that kind of comes into this discussion as we're talking about being able to make things more fair or obviously connects the bias discussion is AI governance and responsible AI. What does that really mean? Like when we're talking about that is, obviously you can look it up and get a clear definition of what AI governance is, but how do you think about AI governance, responsible AI in terms of things today?

- [Manasi] Maybe it's helpful to start from the goals. Why someone might want to do it. You do AI governance because A, you want to be within, follow the law in terms of whether it's fairness. There might be agreements on what kind of data is used within your models. It's about how different users are getting service.

So one key reason why folks are doing AI governance is making sure they're well within the law. The other one, why more and more enterprises are thinking about AI governance is because it signals that their innovation is responsible. When you think about ESG or environmental laws and people say, oh, we're sustainable, or we use materials in our products that have been sustainably sourced, it's indicating to the consumers that this company follows ethical practices and building trust with, between the consumer and the company. AI governance is a bit complicated than other kinds of governance. And that's because it's a combination of the data governance. So are we allowed to use the data that we're using for our customers. GDPR really drew the lines around what data can be used in what manner. AI is going to be using that data and even more complex ways.

And so if you're governing AI, that means you need to have a good handle on how are you governing data and consumer data? So there's data governance, then we get to the model governance itself, and that's where some of the questions around is this model fair?

Should we even be building this type of model? Or how has this model been validated? Does it hallucinate? Can we predict how well it's going to do? So the model governance piece comes in. And then the final one is, I call IT or software governance. Your chatbot or whatever it is is a piece of software.

And so just like there can be a cybersecurity attack on software, there could be cybersecurity attacks on your AI. And so if you're trying to implement AI governance, it's data governance, it's model governance, it's software governance. And you want to tie it all together to make sure that you are following the responsible AI practices or ethical AI practices that your company has laid out, and you're doing right by your customers.

- [Ryan] That was actually a lead to my next question, which is who's responsible for making AI responsible? Who's responsible for that? Is it the people, the companies that are building the tools and putting these things out to market? Is it the users? Like how involved are they? And generally speaking, I know like governments as well are starting to get involved when it comes to regulations, but just when it comes to, when we think about AI, and I'll say it again, but who's responsible for making AI responsible?

- [Manasi] I think the regulations do need to come from the government. You can expect or hope that the people building the AI systems will be responsible, but they're corporations at the end of the day, where if they're not required to, then they're going to maximize profit. And so the regulation does need to come from the government just like it did with GDPR.

And then companies, I expect, will do more than what the regulation requires to build trust with their end users. Because if you pick Alexa, for instance, if Microsoft comes up with a device that's similar, but our AI is more responsible, and it's also, it's more powerful but also more responsible, that's why someone might choose a Microsoft device versus an AWS device. So I think it's going to become a differentiator when consumers choose the products they use.

- [Ryan] Yeah, that's really interesting. I actually never had anybody mentioned that in that way where it could be a differentiator for people when it comes, it's like when, we obviously have a background in IoT, and we think about buying devices from different areas of the world. And a lot of it has to do, one of the big differentiators is the level of security and control over data and privacy and stuff.

But with AI, those elements that are going to matter to which tool do you use, one of those is going to be which AI is potentially more responsible or has less bias in it, is more fair. So, it's very interesting to think about it from that perspective, and that will be something people will evaluate and companies will want to stand out and try to distinguish themselves out as a more responsible AI if that leads to more engagement, more adoption, more users at the end of the day.

- [Manasi] There is a parallel that we draw a lot in our team, and I think we have a blog post about it, is how the environmental regulations were similar. Where are these parts sourced from? Oh, is this part coming from certain places that are not permitted or our ESG policy doesn't allow and then tracing it back to ultimately my Apple laptop, where are its components coming from?

This is going to be very similar. So if you think about IoT devices that have AI in it, then where did the AI come from? What data was it trained on? And is there, can you provide evidence that it was done in a sustainable manner? There's also a lot of discussion on how the labeling for training these large language models comes from folks who are poorly paid and sometimes need to watch, in case of video labeling, need to watch a lot of gory videos in order to label them. And so there's a very human element there too that we'll see come up a lot more.

- [Ryan] Yeah, definitely. Now you mentioned something a second ago I wanted to talk about is the government's involvement in all of this and the AI governance and such. How involved do you believe governments need to be? I know we mentioned at the beginning of this talk the EU's AI Act. Maybe if you could give a quick overview of kind of what that means, what that is because I'm curious to see or understand how governments are starting to involve themselves in this discussion, how they are starting to think about regulation because obviously that's going to impact a lot of us and a lot of things that are out there now.

- [Manasi] If we talk about the landscape of regulation today, the EU is farthest along with their EU AI Act. It has passed the key committee approvals already. Now the member states need to hammer out what those laws mean for each one of their states, and then it's expected to go up for a vote later this year and get passed.

So that's farthest along. Which is not to say that the US is not trying to do similar sort of put similar legislation in place. The states have been more forward thinking. So there is more limited acts in New York and in Colorado that are targeting AI in different ways. The NIST, NIST published a AI risk management framework last year that is a great start.

The White House published the AI Bill of Rights. And so if you try to summarize them, they're all trying to say similar things on what is expected of let's call it high risk AI systems. And the EU AI Act specifically applies to the high risk systems. The high risk systems are the ones that impact fundamental rights of an individual, so their access to benefits, their access to citizenship, voting, healthcare, education, all of that, and then of course, security, critical infrastructure, etc.

So EU has taken the approach of defining what high risk applications are and then having very thorough, they are going to be onerous, but these applications are very important and critical applications, so I think in some ways that's justified, requiring very thorough processes and documentation on how the model and the AI system was developed, how it was validated, how it's supposed to be used.

And I think it's the, it's a right step in some ways. There's a lot of requirements, but given the power of what these models can do and just the different applications that it could be used in, I think it's a step in the right direction. And there's a surprising amount of overlap in the AI Bill of Rights from the US and the EU AI Act.

So I think everyone's kinda thinking about the same stuff.

- [Nikolai] How prepared you feel governments are to regulate AI? Because I know a lot of people in AI commented that the government was slow to regulate social media. Do you feel that these new acts are reasonable and well researched, and they've consulted with the right people, or is it being rushed or what are your general thoughts on that?

- [Manasi] We spend a lot of time reading the EU AI Act, and as a technical person who knows just the ad hoc nature in which these models and things are built, it was fairly well researched. I was pleasantly surprised at how detailed it was, and it covered a lot of the key concerns I would have as a builder of technology.

And so I thought it was very well researched. It is, I think, builders of models are not prepared to have that level of oversight, and so having a risk based approach actually makes a lot of sense. Not every model, not every AI system needs to have that. The challenge I think we're going to run into is assessing whether an AI system poses risk or if the builders have done the right things requires a high degree of expertise. So think about the auditors for ISO 900, I guess, 9000, your SOC 2 or GDPR. In this case, the auditors need to have a very high degree of technical skill. And currently, I don't think we have enough auditors to even check if someone is in compliance with these laws.

And so before or as soon as these regulations look like they're going to pass, the government needs to figure out, okay, how do we actually implement the checks that we want to implement? How do we find enough people who can do these assessments?

- [Ryan] A couple interesting things there, I wanted to ask you the opposite side of one of them. Talking about the companies who are the ones building this and how this is going to impact them, but if you look at the other side of those who are going to be adopting this, let's talk about companies, for instance, who are going to be adopting AI tools, AI technologies and solutions, how will these regulations or how do you suspect these regulations are going to influence the adoption of AI tools into businesses or for companies to adopt to then pass on to their customers for instance?

- [Manasi] I think businesses will need to be more selective in the technology that they bring in. And that's where the things that we're talking about is like, where was this device manufacturer. There's a parallel that these companies are going to select AI systems that have been developed in a responsible way, have been proven to check the boxes that the EU AI Act needs.

And so I think that's gonna be a net plus for the consumers. It is going to mean that the bar for an AI system being used is going to be higher. I think that a lot of companies are going to look at these acts not only as being applicable to high risk, but also anything that's user facing. So a chatbot, you might not think it's high risk, but if there's a way to jailbreak the chatbot to do bad things, then yes, it is high risk.

And so I think it won't be limited just to high risk. A lot of companies will proactively try to understand similar restrictions or limitations of other AI systems.

- [Nikolai] So companies, more and more, will be adopting AI, but right now it's like a data gold rush. So data is up for grabs off just any, like the internet. But do you see that changing, where there's going to be more constraints on data? We had Reddit charging a premium on their API access.

So will that make AI applications more limited and not as useful in the future? And how will that affect adoption?

- [Manasi] I think that AI depends on data, and so I believe it was also Stack Overflow that said that GPT or OpenAI could not scrape their results because their business model is at stake. And I think that's a very fair thing. For the content creators and this could be very broadly content creators to do because they can get this intermediated when the AI no longer refers back to their original work.

They're pointing to derivatives. And more and more, I believe that the content creators will wall off their content, and the folks who are training the AI models will need to explicitly ask for consent before they can use that information to train their bots, and to me I think that's a very fair give and take.

Hopefully these folks will get revenue out of it, and then they can incent their own content creators to produce good content. Otherwise if you think about all the recipe apps that exist in the world, if Google just produces the recipes and there's no click throughs to the website, then how do they make money?

So I think we're going to see some changes of business model happening there.

- [Nikolai] There's a conflict on the horizon of like, we might know that these AI systems could do better than say Stack Overflow, but we can't actually use them because Stack Overflow has to preserve its business model. It's not going to just go out of business. And so I think the pressure will be on to, eventually the ideal would be that, we don't need as much data anymore.

We have new innovations that, AI can learn with less examples.

- [Manasi] I totally agree. I think one world that I'm a bit concerned about is like when AI is learning on AI generated content, it's just going to be really bad because you need the humans to create and verify the content and then you can train an AI on it. But if you only start training on AI generated content, it's going to be watered down, and your accuracy is going to be so much worse.

- [Ryan] Do you ever think as AI, AGI becomes more of a thing and we get closer to AI behaving on its own, do you think there's ever going to be a situation where we struggle to govern AI?

- [Manasi] Being a practitioner who built these systems. I just think that we're so far from that, yeah, I think we need to under, before that happens, I think a lot of shifts will happen in the world that might equip us better to answer that question.

- [Nikolai] I mean, if AI is better at governing than the government, it's the same problem because the people in government aren't going to give up their power.

- [Manasi] I hadn't thought about that. AI becomes better at governing than the government.

- [Ryan] I'm sure that's a theme for a movie or a show at some point in time. But yeah, this has been a fantastic conversation. Really appreciate you taking the time. Last thing I wanted to do just for our audience who is listening to this, watching this, may want to learn more about what y'all have going on at Verta or just follow up on this discussion, touch base in any capacity, what's the best way they can do that?

- [Manasi] One hundred percent. So our, as Ryan mentioned earlier, Verta builds ML infrastructure for traditional ML models, also generative AI, and making sure people do that in a responsible way. You can check us out at www.verta.ai, v e r t a dot ai. You can also look me up on LinkedIn or send us an email at [email protected], and we think about this stuff a lot and would love to chat.

- [Ryan] Well, thank you so much again for your time. Excited to get this out to our audience as one of our first episodes. I think they're gonna get a ton of value out of this.

- [Manasi] Thank you for having me. This was super fun.