Duolingo's Winning Data Strategy with Bilal Zia, Head of Data Science and Analytics

Transcript

This has been generated by AI and optimized by a human.

Show ID (00:04):

The power of data is undeniable. And unharnessed, it's nothing but chaos.
The amount of data was crazy.
Can I trust it?
You will waste money.
Held together with duct tape.
Doomed to failure.

This season, we're solving problems in real-time to reveal the art of the possible. Making data your ally, using it to lead with confidence and clarity, helping communities and people thrive. This is Data-Driven Leadership, a show by Resultant.

Jess Carter (00:33):

Hey everyone. Welcome back to Data-Driven Leadership. I am super excited about today's episode because we're joined by Bilal Zia, head of data science and analytics at Duolingo, one of the most innovative ed tech companies out there. If you've ever tried to learn a language online, chances are you've used Duolingo. Founded in 2011, it's now the world's most popular language learning platform, offering courses in more than 40 languages to hundreds of millions of learners around the globe. At its core, Duolingo is on a mission to make high-quality education accessible to everyone, and it does this by combining clever design, powerful data, and cutting edge technology.

Jess Carter (01:13):

Bilal is right at the heart of that with a background that spans economics, machine learning, user research, and global organizations like Amazon and the World Bank. He brings a rare blend of technology expertise and leadership insight. In this conversation, we dive into how Duolingo personalizes learning using data, the surprising ways that AI is shaping the future of language education and the real-world challenges of building ethical inclusive technology at scale. One of my favorite parts of this episode was when he walks through the S-curve for a lot of companies and what it looks like to apply and build out a data science team to meet the needs of the company where it's at. Whether you're a data leader, an Edtech founder, or just someone curious about how apps like Duolingo actually work behind the scenes. This episode has a ton to offer. Let's get into it. 

Welcome back to Data-Driven Leadership. I'm your host, Jess Carter. Today we have Bilal Zia, head of data science and analytics at Duolingo. Let's get into it. Bilal, welcome.

Bilal Zia (02:15):

Thank you for having me. I'm excited.

Jess Carter (02:17):

Yeah. We're so excited that you're here. I am fascinated by your career. So I mean, first of all, as an international studies major, I want to talk about the World Bank a lot, but we're not going to today. We can do a little bit. I just couldn't believe when I saw that you worked at the World Bank, Amazon, and Duolingo. So one of my first questions is sort of open, but I am curious what drew you into the world of education and Duolingo and how maybe those earlier steps led you there. I think a lot of people who joined from a data-driven leadership perspective, their career path doesn't make sense on paper, but it totally makes sense when they tell it. So I'm just curious if you'd walk us through your background a little bit.

Bilal Zia (02:58):

Yeah, sure. I think the term that resonates throughout my professional career is intrinsic motivation. That's what’s driven everything I've done. And I would say my interest in education started even before I started my professional career. I would say it's in my formative years. I consider myself a global mutt. So I grew up all over the world. My family is from Pakistan, so I have spent my formative years there. So the fascinating thing about that experience is that I got to witness firsthand what poverty looks like, and that's not just lack of money or lack of physical capital, but lack of human capital, and that's education, essentially. And so education, we kind of all think of it as sort of a grand enabler, but it can also exacerbate inequality when the rich have access to education and the poor don’t, or at least good education. And that sort of keeps inequality growing even further.

Bilal Zia (03:54):

So, I was always interested in education and one of the main things I focused on at the World Bank over the 15 years I was there was to actually help folks who are less fortunate than I am make better decisions through education. So specifically, I work a lot on financial education, so helping people make better decisions about credit and savings and important aspects of growing a life and moving up the ladder. 

And then intrinsic motivation is also what took me to Amazon because Amazon was starting a central science team that was inward facing. So looking at, so basically within the HR department. And Amazon, as you may or may not know, is essentially a small country. So they have at the time 1.5 million employees. About a million of them are the workers that work in warehouses, deliver our packages. So when I look at their socioeconomic profile, it's actually not that different from say, a school teacher in Brazil or a farm worker in India. And so the same motivation that I had to work at the World Bank carried over to Amazon, and that was a very fulfilling experience for me. Duolingo was an opportunity that essentially ticked a lot of boxes. It was bringing me back to education directly and also delivering education through a means that is very impactful, which is through technology. And that's the reason why I'm here and super, super excited. Probably the best career move I made.

Jess Carter (05:22):

That is super neat. And those are, I mean, I would say those are all such different entities that it's really interesting, I imagine, going from the World Bank or Amazon to Duolingo, the size, the complexity, the maturity of those companies probably vary greatly. So maybe one of the questions for me to ask, too, is when I look at your career, is there a moment where I think this sort of is a driver, especially for data scientists, analysts, BI developers…where you really felt like there was a really rewarding project you worked on at Duolingo, and it's okay if it's the other places too, but where you feel like there is a little bit of a tug at the purpose that education, that intrinsic…I'm just curious if you feel like you've had one of those moments where you're like, wow, this is super neat.

Bilal Zia (06:12):

There have been several, but I would say the thing that I'm most proud of at Duolingo is actually not one particular project, but it's actually building an inclusive and impactful team culture. When I joined Duolingo, the data science and analytics team was a little bit disorganized. They had been searching for a head of data science for a while. So what I inherited was some very talented folks, but also folks that didn't have good direction and hence, they were underappreciated. And then the morale was also not so high. So all I did was essentially build the structure for them to thrive and then get out of the way, essentially. And essentially, that's the thing that I'm most proud of, that we've done so well that now we are considered as one of the best performing teams at the company. And that has led to many projects that have been very successful. But I would say that building that overall structure has led to those successes.

Jess Carter (07:10):

When you say that, I have met a lot of data scientists in the last seven years at different entities, and it's been super interesting to see as the market started asking for data scientists, it feels like almost every entity was not prepared to operationalize data science. And then there's these waves of maturation. So it's like we got 'em, but no one's smart enough to lead 'em, and then we look for the leader. And then when we got 'em, were they analysts that got promoted or are they data scientists who can build their own models, understand what we need to do, real mathematicians, et cetera. So there's these waves of understanding. And then I watched as other entities got to a point where they realized, oh, these are not just technical teams that shouldn't know anything about the rest of the business. Actually, bringing them in and integrating into our business so they can help us with a whole bunch of things around our business strategy, our competitive nature, et cetera, was vital for their value. Does any of that resemble where you joined at Duolingo and some of the things you were referring to?

Bilal Zia (08:15):

Absolutely. I think the immersing of data science into the culture of the product and the direction the team is critical. I don't think that we are a modular input into a production function of a company. That doesn't really work because we don't add value unless we know what's going on. One of the main things in building this team structure was to actually embed data scientists deeply within teams across the company and essentially, the marching orders for them were consider yourself part of that team and, essentially, think of yourself as a technical PM as an example, so that you take ownership of the direction of the product, raise your hand when there's a thorny question that comes up and you say, hey, I can bring data to this question. And that has empowered the team to actually do that precisely. And then that does two things. One, it raises their morale, and B, it raises the stature of them in the eyes of the product leaders and other senior leaders of the company. And that then starts this virtuous cycle.

Jess Carter (09:17):

Yeah.

Bilal Zia (09:17):

Positive externality. I'm using an economics term, but the other thing I would say to your point about how do you adapt data science to where the company is? So I would think that the profession isn't sort of one size fits all. And that's important for us to recognize and realize because let me just take two stark examples. Let's take Amazon versus Duolingo. So you think of a business growth and a business typically grows in along an S curve so that there is an initial period which is sort of just getting your feet under you, and then there's a sharp growth. So you're on the backbone of the S-curve, and then basically eventually some diminishing return set in and then you're growing slowly. And if you think from a data science perspective, there's this constant balance between speed and rigor that you have to toggle depending on where you are on that S-curve.

Bilal Zia (10:12):

So at Amazon for example, they are much more mature than Duolingo. So they've already hit that concavity on the S-curve. So they are, in order for economists and data scientists to add value, we have to turn the knob on rigor up so high. And the work that we did and the work that the teams are doing are publishable versions. They could be published in academic journals. I mean, that's the level of rigor that needs to be applied in order to add value because all the low-hanging fruit has been taken already. Now you come to a place like Duolingo, now we're on the backbone of the S curve and what matters for us is not super rigorous sort of publishable analysis, but speed. We want things done fast because we want to make decisions fast because there is a lot to be gained by speed. So making that adjustment was sort of one of the biggest learnings I had when I joined Duolingo. And I think that's something that data science leaders and teams should be aware of as they adapt themselves to their particular contexts.

Jess Carter (11:17):

That is fascinating and makes tons of sense. Now, I think what makes some of this a little bit stickier. I've spent years working with data scientists for people who haven't, can you make it real? So what is an example of something a data scientist has done at Duolingo where we can tell from a business or an app or a functionality perspective that that's been influenced by data science work?

Bilal Zia (11:39):

Here's an example, forecasting. So basically we have users on our app and we care a lot about growing those users. So the metric that we use is daily active users or DAU. And every quarter we have to forecast what our growth is going to be for the next quarter, for the next year. And there's obviously a science behind how you do forecasting. So the models that we build to forecast what the growth trajectory is going to look like depend on things like, well, what has the trend been in growth of users in the past? And can you extrapolate that trend over to the next quarter or the next year? But then on top of that, you also have to account for things like what are the big things that we're doing on the product side that will increase an influx of users onto the app? So you have to kind of account for those events within your forecast as well.

Bilal Zia (12:29):

So it's this balance between sort of rigorous science, but then there's also an art component because you are kind of guessing you are making a very educated guess, but it is a guessing game. So data scientists play a big role there. And that's one of the big successes that we had at the company is that prior to, or just two years back, essentially the way that our forecast worked was that we had an underlying scientific model, but then we were also adding in a lot of subjective decisions on top of the model. And what we did over the last two years is essentially wean off of that subjectivity and increase the rigor a little bit of how we do the modeling. And what that has led to is a much tighter and much more accurate forecast. And that has built a lot of trust with senior leaders that they now trust the forecast because over the last four quarters we have been within zero to three percent accuracy, which is quite impressive. I give credit to my team for working on this and doing really well, but that's an example of where data science or data itself has essentially changed the mentality of senior leaders and built trust. And now the folks who used to fret about, oh gosh, we have to do this forecast and now we have to think about different things and how we change the numbers. They're like, we don't have to do anything. This is great. We trust the science. And so it's been a win-win in that sense.

Jess Carter (13:57):

That's amazing. And I think that when I'm thinking about some of the work that you guys are probably doing, there's also this curiosity I have. So in a product company like Duolingo, or an educational platform, there's other companies that I would—you probably wouldn't consider, they're not competitors, but they're similar in structure, I imagine. Some of the ones we've talked to like Bumble or others where it's like they have users they want on the platform, the users have a goal, so these…yours want to learn a language or several, theirs want to find somebody to spend time with…as a scientist is there any ethical discussion about what if the science got so good that people could learn the language so much faster and more effectively and then they wouldn't stay on the platform? Is that ever a conversation or is that not proved out?

Bilal Zia (14:49):

I think learning a language is probably going to be a demand from users that's not going to tap out anytime soon. So we use this terminology called total accessible market or TAM, and essentially the guesstimate that we have of what the TAM is, is about 2 billion people worldwide want to learn a language. And that need is not going to go away despite of AI, despite of other.

Jess Carter (15:16):

I was going to ask that question. That was where I was going next

Bilal Zia (15:19):

Because I mean, if you think about it, so a lot of reasons why or one of the main reasons why people want to learn a language is so that they can have a conversation in a natural way with someone. If they travel and they're meeting somebody who doesn't speak English and they want to just have a conversation to learn about their life. Now, you introduce AI as a sort of real-time translator in the mix. There's a big disconnect in how natural that conversation feels. So people will always want to learn a language and they want to learn it by speaking, not by having a real-time translator. Real-time translation has been here for a while, even well before AI. It works. I mean, in my former world at the World Bank, we had rea-ltime translators all the time because we're constantly meeting with people who didn't speak the same language and it worked really well. You can literally hear within sort of a delay of one second what the other person is saying, but it just always felt unnatural.

Jess Carter (16:17):

That's really interesting. So you're basically saying when people desire connection, I think about a friend of mine who married someone from another country where their primary languages were different. They're not going to use an AI tool all the time. They want to actually have a connection. So when you're looking for transactional communication, maybe there's a space in the market for AI tools, but when you're looking for relationship connection, developing something with someone else, you're going to want to learn a language. That's so interesting. As I talk to entities who have data science teams, and you're talking about embedding them in departments, again, there's sort of this, are they helping with business operations? Are they helping with the product itself? So do you guys span both of those areas?

Bilal Zia (16:57):

Yeah, so we have product data scientists that are working directly with product managers and product teams, and those would be the product managers themselves, the engineers that help them and the product designers, and then data sciences sort of the fourth pillar within that decision hierarchy. So then there's also sort of helping working closely with the finance team, for example, and then having a business intelligence functionality. And we've made some great strides in that space as well. When I joined, we actually didn't have a BI team, a business intelligence team, and that was leading to a lot of stress for our product data scientists because they would have their sort of OKRs for the quarter things that they have to deliver, and then they would get pulled off when there's suddenly some fire or some metric has tanked and we don't know what's going on. And they would go and investigate, do a very good job, but then they would come back to their regular OKR and still be expected to deliver on that with no sort of discount applied. So you can imagine that their morale was pretty low, also very stressful. So basically we created a firefighting squad, which is essentially their job is to do this, and they're very motivated, they're super excited about this, and they're doing a really good job. So they are happy. The product data scientists are now happy because they're insulated from these firefighting duties.

Bilal Zia (18:17):

And the BI team actually is one of our at the forefront of the AI wave. So we are trying to build automation into the workflow so that the work that would normally have taken one to two days, now we can do in 30 minutes. The work that we could do previously needed two months of investment, we are now actually experimenting to make that AI-first entirely.

Jess Carter (18:41):

That's impressive. A lot of people are, so they're either just dabbling in AI or maybe they've just got their data science team or their BI team. The idea that you've got, for lack of a better phrase, you've got these big, big brains who are working on really big complex problems and you're able to outsource the other stuff to AI to build efficiency and scale for your team. That's probably keeping you from having to overstaff in substantial ways, which is pretty neat, too.

Bilal Zia (19:05):

I think the staffing is always going to be a need. I think the way that I think about AI helping within data science is that you can think of essentially every single employee having a small set of RAs: research assistants. Those research assistants are the AI agents, and they're basically, well, there's a repeatable, predictable task that needs to be done that usually was previously manual, you had to write a lot of code from scratch. Well, why not let AI do that? And then you can check it, which is really important, but then that saves you a lot of time. And then there's also repeatable tasks that previously were not automatable, but now we can automate them away. So in that sense, we still need a human in the loop, and that's important for quality control and other reasons.

Jess Carter (19:59):

So one of the questions I wanted to ask you, I imagine something an end user cares about is their progress towards fluency. They want to know that they're getting better. I'm curious if your team helps, how do we think about measuring fluency or tracking fluency or demonstrating progress on the app? And is data science involved in any way?

Bilal Zia (20:21):

Yeah, so there are a few answers. So first is that we have the Duolingo English test. So this is obviously relevant to learning English. Obviously people are learning other languages, but let me just focus on English. So that’s basically a proficiency test and the purpose of that test is to help students who are trying to get admitted to higher education institutions in English-speaking countries and are testing requirements, et cetera. But it's also a great tool for folks to understand what their speaking level is. And the technology that we've invested in is really, really good. So we can actually pretty accurately assess your speaking ability on a scale based on a conversation. So based on the test that's been developed. So that's number one. So there is definitely for folks who are interested in assessing the proficiency, well, we have a tool for that. 

Second, we have the Duolingo Score, which is basically a progress score depending on where you are in our path. So the path is basically the main track that you follow in our app, and depending on where you are on that path, you get assigned a Duolingo Score and a higher score means that you are at a higher point on that path. The last thing I would say is that we are also exploring from a data science perspective, developing some sort of predictive tool that we can actually understand whether our users on the main app are proficient in the language as well. And that's something that we're actively working on right now.

Jess Carter (21:55):

Wow, okay. Well all of that is incredibly cool. And do you mind me asking how many employees are at Duolingo? How many people are in the data science division?

Bilal Zia (22:05):

Duolingo overall, please don't hold me to this, I think is around 900 and data science is 25.

Jess Carter (22:13):

Wow, okay. So that's what I was going to ask next. So as the leader of that department, I'm curious, obviously you are captivated and passionate about the work that you do. I'm curious about, as a leader what you enjoy the most. I'm dreaming of a world where I'm doing your job, and the most fun part in my opinion would be that you get to hear about all of these really complex problems from your team that are embedded in all these different areas, and you're able to wax poetic, if you will, or philosophical or logical about what problems they're having and how to help or how to look at it differently. Or if you're stuck with a model, what we can do about it or how to get to your earlier comment, how do we get subjective bias out of some of the game here, but then there's also the hands on keyboard. So most leaders of data science became leaders of data science because they were data scientists. So I'm curious, do you still get to play and have hands on keyboard or are you like, you know what, I miss it, or no, this is so much more fun. I'm just curious what your job looks like.

Bilal Zia (23:15):

Yeah, I definitely miss it. So I think the analogy that I like to use, and this is not mine, I'm probably reusing it, is basically an orchestra conductor. So the orchestra conductor's job is to basically manage the individual instrument players and the instrument players are the best players for their particular instrument. And the orchestra conductor's job is to make sure that they’re, the tune is there and they're all operating in sync. And if even one person is off, then the whole orchestra kind of sucks. So for that orchestra conductor to just focus on do one thing and ignore the rest, the whole system is going to break down. So in some ways it's a requirement that I have to operate at that higher level and make sure that I'm observing the whole system, but that also means that at times when somebody is stuck, that I do kind of have to go down deeper and work with 'em, and that's actually quite enjoyable. So I'm an economist by training, so we have a lot of tricky questions around experiments and causal inference and machine learning, and those things are very exciting to me just to get my brain really sort of working in a different way.

Jess Carter (24:30):

Yeah.

Bilal Zia (24:30):

And I love that. So I'm able to do that, but then very quickly extract myself out of that and go back up and make sure that the whole system is back and humming. So I think that's my role, and I actually really love it because if you think about it, the number-one music fan is actually the orchestra conductor. So the number-one data science fan is me.

Jess Carter (24:50):

I would imagine part of what's tricky there or a skill or an art is once you rolled up your sleeves and help somebody figuring out when do you pull back away? In my head it would be hard to always discern that. It probably depends on the person, the skillset, the context, the environment. But it sounds like you are a good conductor.

Bilal Zia (25:12):

I don't think there's a standard answer. It really depends on what the problem is and who is the one who's solving it.

Jess Carter (25:17):

If people have used Duolingo front end, if they're engaging in the platform, the learning paths feel incredibly personalized. So I am kind of curious how your team might use data and machine learning to shape each learner's journey, or what signals are you looking for to adapt the experience in real time? Is that a thing?

Bilal Zia (25:38):

Yeah, so we have lots of teams at Duolingo who are working and using data for, I mean, we are a data-first company, so we use data every day in the way that we make decisions. There is a learning and curriculum function within data science that is made up of learning designers. Their job is to, for example, make decisions like the one that you just mentioned on how to personalize and make the content more engaging for our audiences. Some examples of the way that we in data science have used data to personalize content is actually applications of machine learning. I can give you two examples. So let me start with one, which is within our monetization pillar, we are a freemium model. So what that means is that we have, most of our users can use the app entirely for free, but then if you want sort of some additional bells and whistles and you don't want to watch ads, you can subscribe.

Bilal Zia (26:33):

Obviously from a business perspective, the folks who are able to pay and would benefit from a subscription, we would like them to be subscribers. So we can actually use data to predict someone's likelihood of subscribing. And at different points of their journey, we have a choice on whether to show them a Duolingo ad or a network ad. And essentially there's a machine learning model that we built that predicts each user's likelihood of subscribing. And just at a high level, if that number is high, that probability is high, we show them a Duolingo ad because we would like to have them as a subscriber, and if that likelihood is low, we will rather show them a network ad so that we can get ad revenue. So it's essentially this type of thinking, and we've applied this model and more sophisticated versions of this model elsewhere as well has opened sort of a new vector for the monetization.

Jess Carter (27:32):

Wow. It's rare for a company to say our internal data science team is directly responsible for driving revenue increases in either ads or in subscription modeling. That's neat.

Bilal Zia (27:44):

Obviously the credit is shared with product teams and engineering, but the engine behind it is sort of this machine learning.

Jess Carter (27:52):

That's neat. Just being seen as a revenue generator is probably a bit bespoke for data science teams everywhere. That's cool. And then the last thing I was going to ask you is, I know we already talked about AI a little bit, generative AI, et cetera. I'm just curious if you have any examples of how you all are using it today that you haven't already mentioned, or as you’re thinking about where you all need to be in the next three to five years or 18 to 36 months, if you've got some thoughts on how to leverage AI that we haven't already discussed.

Bilal Zia (28:21):

Yeah, for sure. I mean, I mentioned, I gave some examples of how we're using it to supercharge business intelligence functionality, automating a lot of things, including things like anomaly detection and then root cause analysis as well. So we have some tools that are essentially text-based instruction that essentially result in the AI going and writing SQL code. We're also investing in and playing around with a functionality that where it runs not just one SQL analysis, but it does multiple analyses, some form reinforcement learning, and then it actually writes a report on its findings. So it's essentially doing root cause analysis as a first wave. So obviously we're working on it, sometimes it does a really good job, sometimes it does a poor job, so we want the quality to be more consistent. So still the need for humans in lulu, but essentially the idea is that our data scientists will have a very reliable RA that will go and do this.

Bilal Zia (29:18):

I think at a high level, we are pretty well placed as a company to succeed in the AI world. And the reason for that is that we as a company have what we call a “dogfooding” culture. So we kind of dogfood the heck out of everything. So we are ourselves kind of the most critical about our own product. So we test new features, everybody's on their phone testing new features, and then calling out issues and problems, reporting them. We have a shake–to–report functionality. We dogfood and we find mistakes and we make things better. So with AI, it's the same thing is that we're going to dogfood the heck out of AI, and some things are going to work really well out of the box. Some things are going to suck, and we're going to encourage people to dogfood. And the incentive is not that you have to be right every time you try something, the incentive is just try. Sometimes you're going to come up to it with a goldmine, and sometimes you're going to be like, oh, this thing sucks. Let's move on to the next thing. It's not a punitive culture at all. It's like we're dogfooding. So that has been, I think, is going to be a key to our AI adoption and success hopefully.

Jess Carter (30:24):

I have never heard this concept of dogfooding. Is this a Duolingo special or am I living underground? Is this something you've used elsewhere?

Bilal Zia (30:33):

Well, it's kind of like the concept is you eat your own dog food, so.

Jess Carter (30:36):

Yeah.

Bilal Zia (30:36):

You taste essentially. Yeah, I don't think it's Duolingo-specific, and it's something that we really adopted.

Jess Carter (30:41):

The phrase was just new, but I hear you. Yeah, you gotta eat your own dog food. That analogy. Yeah, I think that that's a good way for us to wrap because I think the concept too, the reflection point of, hey, we're all here for a limited time. You obviously have this really neat career where you've been tying your passions and purpose to what you care about since its onset, I think there's a chance to reflect and say like, hey, where are there areas where what we're putting out on earth, we can make sure we're eating our own dog food, making sure that it's adding value, making things better in whatever ways that we're empowered to do that. It's just really inspiring. So Bilal, thanks for joining us today.

Bilal Zia (31:17):

Thank you, friend.

Jess Carter (31:18):

Yeah, and if people want to follow you, we'll put a link to your LinkedIn in the show notes on the podcast app, okay?

Bilal Zia (31:24):

Sounds great.

Jess Carter (31:24):

Thank you for listening. I'm your host, Jess Carter. Don't forget to follow the Data-Driven Leadership wherever you get your podcasts, and rate and review letting us know how these data topics are transforming your business. We can't wait for you to join us on the next episode.