How to Answer the #1 Question in Enterprise Data Science: “So What?”

Dave Cole

Hello, welcome to another episode of the Data Science Leaders podcast. I am your host, Dave Cole, and today’s guest is Khatereh Khodavirdi. She comes to us from PayPal, where she is the Global Head of Analytics and Data Science for Global Merchants, which we’ll ask you a little bit of what that entails. And prior to that, you spent about five, six years at eBay, is that correct?

Khatereh Khodavirdi

Yes, that is true Dave.

Dave Cole

Great! One thing I just want to dive into to hear a little bit more about, at PayPal, tell us a little bit about your role briefly with regards to Global Merchants. What is it that your team works on?

Khatereh Khodavirdi

Sounds good. First of all, thanks a lot for having me. It’s a pleasure sitting down with you and talking about such an important topic. About the PayPal role, I would say I came to PayPal to lead their analytics and data science team for their small business side of the house. PayPal is a two-sided network. You have consumers and you have merchants. And your merchants, you have a wide variety of groups from really big enterprise merchants to the mid-market to the merchants who are joining through channel partners, like Shopify and WooCommerce, to all the unmanaged business.

My team was on the hook to actually help grow the business through data and insights. And by that, we mean growing their net new acquisition, growing the cross-sell side of our business, and reducing the churn and decline. Almost about two years ago, I expanded my responsibility to cover the commercial side of our business. So pretty much, we’re working with all of our sales team, enterprise sales, inside sales, all of our customer success management, our demand gen team to really partner with us and help them grow our business through data and insights, but also most importantly, provide a better experience for our customers globally.

Dave Cole

That’s awesome. Thank you, Khatereh. Let’s start off with your experience at eBay that kicked off your career. In preparation for this podcast, you had a fascinating story on how you injected data science into one aspect of eBay’s business, so take it away, Khatereh.

Khatereh Khodavirdi

Thank you. I would say it’s actually an emotional story for me, as well. It was my first job in the industry and I said that I grew up in advertising and I grew up at eBay and I had such a fascinating experience. I have to say the piece that made that experience very exciting and motivating for me was that, first of all, advertising was a smaller startup within the company. You can imagine you had the funding and sponsorship of the big organization, but you had the luxury of moving really fast, working cross-functionally to make things happen really quickly.

I was one of the founding members of their data science and analytics unit within the ads business. We partnered with all the product organizations, engineering organizations, the sales teams, go to market teams and operations teams to build up a data and analytics organization, to help grow that advertising business through data and insights.

As part of that journey, we actually launched many new advertising business lines for eBay. And, you can imagine, eBay’s core business is commerce. Really figuring out how to scale the advertising business within a separate business unit that is the main core business in commerce, it’s very interesting. There are a lot of great applications of data science and analytics that you can leverage to grow the business with.

Dave Cole

Right, so this is advertising within the eBay site itself, is that correct? This is not advertising, brand advertising, outside of eBay’s walls…television commercials and things like that, is that correct?

Khatereh Khodavirdi

Definitely, it was not on the marketing side so I was not part of the ecosystem, like buying ads for the eBay brand, but we actually had advertising platform, both for on eBay helping sellers and brands grow their businesses on eBay and also at some time we had an off-network advertising as well. But not advertising on behalf of eBay, advertising on behalf of other merchants.

Dave Cole

If you’re a merchant selling your wares on eBay and you want to post an ad to a specific audience within the eBay platform, so there’s the customer on… You think that your product is going to resonate with them and might be something that they’re interested in and that’s what you were helping to inject some analytics—how did you go about doing that? You said there’s sort of an emotional story here. How did that come to be in your first job out of college?

Khatereh Khodavirdi

I would say it’s funny because actually I remember, I think it was around 2015, I was still managing some of the other advertising units and exactly the business line that you mentioned currently is known as a sponsored ads or promoted listing on the platform. They decided that the analytics and data science unit is moving to my organization. I remember I had a couple of really interesting conversations with some of the go-to-market and sales leaders as well. So I said, “Okay, our product is fairly new.” And you can imagine it doesn’t really matter if you work for a big company or a smaller company. At every point you are making the call about prioritization because all your resources, engineering, product, all the functions are limited. I asked them the question, “Actually, what type of sellers are we going after with existing products to get out of this platform? But more importantly, what type of listings or items are we asking these guys to put on the platform?”

I had a great conversation, it was a hallway conversation and I was actually thinking about it over the weekend. And I get back to our GTM leader and I say that I think we need to be more intentional around where we need to go after because with the current existing product we have, you can imagine that consumers are already doing a lot of searches on eBay so you can imagine where you have a selection gap and you can build a data science model to see who out there in the marketplace has the right listing. Pick up the targeted listing for the go-to market plan and say, “Hey guys, go after the list of these sellers.”

But also, importantly, imagine you are a seller and I reach out to you and say, “Hey, Dave, you are a great candidate for this product. You should hop on this because of X, Y, and Z. There is a great benefit. You can grow your GMV and yada yada yada.” But the reality is that you will ask the question, “Hey, what type of listing should I put on that advertising unit?” Probably from a margin standpoint, it doesn’t make sense to upload all your listings on the advertising platform. So we went one step further and built the item recommendation that builds a model that recommends to each seller what type of items you should actually put on the advertising platform.

Also the third part, I would say, to complete the demand side of the equation, was the pricing recommendation. Pretty much seller recommendation, item recommendation, and pricing recommendation on how you can actually grow the business. Because I always say that it’s a flywheel effect, that you get the right demand on the marketplace that consumers will come and do their checkout around them and it’s more about the seller gets more revenue. You have a greater story to onboard more sellers. That was a greater story that proved thematically how data science and analytics helped us to start building that business line.

Dave Cole

Let me slow it down here for myself. You have this new merchant advertising platform that you’re rolling out at eBay back in the, I think you said that 2015 timeframe. The first problem you thought about, I think you mentioned you were talking to some of the leaders who were rolling out this platform. Which sellers are we going to go after? Which sellers do we think this new ad platform might serve? Which audience would be interested in this? It can’t be all sellers. We can’t invest, or just spam everybody to let them know about us. We needed to be more targeted, right? I think the first place where you saw needed some sort of analytics was, maybe it was a segmentation or some sort of an analysis to figure out what sort of cohort or groups to go after. And I imagine what messaging to go after with them. Then the next thing was actually item recommendation, so was that part of the ad platform, as well? If I’m a consumer and I’m buying a product like, “Here are some other products you might be interested in,” and building that as a part of the ad platforms, so to speak? Do I have that right?

Khatereh Khodavirdi

Yes, so think about it this way. If you’re a seller, you need to onboard your listing as part of our advertising platform. So imagine you have 10,000 listings on eBay or a thousand listings, you need to decide: Are you uploading all your listings or are you uploading a selection of your listings? The whole point of any advertising unit is that the seller is willing to spend some money on advertising, with the hope that they are getting the return and their will GMV increase. So that is why we built the item recommendation that these are the items that for seller X, that we believe that if you put on our advertising platform, it still helped you grow with a GMV.

Dave Cole

Right, so similar to the pricing, which is the next type of use case that you went after…Instead of just saying, “Hey, merchant. Come to this new platform and figure it out.” You were using analytics to be more prescriptive saying, “First of all, let’s target which merchants we want to go after, number one. Number two, let’s help them figure out what items they should be advertising, and then also number three, how should they be pricing them in order to get the best return on their ad dollar?” Do I have that right?

Khatereh Khodavirdi

Yes!

All right, wow. So even a blind squirrel…no, that’s really interesting. So you propose these three different approaches where analytics can be integrated into this new rollout of the new ad platform. What was the response? I imagine it was a good one. I imagine they said, “Yes, that kind of makes sense, Khatereh!” What was the response?

Khatereh Khodavirdi

The response was very positive. I would say one thing throughout my career I learned is that it always takes a village. And cross-functional collaboration between the data science team, the engineering team, the product team, and the business unit is super critical for any successes on the business side. It’s always that, it’s not like the data science team can solve this in isolation, so it was a close collaboration with the business unit and with the product organization. We had a very pragmatic approach of “test and learn.” Let’s build this quickly and test it for a couple of sellers and see what the response is, and see if we want to scale it further as well.

Dave Cole

So you took a “test and learn” approach. I forgot at the outset to mention the agenda, but this actually segues nicely into the agenda. We wanted to talk about injecting data science into the business. This example has been fantastic. The other thing I wanted to talk about is experimentation in the corporate world and just how do you educate the business side to be okay with doing this sort of “test and learn” approach? And the third thing I wanted to talk a little bit about with you, is how you’ve built out in recent years, your data science team and what are some of the things that you look for and what are some of your philosophies? At any rate, getting back to this fantastic and fascinating story, you said, “Hey, you don’t need to take my word for it. Let’s do some tests.” What did that look like?

Khatereh Khodavirdi

We built the item recommendation and seller recommendation model. We came up with the list of targeted sellers and asked them, these are the folks that you should have analysis around them, reach out to them and get them on the platform. And if they show some interest, you should actually tell them that this is the item recommendation that you have for them, that these are the lists we recommend them to put on the platform. Literally, there was a data scientist on my team. We brainstormed on the first version of that approach. We built it pretty quickly. I still remember that at the time, we did not have an automated product to upload everything in the system. We’d been working with a couple of our account managers to upload the items on the platform. We started actually monitoring that very closely that, “Okay. These couple of sellers came through our recommendation. These are the listings we recommended. Are we actually showing the ROI for them?” Because at the end of the day, it’s just the seller getting extra money on the table. And they have some expectation of incremental sales that they are getting. Eventually we ended up actually a scaling program to more sellers and broadly in the organization.

Dave Cole

So, the end of the story is: successful, ended up rolling it out. Did some of the places where you injected the data science, it sounded like it was fairly manual in the early days, but did it become more automated, more integrated within the platform itself?

Khatereh Khodavirdi

Exactly, and you touch on a great point. I would say there are three areas that data science can help a lot. One is probably through automation. In any organization, there are a lot of things that, either it’s a pain point for the consumer or it’s a pain point for the existing people within the company. There are processes, tasks, activities that they are doing manually. We did all the advancement in data science. You can pretty much automate a lot of manual tasks. And the second, I always say that intelligence, you don’t want your business unit or your product to think about what to work on, who to sell to, what to sell, who to service, what to service, or at what price. You really want to work on the problems that the “so what?” to the business is very clear.

If I go back to your question that, “Hey, how do you convince them?” Because the reality is that you cannot solve any problem overnight. And for me, it’s just building that trusted partnership with your cross-functional stakeholders and showing them the path that, “Here is the uber vision of where we want to go, but here is the short-term solution, here is the long-term solution, and here is the path to get there.” And I always remind the people on my team that, definitely solving the problem is really important, but the more important part is why we are solving that problem. And what is the “so what?” to the business. If you are focused on the “why” and the “so what,” usually you can get buy-in fairly quickly within the organization.

Dave Cole

From my perspective, Khatereh, just having spoken with a number of data science leaders on the podcast, if there’s any main takeaway to data scientists out there who are wanting to become data science leaders, it’s to understand the business. Get as close to them as possible, set up those partnerships, be there, become their ally, understand the use cases, understand why it’s important, understand how to measure your projects and show value, all that good stuff. The more you do that and the better you do that, the more effective you will be as a data science leader. That is loud and clear. One thing that I wanted to segue into is that balance between doing purposeful project-based work and also doing some of the R&D experimental type work. What is some of your advice there just based on your experience on how to balance those two?

Khatereh Khodavirdi

It’s a great question. I would say, in my view, it depends on the maturity of the project and the business line as well, right? If you have a more mature project, probably you have a more project-based approach, but you also want to make sure that you’re not only focusing on the short term stuff and you want to have a balance of accelerating your business short-term, but also solving and establishing yourself for the long-term.

Usually, the way I run my organization, we have a quarterly roadmap. For different areas within the organization, the leaders in my group, they put up a quarterly project. We try with 70% of our time, we decide that these are the projects that you want to work on and to get pre-alignment with our cross-functional stakeholders to make sure that we are aligned because at the end of the day, we want to help these guys grow their business. So if you are doing something and there is a need on the product side or on the business side, we want to make sure that you have that cross-functional alignment, or on the operational side.

But there’s 30% of the stuff that I would say, all the business today, especially in technology, is moving really fast so you cannot necessarily predict what’s happening in the business in the next three months. So in other words, you need to balance the agility, that you don’t want to slow down the business, but at the same time you also need to be laser focused on the long-term vision as well. So, I try to do that. But also on the research and development, like a really longer bets. There might actually be projects that we don’t have it right now, so it’s really about, for me, prototyping it in a smaller scale.

I always say that, “Show some quick wins in a smaller scale.” I always tell the folks on my team that in data science, that you have a luxury of working on any projects you want, as long as you prove the value to the organization. But I’m still fully in and I spend months on “prove your point.” Do something quick and dirty, prove the value, get other people’s feedback and buy-in first, and then you can get a lot of momentum in the organization and go. Because the worst thing that can happen is, you are working on a lot of cool projects, but nobody’s using it because at the end of the day, I always say that the data science team and analytics team exist to help grow the business and make all the other people’s lives easier in the organization.

Dave Cole

Yes, so a lot of good stuff there. First of all, a quarterly roadmap. I imagine at the beginning of each quarter, you sit down with your team to figure out what projects you’re going to be tackling. I imagine also the business side is aware of that roadmap. The second thing I heard you say is that, roughly speaking, 70% of your team’s time is focused on some of the roadmap-related items and the other 30% is more ad hoc, right? Because the business is moving fast and you never know when you’re just going to get something like, “Hey, I need this tomorrow,” or “I need some analysis done or model built.”

So R&D, what does R&D look like? Do you have members of your team who are dedicated to just doing some type of research? What does a day in the life of that person look like?

Khatereh Khodavirdi

I have a really small group that’s doing R&D that’s not aligned to a project. It’s like a small team of three or four people. I think about our big bets that we literally brainstorm like nobody working on those projects right now. We truly believe that we can accelerate the business through some data science capability and they will start working on it. I would say, you can imagine, not all the time our bets are correct, so it’s kind of a balance that we are really good at finding out of it and we go and double click on it and build momentum around it. Sometimes, hey, we will pivot quickly. Fail fast and move on to something new. It’s a very small team that we have dedicated completely to the R&D part.

Dave Cole

That’s awesome. They’re working on big bets, which means they’re probably working on new ideas. Do they ever go back to other older models and say, I think we can outperform an older model, with maybe deep learning, some other approach that maybe your team hasn’t taken to date?

Khatereh Khodavirdi

It’s obviously a combination of both. Some of them, that capability doesn’t exist, but some of them, to your point, you always take a step back and you say, “Hey, you build something, and it’s good, but it’s not great and you can make it better.” How can we make it better? So the team spends some time to actually improve it.

Dave Cole

If I’m a data science leader listening to this podcast, I might be thinking to myself, “I don’t have an R&D team. I might just have a team that’s 100% focused on project-based work. Everything that we do has to get approved by the business, has to have a budget associated with it, yada yada yada.” But to your point, there can be some big bet important strategic ideas that don’t come from the business side, that come from the data science teams themselves, right? How did you convince the powers that be, that you needed to carve out this small team to be able to do R&D? I mean, maybe it was an easy conversation, but for those out there that are thinking, “You know what? I probably need to create this function. I think making these big bets could really pay off for me in the long run.” How did you do that?

Khatereh Khodavirdi

Good question. I would say it’s definitely not an easy answer. I always say that, even for the two or three people I have on an R&D, I tried for them to have a hybrid role that had a project that is adding a direct impact to the business. But also giving them some space for exploratory stuff, right? Actually, one of the outcomes of one of our R&D projects, it is just fascinating. We built an algorithm that is pretty much reprioritizing and scoring the leads that are going to all our inside sales ecosystem, so it can unlock revenue directly for our organization. I have kind of an MVP on that project and then presented it to our EVP and said, “There is some revenue out there that we are leaving on the table. Can we actually put more resources behind it?”

And it was a much easier conversation because I had a result of the test, I could ask what the impact is to the business. So easy that I could have that conversation. But I would honestly agree, there is no easy answer. And for me, it’s just, you need to be very creative. Definitely you don’t want to have a team that you feel like, “Hey, they’re working on cooler stuff, but not having any impact to the business.” But at the same time, if you’re just focusing on today and tomorrow, you probably will lose out on the big picture down the road.

Dave Cole

Yeah, I think selling the fact that you need to have that balance, right? I think a 70/30 split sounds pretty good to me. I mean that’s number one. And the second thing I heard you say is that building out a prototype and instead of just walking in saying, “Hey, we’re thinking about kicking off this ‘shoot the moon’ big project. It may flop, or it may succeed.” You actually go in with a small prototype. When you’re talking to an EVP, you need to come prepared, right? If you’re talking to somebody lower level, you might be able to say, “Hey, we’re thinking about building out this prototype,” but I think starting with a prototype is fantastic advice.

I’m going to segue since we’re talking about folks that have a hybrid role on your team, I want to talk a little bit about hiring. On a podcast that I’ve had in the past, I mentioned that it looks like today, there’s 3X the demand for data scientists as of 2020, then there is the supply of experienced data scientists that can meet that demand. What have you done, from a hiring perspective, to meet your own personal demand for building out your team?

Khatereh Khodavirdi

I always hire for diversity and complementary skill sets. And by diversity, I don’t only mean diversity of gender or background, I truly mean the diversity of thought process and skill set. You can imagine that data science and analytics is a wide spectrum of skill sets and you potentially need a hybrid of these skill sets to be successful.

Second is that, for me, I always say that skills are cheap, passion is priceless. Especially these days, it’s super easy to pick up some of the technical skill sets that you need to be a good data scientist. But the most important part is actually thought leadership. Somebody that is a good problem solver, that can take a step back, break down the problem into sub-problems and synthesize the recommendation. And again, elaborate on the “so what?” and also stakeholder leadership.

I think we touched base during our conversation many times that you cannot drive value in isolation. And it is more and more important that the data science and analytics teams work cross-functionally within the business unit, product unit, engineering. So I always try to solve for that. Historically I hire both, I would say, fresh out of grad school people because I moved from academia to industry and I found that topic very fascinating how you can educate people and help them to go through that transition from academia to industry. But also hiring more skilled leaders to bring into my organization.

Especially, hiring leaders in my organization, I always hire for complementary skill sets because I don’t want to be a limit for the organization. I always say that any of the leaders in my organization, there is at least one or two areas that they are 10X better than me at. And we usually celebrate it because I want all of us, collaboratively, to be the superstar and world-class organization. And for me really actually being intentional, that at every stage of the organization, what type of skill sets you need, it is very important.

I would say the second part is that the majority of the time the organizations are putting a lot of effort and energy into the technical training side of the house. Like having seminars, classes, all of this statistics, machine learning, and programming for people versus in my view, there has been limited attention on how we can teach people to be better storytellers. Like how to capture your thoughts. What does the end-to-end data science and analytics project cycle look like? How can you actually have a structured way of starting the project, getting engaged with your stakeholder, and getting it finished, and really easily. We also spend a lot of time and energy in my organization to just train people across that and make sure that it’s for a combination of technical skills and softer skills.

Dave Cole

Okay, there are a dozen phenomenal tips and tricks that you just gave us there. I was taking notes. I like to recap to the audience. Diversity of skills is important, but that being said, skills are cheap and passion is priceless, I think that’s a great teaching tool. I think all of us want to work with people who love what they do, who are passionate about what they do, and generally speaking, they’re going to be the people who drive the most value for your team and are the most fun to work with, quite frankly. You’ve mentioned this a few times, but what’s the “so what?” And I think having that in the back of your mind when you’re presenting…don’t forget that, right?

One of the big mistakes that I think a lot of data scientists make is…sometimes their passion can come through and they love to talk about how they built the cake. All the ingredients that went into the cake. And the audience, they might be sitting there and going, “You find this interesting, storyteller or data scientist, but I’m waiting for the ‘so what?’ Why do I care about the ingredients? Does the cake taste good? Can I sell the cake?” That kind of thing. I’m taking the weird cake analogy too far there, I think.

But at any rate, you also hire a fresh out of grad school, right? I imagine people coming out of grad school, they’re ready to hit the ground running. They’re very passionate like you were, when you started your career at eBay and now PayPal. What else do I have here? The data storytelling, but also the findings from building your models is really important. And you said there’s a bit more of an emphasis on the technical training sometimes than there is on that storytelling aspect. I could not agree more. What have you done from a storytelling standpoint to round out, maybe when you hire somebody who has phenomenal technical expertise, how have you cross-trained them? Have you built your own training course? What have you done?

Khatereh Khodavirdi

We actually had an internal training course for my organization. It’s an hour, and it’s kind of very simple, but it’s two examples. For different groups or audiences, how you put your story together. Some of the mistakes that people can make, some of the mistakes you should avoid, and some best practices. We also ask people to mentor each other. You can imagine the people who’ve been with the organization. Over time, they learn their way and how they offer to scale their storytelling. And you try, especially if you have new people on the team hacking with someone else in the organization. So we can review the agenda on the story and see what is going on. But obviously, unfortunately, there are not many good resources out there on the internet for people, but…

Dave Cole

You came up with a business idea there for you and me. To build a course on that. I think it’s certainly needed. But I think one simple technique you just mentioned is just presenting to your teammates, right? Before you present to the larger audience. I know that’s a technique I’ve used time and again. Even in my past, I have pulled some members of the audience just one-on-one. You talk about partnerships and relationships with the business side. Take a couple of them, pull them aside and say like, “Hey, here’s what I plan on presenting. Do you think this hits the mark? Am I going to wow people. Am I going to land the project?” That can also be very helpful. I mean, the best case scenario is when you walk into those meetings and you get a lot of head nods, right? You get a lot of “Perfect, exactly,” and then people adding on and saying, “Well, what if we also did this too?” And you can do that if you’ve read a few folks in. I think all of that is fantastic advice.

Khatereh Khodavirdi

Two pieces of tactical advice that I usually give junior people that want to improve storytelling…one, I always say that consultants are very good with their storytelling and there is fine material, like McKinsey has a McKinsey Quarterly or McKinsey Insight website. Bain has a lot of great material on site. And I ask them that, hey, just be disciplined that, every night, just review 10 slides online from any of these consultant courses. But be critical thinking about how they are telling their story, how they are visualizing the findings, what are the key learnings for you that you can eventually elaborate on in the future. And that will become your own kind of best practices of what are the levers you have to tell the story.”

So that is one and the second one, I always ask some of my junior people, go back to some of the presentations that you prepared six months ago. The project is over, you’re no longer working on it, just bring it up, but this time be critical about what you could have done better and edit your presentation over and over. We found that these two tactical pieces of feedback help people actually build that muscle up. It becomes easier for them to tell a story and build a story around their product.

Dave Cole

Yes, 100%. That, by the way, I found is very hard to do. Personally, I’m always trying to improve the podcast and listening to episodes and trying to improve. It’s hard. Either listening to yourself or looking at something that you presented and just being like, “Man, I wasn’t even close,” or, “Why did I say that?” Or, “Why did I do that?” That’s hard to do, but it is how you improve, right? It’s how you get better. Well, this has been great Khatereh. If folks are interested in getting in touch with you, other than on LinkedIn, to ask you questions or find out more, is there any other way of getting hold of you? Or is LinkedIn the best way to get in touch with you?

Khatereh Khodavirdi

I think LinkedIn is the best way and my email address is also listed on my LinkedIn, so I’m happy if people reach out on email as well. Happy to get in touch.

Dave Cole

Well, fantastic! There’s so much I learned today. I really appreciate you sharing your tips, tricks, and advice. Wishing you nothing but the best, and thanks for coming on the Data Science Leaders podcast.

Khatereh Khodavirdi

Thanks for having me. It was a great conversation and I also learned a lot from you so, of course, thanks. And looking forward to staying in touch.

Dave Cole

Alright, take care, Khatereh.

Khatereh Khodavirdi

Take care!