There is power in a crowd, and harnessing crowd contributions have become increasingly useful in investigative journalism. Newsrooms are embracing crowdsourcing as a data collection tool, using it to connect with and gather personal stories from readers that might otherwise not have surfaced.

New York-based ProPublica, among GIJN’s largest members, is well aware of the benefits — and impact —  of crowdsourcing. It landed a story about US President Donald Trump having sold condominiums to his own son at big discounts, avoiding the usual taxes, because a reader tipped them off after digging into Trump disclosure documents that ProPublica shared.

The nonprofit newsroom also utilized crowdsourcing to expose the Trump Administration’s poor vetting process in hiring officials, and laid bare lies and misinformation written in letters by members of Congress to constituents about Obamacare. There are more examples in this tweet thread of stories and reader callouts by ProPublica, which prompted GIJN’s program coordinator Eunice Au to speak with deputy editor of engagement, Terry Parris Jr., about the nonprofit newsroom’s strategy in encouraging reader and community participation in its stories.

How and why did ProPublica get started in crowdsourcing ideas and tips and engaging readers in the story process?

Collaboration has always been part of ProPublica’s DNA, and crowdsourcing is another way to collaborate — it’s collaboration with the community. Crowdsourcing, of course, isn’t anything new. In 2009, for example, reporter Paul Kiel was asking struggling homeowners to tell their stories.

ProPublica regularly encourages leaks, reader and community participation in their journalism.

So the idea of crowdsourcing or engaged journalism has been a part of ProPublica from the beginning — from hiring smart engagement pioneers like Amanda Michel and Amanda Zamora to (just recently) building out an engagement-first team and creating the role of engagement reporter. (We hired Adriana Gallardo, Ariana Tobin and, in Chicago, Logan Jaffe for that). We recognize crowdsourcing — and have recognized it — as a smart way to do journalism.

We see engagement as a tool for journalism, and not merely social distribution or comments sections, and that requires thinking beyond merely writing tweets for a story and making sure they are scheduled. I believe that recognizing crowdsourcing and engagement as a tool for journalism shifts your thinking in how you might utilize the public FOR journalism — not just for referral traffic. We know our rolodex doesn’t always have the answer. But the public might. So let’s ask them.

How has audience engagement in your story process benefited coverage? Could ProPublica have otherwise not gotten those stories?

Yes, it’s proved to be successful. Whenever my team works on a project, we ask ourselves how the thing we are doing or might do benefits, contributes or creates journalism. We are not interested in engagement for engagement’s sake.

For example, I did a lot of comment aggregation in my time at other news organizations. Or the whole “send me pictures of your Thanksgiving turkey” callout; I’ve done that, too. There is value in these ideas. There is value in rewarding community by elevating comments or building habits of contribution. But we want to push that aside and think critically and deeply about how our engagement — the things we ask people to do or stories we ask them to share — become part of our journalism. What that looks like is more stories.

ProPublica‘s Adriana Gallardo told Storybench that there were thousands of responses to your callout on the maternal mortality series. How did you deal with such high volume of information?

When you have a crowdsourcing project that resonates with a community, like “Lost Mothers” (more than 4,000 stories collected) or “Reliving Agent Orange” (more than 7,000 stories collected), it’s impossible to go through each and every single story submitted.

Screenshot of a story in ProPublica‘s “Reliving Agent Orange” series.

Crowdsourcing, in a lot of ways, is an effort in waste. It’s just impossible to use (or even read) every single story shared. You try to read as many as you can, but it can be hard to keep up. We use a platform that helps us sort of keep up with the volume called Screendoor (paid). It’s like Google Forms with more control, customization and organization. It allows us to easily sort, segment, label and visualize the submissions that come in. We can also easily message all the respondents to a particular project. We’ve found it to be incredibly useful for capturing contributions, organizing them and really building community.

ProPublica uses Screendoor to track stories via labeling.

Don’t underestimate the need for organization here. If your crowdsourcing tool collects the stories in a CSV file, if your project has 30+ questions, those cells are going to get out of hand really quickly. Screendoor has solved for that in a lot of ways. Additionally, when hundreds and thousands of stories do come in, we are able to find trends or ideas in the submissions and pursue those. Some sort of labeling or organizing principle is key in tracking these stories. A few examples are this story on bladder cancer or this callout on IBM. But also, reporters deal with a high volume of all kinds of information — documents, for example. This isn’t all that different.

We don’t tailor the information screening process to each project. We do, however, tailor the callout or the avenue of contribution to each project based on what we know about that particular community. So, sometimes we create questionnaires to collect stories, or create email addresses for specific to projects (like daca@propublica.org or travelban@propublica.org) for folks to send stories in, or even Facebook messenger bots.

Inside each project — while we read through and organize the submissions — there will be a different organizing principle or labels or notes tailored to that project. But when we get stories we want to pursue from the callout, we approach those as we would approach any source or story: trust but verify.

When crowdsourcing, both the reporters and the engagement team work together to dig through stories. Our team is not separate from the journalism or the reporter. We work in tandem, which is why we created the role of engagement reporter: they work on engagement, they find stories and report, they work in collaboration with the reporter on the project.

How do the reporters work with the engagement team?

Currently we are three engagement reporters, a senior engagement fellow, a social visuals producer and myself. We are not a service desk but a team that works collaboratively with reporters to do the best work possible. While the goal is always the same — good journalism — people have different skills. My team spends a lot of time thinking about how the people most impacted by a story can contribute to that story. Or how the people most impacted by a story will see that story, read it, listen to it, feel it.

Reporters in newsrooms have all kinds of skills and sometimes those skills overlap: there are folks that are really good at programming, and some that are really good research, and some that are really good at sifting through data, and some that are incredible writers. My team is really good at thinking about how we connect with people — and we’re not talking about writing a single tweet or producing a 90-second Facebook video (though we are good at those things, too). But we’re really thinking about how do we connect with people where they are — beyond distribution.

Here’s an easy example: if we’re doing a story about Facebook hate speech, we probably want to connect with folks directly on Facebook. So, we made a Facebook bot. A harder one: if we want to reach folks in Liberia, to get them to engage with this project, how the hell are we going to do that? Well, what we ended up doing was creating a WhatsApp broadcast list, researching community orgs and media companies in Liberia and asking them to share our number. Here’s a write up on that.

Screenshot of joint reporting project by ProPublica and PBS Frontline, “Firestone and the Warlord.”

We’re always asking the question: who is this for and how do we find them? If we’re doing a story about Vietnam veterans, how do we connect with them and where do we go? If we’re doing a story about black women in the south who are nearly dying from birth complications, how can we make sure they are a part of this story?  We know that the answer isn’t always social media. We are thinking about this specifically, we have the skills and the expertise to dig deeply into this thought process and to execute on many platforms.

Can you talk about how ProPublica uses the call out to readers?

One reason we like Screendoor is its ability to message back to respondents. We are able to send messages in bulk to folks who contribute. We call this the feedback loop and it’s a key component of engagement. If someone contributes to one of our projects, we get back to them with updates from the investigation or send out questions that’ll help us navigate the community or inform the reporting in some way.

But, it’s not a platform that makes getting responses successful. It’s the method, it’s the story, it’s finding the community and connecting with them in the way they want to connect or are familiar in connecting. Sure, we use Screendoor, but we do a lot of research and targeted outreach to put these forms in front of people. We do not expect the folks we write about or the folks we hear from to find us.

If we approach our engagement strategy as merely tweeting from ProPublica or posting on our Facebook page, we’re not going to be successful. To truly connect with a community, to get them involved in a real way, you have to find them, figure out where they are, put your work and yourself in front of them. It can’t be transactional.

Can you walk us through the fact checking process once you receive information from a reader?

We’re still reporters. That doesn’t change. So anything we use, or publish, is verified and vetted as if anything else we’d put out into the world. If a veteran said he served on this ship and was discharged in this way, we would request to see his DD-214 form that verified that. The process is still reporting.

ProPublica also asks readers for help on areas that need more expertise and commitment, like combing through financial disclosures of 400+ officials Trump installed. What was the response to that callout?

Screenshot of ProPublica’s call out to readers to find Trump-branded hotels.

This year, we’ve had a handful of projects — mostly around Trump — that had a “help us with a task” focus. The financial disclosures project was one of them. We’ve also asked readers to share any letters they sent to and received from their member of Congress about the ACA (the US Affordable Care Act, or Obamacare) or its repeal so we could fact check them. We asked readers to help us figure out where the new Trump-branded hotels might be. We’ve asked folks to help us figure out the names of the regulatory reform task force members put together by Trump to roll back regulations.

While we’ve done our own reporting on each of these projects, there are gaps in figuring out who these folks are or what members of Congress are saying or what hotel deals may exist out in the world. In this capacity, the “help us with a task” capacity, the public is a giant body of expertise, a watchdog, a billion eyes out in the world. They can be incredibly helpful in these projects — and have been.

For the 400+ financial disclosure projects, a handful of the tips produced more stories. For the hotels project, we got a bunch of tips that we dug into that has helped us shape this story. We found that while the Trump Org has boasted 39 letters of intent (or potential deals), but we have yet to find any beyond the four we originally reported about. We don’t get there without the public.

And for that ACA letter writing project, we’ve had more than 1,000 people contribute. That’s 1,000+ civically engaged people. That’s 1,000+ who took the time to write a letter to their member of Congress. ACA will not be the only policy issue that folks will write their member of Congress about. So, that means we have a group of folks here that might help us on a number of unforeseen projects that require hearing from our government.

What lessons or pitfalls has ProPublica learned in engaging the community for stories? Any tips for other media organizations considering doing the same?

There’s a bunch of methods and pillars of engagement that I have but, really, the biggest thing is to think deeply about who the community is in the story. Who is it for? How does their voice/story/time contribute to your journalism? In turn, how are you talking to them to make your journalism accessible, usable, consumable — and ultimately a tool for change within their community. Everything starts from there.

And again, engagement is not merely retweets and comments. It’s deeper than that. Engagement, to us, is journalism.

Eunice Au is GIJN’s program coordinator. Previously, she was a Malaysia correspondent for Singapore’s The Straits Times, and a journalist at the New Straits Times. She has also written for The Sun, Malaysian Today and Madam Chair.