Title: 8-26-20-caleb-1245p-kliska
Last saved by: Cheese Itz

RAW COPY

SOCIAL POLICY RESEARCH ASSOCIATES (SPRA)

WDQI WEBINAR ‑ DATA GOVERNANCE (1289‑200)

AUGUST 26TH, 2020
12:45 PM TO 2:00 PM CST

Services provided by:
Caption First, Inc.
P.O. Box 3066
Monument, CO 80132
1‑877‑825‑5234
+001‑719‑481‑9835
www.captionfirst.com

***

This text document, or file, is based on live transcription.  Communication Access Realtime Translation (CART), captioning, and/or live transcription are provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings or used in any way that may violate copyright law.

***

>> Maureen:  All right.  It looks like it's 11:00 here on the West Coast, so we'll go ahead and get started.  
      Welcome to the WDQR webinar responsible data governance playbooks.  Just a reminder before we get started, as always live transcription is here.  Thanks very much to our partner Caption First.  The link to the live transcription is in the chat future.  
      Eduardo, if you could just add that link, thanks so much.  
      Please go ahead and introduce yourself in the chat, so we all know who's here, and then I'll just do a quick round of introductions.  
      If you move to the next slide.  Thanks.  
      So from left to right we have Toqui Ahmed at DOL.  
>> Kristin:  Maureen Sarver, Eduardo Ortiz, Katlin Grey, Natalie Evans Harris and Samantha Levy.  I just want to take a quick moment to shout‑out our team member at SPR, Edward -- he's been a critical member of the, and he's leaving us heading to grad school.  While you may not always see his face or these webinars he's the back making sure it runs smooth so thank you so much for all your hard work and your contributions, Eduardo, and best of luck in San Luis Obispo.  
      We're also pleased to introduce the newest face on this slide, Katlin Grey, and we have new WDQI teams from Kentucky and anyone from the states welcome to the SPR webinar, and you're in for a treat today, so with that I'll pass that over to you, Kristin.  
>> Kristin:  Sure, thanks, Maureen.  
      So just a quick word about a few more changes.  Teams unlikely that we'll be convening in‑person any time soon, so in response we've been introducing some new kinds of activities in these convenings and which started including a greater range of conduct in the making better work newsletter and reaching out to adjacent communities like SLDS and the evaluation and research hub and many others, and that's how we found the opportunity to partner with ODEP in our next two convenings coming up in September as part of our COVID response series.  We wanted to extend an invitation what else to do going forward.  Content for the newsletter, podcast guests we might interview and even work sessions we might even doe collaboratively to kind of advanced work on specific topics, whether specific to your state or across some number of states, so we've done that kind of thing before in‑person and increasingly online, and we just wanted to extend that opportunity once again, to extend things that are of particular interest at the moment.  You know how to find us, and we'd invite you to reach out any time.
     And then pivoting to today's session, one of the things that we've seen and spoken to several of you individually about is this shift that has occurred in the appetite for data, data vis, and all of the products that you've been working on.  The appetite for those things has really just increased since the pandemic took hold and job losses began to accrue.  
      And despite the fact that many of your teams have struggled is keep their staffs intact while so many were redeployed to filing UI claims, processing UI claims, you've been continuing to produce more than ever relevant content and closer to real time data for peers, colleagues, policymakers, new communications vehicles, and so Teams that it's even more important now to do more connecting with the broader public data and civic tech ecosystems that are helping to develop these tools and that are using the data that you're producing in new ways, and so that's part of what today's activity is all about.  
      Practically it's very much like other kinds of things we've done where we used shared documents on Google or other platforms to crowd source information and wisdom, but the content is a playbook that's intended to help kind of inform the, you know, educators and policymakers.  The ‑‑ the response to the pandemic and has the response to the underlying future of work which the dynamics in that are still at play and in many play are what counts, so these data systems that we rely on and are going to rely on for years to come seem unique and pretty special, so with that, we're super excited to turn it over to our BrightHive partners, and we have ‑‑ we have Matt Gee standing in for Natalie, big shoes to fill there, Matt.  
      And also Samantha Levy, and they'll be familiar to you, and then we also have Michelle Weise know from the concert circuit and the public world in the ‑‑ before time and also from the incredible useful and important publications and work and survey data that strata has really been seeing in recent months and something that came out, a viewpoint survey that just came out today.  We're really pleased to be collaborating with you all of you today and just welcome today and welcome to WDQI.  
      Go ahead, Sam.  
>> Sam:  Thanks, Kristin.  Just to give everyone a quick overview what we're going to be doing ‑‑ we'll set up a little bit of context, you know why we decided to create this playbook and why now and why the playbook focuses on the job‑seeker, and then we're going to open it up for a discussion and ask you a few questions to really, you know, help us develop this playbook further and make sure it incorporates your experience and perspective, so I will turn it over to Matt.  
>> Matt:  Thanks, Sam, thanks, Kristin.  I'm sorry I have to start the webinar by disappointing all of you I'm not Natalie Evans Harris, but I'm going to do my best to channel her brilliance.  This is a workforce data nerds super unit.  I was at a city the other day ‑‑ I spent years what ‑‑ in doing what felt just like yelling in the void by the importance of workforce data quality and investing in it at our state, and now suddenly people are listening intently to me and to everything I have to say, and they said to me:  It's awesome and terrifying because now I got to figure out what to do next, and that's ‑‑ to me ‑‑ them sharing that anecdote really chapters so much of this kind of now more than ever moment the need for this group and what it represents to have the ‑‑ the capacity on your teams for responsible data‑sharing and data use.  
      If you go to the next slide ‑‑ and we all know what's driving this.  Like responsible data‑sharing is now on everyone's minds.  It's in the news, in, like, top line headlines, and that means there's increased need but also increased scrutiny for all of ‑‑ all of your work, and it's really hard to work in a high‑stakes environment in normal times.  It's even harder now.  
      But if you go to the next slide, you know, what I'm sure a lot of you has experienced, which is true for the different stake partners we've talked to, everything how states are responding with testing and contact tracing so unemployment insurance and ‑‑ and retraining and re‑skilling initiatives to resource coordination; right?  All these different use cases that are critical for ‑‑ where states are playing a critical leadership role are ‑‑ have at least in one state that we know of, for each of these different use case have had to touch one or more of the core workforce datasets, and ‑‑ and so the ‑‑ these critical use cases are ‑‑ are ‑‑ are often stretches, right, for new uses of data that you as individuals and organizations are stewards over, and it stretched both your capacity but also the sort of patterns and plays you have to work from in being able to put your data to new use.  
      If we go to the next slide.  
     You know, things we've heard from many of you and other key partners is that the COVID crisis has accelerated interest in your data.  It's focused people's attention and desire for ‑‑ for helping out, which is great.  You've got wonderful initiatives like the U.S. digital response initiative where that they're knocking on your doors and saying:  We got folks ready to volunteer to help you out; where could we deploy?  And that's wonderful, but it also takes time to manage that kind of work, so that accelerated interest from the volunteer side, from philanthropic interests and from ‑‑ from largely tech providers and colab providers is great to see in workforce data but also comes with a whole host of challenging issues and resource restraints that are real.  
      And for things that state leadership are saying yes to; right?  You're having to figure out new kinds of data partnerships and join new kinds of structures and data collaboratives intrastate ‑‑ we've seen these different modalities proliferate, which is really exciting which, you know, again, has its own sets of challenges.  
      And a third thing that we've heard a lot from you and from other partners is that a lot of the kind of typical patterns that you have from gauging with these other partners, whether they be other agencies or outside entities like philanthropies and private vendors be the bilateral MOU model is ‑‑ is breaking down in a lot of context, and it's making it hard for you as agencies and in your state leadership to accomplish the goals that you want in the time that you have, and you're having to rethink data governance on the fly.  
      And all of those things are happening at once, on top of you having to do your day job, so there's ‑‑ there's bucket loads of empathy on this call given ‑‑ given all of that.  
      And Natalie ‑‑ as we were talking internally about how we, you know ‑‑ how we could best help out the community and the broader community, she looked back on a similar moment that she experienced when she was putting together the federal data cabinet, which was the chief data from officers each of the federal agencies when she was at the office of science and technology policy, and they similarly had come with a bunch of similar, you know ‑‑ different ‑‑ different kinds of crises but clusters of challenges around ‑‑ across all federal agencies needing to do digital transformation of the services that they offer and ‑‑ and the chief data officers for each of those agencies struggling with what their role was in that digital transformation at the federal agencies.
     And one of the things that they were able to do in response to that need was that U.S. digital services created an open source playbook ‑‑ now you can go to playbook.CIO.gov or you could just Google it they created a playbook aimed at federal agencies that was really a set of plays and checklists to say:  We know you're having to navigate digital transformation and responsible data use within your agency, here are some things that we've learned ‑‑ that different agencies have learned and pooled their learnings together into a course of play that folks used, and she pointed out and said can we do the same thing for a community in the workforce agencies and the digital transformation that everybody is having to undergo right now, and do it with a set of partners who really know their stuff?  And so that's what we aim to put together.  
      If you go to the next slide.  We worked with Michelle and Allie bell and other key partners and stitched together a first draft of an open source playbook that's available on GitHub that is aimed at helping ‑‑ start the conversation in this community for what the core plays are that helping the job‑seeker across the state or moving across your state to get the data how to do it responsibly and do in a way that keeps them at the center.  
      And our goal is to have this be a living, breathing playbook.  That we could get it started, but we could have it kind of grow in the set of use cases and plays that are meaningful and useful to you and other community members.  
      So our goal for today is to kind of share out what some of the ‑‑ those plays are, some of the core principles and plays are that the initial set of partners thought up and put this together.  Get your feedback on whether or not those are helpful and get your input on what's missing?  What have you learned from the work that you've been able to do today that you think could benefit other states and localities and what ‑‑ what you'd like ‑‑ what you need in plays that might be missing.  
      If you go to the next slide, and then I will stop talking and hand it over to Michelle.
     So this is just a high‑level overview of the job‑seeker playbook.  It has 4 guiding principles around job‑seeker centeredness and keeping their experience at the center, equity, collaboration and sustainability.  Some very experience playlists that will over ‑‑ you get a sense for its general structure.  I think we've already shared up ‑‑ it follows it is exact structure we just forged the GitHub that the U.S. digital services put together, so you can see we have the exact same structure as the U.S. digital services playbook.
     So that's a little bit of a preamble.  What we're hoping to be able to contribute and have be useful and do I now pass it over to Michelle or do we open it up for any questions on kind of background?  
>> Sam:  Let's pass it over to Michelle who can cover why the focus of job‑seeker and how the broader questions, and then we can move over to the broader questions.  
>> Matt:  Take it away, Michelle.
>> I'm Michelle Weise and it’s my great pleasure to work in an advisory role with the BrightHive team.  
      I just want to share a little bit of research that might be relevant.  I think it speaks to all the work that you do on a daily basis.  I thought it would be kind of national from a national perspective hi used to lead the Estrada Institute for The future of work, and one of the things prior to COVID that we were sort of obsessed with was this idea on the screen of this uncommon unevenly work.  A lot of people have been different change, exponential futures and all these kinds of anxieties about the future of work, and what was fascinating was just to sort of see what has been going on even since the last recession really in 2008 and that deepening inequality between top 1% who really captured 85% of the post‑recession growth, that economic growth and seeing how far behind, however, being left behind are the folks who are in the bottom quartile of the worker and learner population?  These are folks who are earning less than $35,000 or an individual less than 75,000 is the family or, you know, have very little beyond a high school degree and as we think about this very uneven distribution of that kind of wealth and opportunity and access to advancement we engaged in over 100 one‑hour interviews with folks from this particular sort of part of our population and if you go to the next slide, Sam, one of the things that we kind of tried to capture were these sort of recurring themes and obstacles that gets in the way of a job‑seeker's pursuit of moving ahead and as you think about just even within the context of a state, right, we think about all the different kinds of innovations going on in Ed tech alone when it comes to workforce tech between 2015 and 2018, we saw this kind of ballooning of a 2.9 billion dollar marketplace of over 250 companies that were funded specifically around workforce tech and yet we still haven't seen that kind of moving of the needle of folks who really need the most help launching into better opportunities.
     And one of the things that kind of became clear was for 41 million Americans, they were not able to kind of actions a seamless way of navigating a job transition and what that involved was the stitching together of career navigation services, ways of identifying kind of skills gaps that they might have and how to fill them.  How do they know what's the right kind of educational path to take?  How do they trust it and how do they know an employer might be able to validate that opportunity so being able to navigate ‑‑ we kept hearing from learners they needed a roadmap an education GPS some sort of skills compass.  They kept using this diction of mapping around the world and, obviously, for adult learners from 25 to 64, they have a very different set of issues that they're dealing with than a traditional college‑growing population, and so the kind of wrap‑around supports are needed to be successful were very different and as you think about the need for faith‑based organizations and community‑based organizations and workforce boards to come together in new ways, there's real opportunity there.  
      When we think about educational pathways there's very little beyond sort of the certificate pathways and the 2‑year to 4‑year degree programs especially in the time of the pandemic where people are struggling to survive, we see how inadequate our workforce education is facilitating just in the bite‑size pieces and the competencies to upskill or to move to a totally different domain or industry.
     And then, of course, we have the challenge of not only the cost of learning, but the cost of time, right, time is the most sort of precious resource of adult learners.  We don't have great ways of carving out time for people to upskill and skill up and transfer their skills into new domains, so we kept hearing from learners this real kind of barrier that comes into play, and that's what we mean by kind of integrating earning and learning.  There's no work‑based on the job opportunities to ‑‑ to advance clearly.
     And then you heard a lot kind of in the air ‑‑ in the ether around skills‑based hiring, but we just don't have really great mechanisms for fair and transparent hiring and, obviously, for many states, we have pockets of the innovation going on.  But again, but from the perspective of the job‑seeker, how well do we stitch all of this together in a way that is truly easily accessible and navigable and made to ‑‑ to serve the job‑seeker specifically?  
      So if we could go to the next slide, when we think about ecosystems and the state or even a city, we can think about ecosystems at different levels and the reason why I love just sort of thinking about the ecosystem as the work ahead that we need to build, as we think about the potential for our lives to extend in terms of our work lives just getting longer, we're ‑‑ we're extending our stays in the workforce and that accelerated pace of change.  We already knew prior to the pandemic we would have to kind of navigate more ‑‑ more job transition than ever before, but we just don't have a great infrastructure to take those on and off ramps, in and out of learning and work.
     And one of the things that I find fascinating as you think of an ecosystem and just a natural ecosystem you think of just the roots of trees.  There's this lovely forestry researcher who has this famous TED Talk talking about when you step on a forest floor we don't realize the kind of iceberg density of ‑‑ of this kind of maze of fungal networks going on that connect all the roots of trees in that area so, you know, a fur, a Douglas fir can be connected to a pine and what that they're doing is communicating because their root systems are connected by these things called mycorrhizal networks, so when you step on the surface of a forest floor what you don't understand is that there's hundreds of thousands of kilometers of that kind of ‑‑ that fungal network that you're ‑‑ that you're standing on.  
      And what that actually leads to is this kind of near intelligence where, you know, a mother tree can actually send more nutrients to a seedling tree.  There's this kind of amazing transfer of nutrients that goes on in this ‑‑ in this kind of below‑ground context, and if we just kind of translate that same metaphor to what we have to deal with when we're trying to connect our post‑secondary education, our career technical education programs, our workforce development programs to one another, right, we don't have that connective tissue.  We have so many silos, we have so many datasets that do not speak to one another, right, we're speaking multiple languages plus we have the language of the job‑seeker, the learning provider and the employer which aren't connecting.
     And so as we think about the work ahead, I think this kind of give us something wonderful to aspire to.  I think in Utah there's actually this set of 47,000 Aspen trees that are connected by one 13 million‑pound root system, and it's kind of amazing to kind of think about that natural intelligence.
     And you see this also happening, right, in other industries where Amazon, Google are closing these closed set data systems that are always getting smarter about our habits, about our consumer behavior.  Sometimes in scary ways, but, you know, there are ‑‑ there's so much work to do ahead as we ‑‑ as we think about data use.  
      If we go to the next slide.  
      So as we think about ‑‑ this is job‑seeker ‑‑ this is play No. 4 from the job‑seeker that we developed.  We ‑‑ we all kind of collaborated on this with kind of a deep sense of humility knowing that was just kind of the starting point, and we hope to ‑‑ we hope to build on it and make it stronger over time.  
      When we think about this idea of a ecosystem pulling all those services and innovations, obviously, no one group is going to be able to do it alone so how do we actually think about behaving differently together and collaborating in new ways 'cause this does require a very different set of behaviors and incentives.
     So I'm going to give you all just a quick minute to read through ‑‑ just to remind yourselves of what this job‑seeker, play No. 4 is.  
(Pause.) 
>> Michelle:  All right.  And if we go to the next slide, we're going to start moving into a much more interactive, less talking at you space.  
      We want to share with you what we initially put up as our checklist items, and we'll have you kind of react to them.  We're going to use a special tool called Mentimeter and hopefully it will work out and if it doesn't we have a backup plan, but we're first going to have you take a look at that.  Why those responses came out and what we may be missing, and we'll do the same thing for the key questions that we ask in the job‑seeker.  
      Why don't we take a look at the checklist, and then I'll show you how to get on Mentimeter.
     So this is the checklist, and we're talking about the governance structure that may be needed.  It's got to be somewhat flexible enough to adapt to future needs, but it can't be too constrictive.  We're talking about shared ownership, making sure people can identify how you control the data.  We're talking about the job‑seekers' perspective, how do we include them in these kinds of data governance processes.  How do we think specifically about a very diverse governance body, and then how do we know ‑‑ how do we make sure that this data will actually help the job‑seeker directly?  
      So what we're going to do now is ‑‑ if you could all just go to the link that is Menti, menti.com.  We're going to put this ‑‑ yep.  Someone just put it, so it's menti.com, and you're just going to put this code number that's on our current slide, which is 5815510.  You don't actually need to space it out.  I don't think it matters.  You can just put the numbers in, and you are going to be brought to our ‑‑ a page where it's got kind of almost Likert Scales.
     And so what you're going to be able to do in this space now is actually move the dial in response to each of these points on our checklist to say from your own perspective:  Do you actually see this as a priority or not?  And over further to the right, so you believe kind of more of an essential priority, less so to the left, so you should be able to physically manipulate each of those slides.  
      And go ahead it and just take two minutes to think ‑‑ 2 to three minutes to think about each of these, and then, hopefully, some of you can actually share ‑‑ share why you actually rated these in this way, and we'll kind of take a look at the collective response.  
      Is anyone having trouble getting on to Menti?  
(Matt humming the jeopardy tune.) 
>> Sam:  No pressure.
(Laugh.) 
>> Michelle:  I don't think anyone touched it yet, right, Sam.  
>> Sam:  We'll be able to go through the presentation ‑‑ just waiting for our first ‑‑ our first input.  
(Pause.) 
>> Matt:  Should this be going up 'cause I know I already put stuff in.  Should this be updating?  
>> Sam:  Yes, as people answer via menti.com, it should be updating on our presentation.  
>> Matt:  It may not be doing that 'cause I know it should have gone to 0 to 1 at least.  
>> Sam:  Oh, interesting.  
>> Matt:  It looks like we've got other folks who have submitted.  
>> Sam:  Okay.  I wonder if this is a screen‑sharing issue.  Interesting.  Okay.  We may have to go to our backup plan.  
>> Matt:  All right.  So while we figure this out, if you ‑‑ oh, wait, actually, it's moving ‑‑ it's a screen‑sharing issue, so it's moving on the ‑‑ and if you want to, like, unshare, and then share again, it is actually changing on ‑‑
>> Okay.  Let me do that.  Maybe it'll update.
>> Yeah, 'cause I'm seeing on our slide deck.
>> Sam:  Okay.  Hold, please, everyone.  Sorry about that.  Ah, yes, okay.  So let's do this.  
>> Michelle:  It looks like we got about 24 responses.  
>> Sam:  Cool.  Okay.  Bum, bum, bum.  Let me share my screen again, so you all can see.  
>> Michelle:  And then just keep your tabs open because which we have another one of these interactive pieces it'll just ‑‑ it should automatically populate the tab that you opened for Menti.  
      All right.  It looks like identifying someone who specifically represents job‑seekers and including them in the decision‑making process seems to be less of a priority than one of the other pieces.  The most significant or essential being the first.  I would love to have anyone just kind of maybe just talk about that that fourth one if they could 'cause it's fascinating 'cause we're trying to ‑‑ we tried to make this handbook and the job‑seeker specifically with the job‑seeker at the center, and I'm curious what maybe complicates this part of the ‑‑ 
>> Matt:  Is it complicated or is it easy?  In our application the person who speaks on the job‑seeker data governance are the program monitor.  That they're the ones for delivering the services, for ensuring the outcomes, for gathering the data, therefore, they're the ones who are the closest to the voice of the customer.  
>> Matt:  So you have ranked the lower, not that it's an essential priority, but it's something you already got covered ‑‑ or that you feel is built in to governance as it stands for in Texas?
>> Adam:  Yes.  
>> Matt:  Is that accurate?
>> That is the standard we go to division directors when we're looking for new uses and things like that.  
>> Matt:  Is that the case for others who also thought about this question or did anyone kind of think about it slightly differently?  
(Talking Simultaneously.) 
>> Michelle:  Yeah, someone said No. 4 is less important than actually talking to the job‑seekers themselves.  
>> Matt:  One of the tensions I've heard around this issue from talking to the number of folks who are thinking about kind of voicing an inclusion in power structures in data governance boards is that the conversations at a data governance board meeting are pretty wonky.  They typically don't relate to the day in the life of ‑‑ of a typical job‑seeker, and so finding that person that can both authentic ‑‑ like is closed to, and authentically can represent the needs of ‑‑ privacy concerns, communication concerns of job‑seekers within a state or region and also can act as the impedance match how that relates to specific data issues and data decisions.  It can be a hard ‑‑ that can be a hard dual role or bilingual a person to find.  Bilingual in that sense.  
>> Michelle:  Yeah it sounds like Shelly was kind of saying balancing the wonkiness with the perspective of a job‑seeker is kind of that win‑win potential balance to the situation.  
      I do want ‑‑ Kristin, your point about priority versus urgency is an important one.  I'm curious as you think about ‑‑ especially the real sort of prioritization of the first the third and the final pieces of this checklist, if anyone could kind of speak to how urgently you feel like you need to engage on, you know, an element of this checklist?  
>> Kristin:  I'll jump in, Michelle, I thought the last point in particular is really ‑‑ like there's a lot packed into that point, and it's sort of suggestive of multiple processes, and so that just seems to me, like, a very ‑‑ you could really work a lot into that point.  It's a deep process point.  
>> Michelle:  Uh‑huh.  I wonder if it makes sense to pull some of those pieces apart, so that it maybe isn't quite so much all in one.
>> Adam:  It also feels like a goal or end product than a precondition or foundational issue of governance.  It's like, okay, once you have governance, then you're doing this thing with it, so it almost feels like it shouldn't be part of this package.  It's not that it's not important.  It is.  It's just that isn't the end, not a beginning.  
>> Matt:  That reminds me of ‑‑ I mean, one of the things the last one mentioned ‑‑ a term that's used a lot in the industry and thrown around a lot in industry is agile data governance.  It's applying the ‑‑ some of the playbook of the agile playbook and thinking data governance processes other than ‑‑ to Kristin, your point that you made in chat, and I think this ‑‑ you know, this last one really kind of lists that up ‑‑ lifts that up in the way we think of data governance, and it's focus in inclusion and its role in data product development is changing to keep up with some of the increasing dynamism that we're talking about.  
>> Michelle:  It's interesting in that sixth one as we think about sort of the aspirational goal of making this, you know, affecting job‑seekers directly in a different vein of work in this sort of rapid scaling attempt there's a group called Skillet Coalition that's been trying to also stitch together some career navigation and some education and connections to employers, and they were really focusing on those who were most impacted by the pandemic, and they built out 3 pathways okay.  Here's how we're going to stitch together all these things and make it really accessible to this target population, and the immediate feedback that they got from the learners is actually those pathways, those educational pathways seemed intimidating, so even though they'd done kind of all the work of building the systems to speak to one another, connecting all the stakeholders together, what they kind of forgot to do was actually do enough of that user testing to find out, oh, this is actually ‑‑ this is not actually filling the job‑seekers need, so it is interesting to sort of think about how ‑‑ how we ‑‑ how we center on the job‑seeker.  How we make sure it's tailored to them but also how we think about the process goals of this work as well.  
      Sam, I wonder if it makes sense to go to the next slide because I think we can maybe get at some more of these issues.  
      So as we think about, you know, this, which I say, we'd love to hear from you 'cause again we're trying to make this checklist more useful and actionable for more ‑‑ for more stakeholders, what is not reflected from your own experience doing this work?  What do we need to be identifying?  What should we consider to identify the right data, the right partners, new tools to use so?  ‑‑ so, Sam, is this a new slide that should be coming up for folks?  
>> Sam:  Yes, so same menti.com link and code, and it should switch to this slide, so that everyone can input some answers.  
(Pause.) 
>> Sam:  And I'm just going to stop sharing for a moment, so I can give everyone a minute or two, so we can see all the responses on the screen.  
(Pause.) 
>> Michelle:  For those of you who aren't seeing ‑‑ you know, who are kind of in the more not applicable of this response, feel free to just chime in and tell us what you're thinking.  We're actively taking notes.  Matthew Matthews and Sam, it looks like we have a request to show the original survey results while we're waiting for folks to respond.  
>> Sam:  Oh, yes, okay.  
>> Matt:  Anything that we can do?  
>> Sam:  Oh, yes.  Absolutely, so these are the inputs that we've gotten so far.  
>> Michelle:  More human‑centered design and data matching right partners, so we can actually figure out what information is needed and useful.  MobileFirst decision support tools, and then communication between partners ski, getting them ‑‑ is key, getting them involved, engaged, committed and provide a secured data share environment, and then you can go forward.  
      Okay.  There's a few more, Sam.  
>> Sam:  Okay.  
>> Michelle:  One is enjoyed book for ‑‑ guidebook for state and federal laws governing data policy use, and policy and state leadership ability to lead these discussions; make data partnerships easier to establish by training leadership and policy staff on the terminology and concepts of data‑sharing, data stewardship.  
>> Matt:  Those are really great suggestions.  
>> Michelle:  Can anyone actually used some significant kind of human‑centered design or user ‑‑ different kinds of user‑design approaches that they want to share?  
>> Matt:  Yeah, I'm really curious about the person who future in calling out human‑centered design in data matching.  That's an area where I've yet to see human‑centered design applied to.  I'd be very curious here what folks were thinking on those.  That's a great suggestion.
>> Kevan:  Yeah, we did that in Colorado.  We worked with different state‑ and local‑level programs as well as working directly with customers kind of the higher levels, you know, the higher level, kind of the more management level, and then the front line level and Evan kind of the customer level in order to kind of determine what the real needs were with that we need to match the data, and then we'll figure out how it's going to be used, you started with let's ‑‑ let's speak to folks who are involved and allow that to prioritize which data we should actually match?
>> Kevan:  Correct.  
>> Matt:  That's great.  That's a great example.  
>> Michelle:  Did that sort of undue any sort of initial approaches you thought you were going to take, and then you had to kind of do differently because of that research?
>> Kevan:  Absolutely, you know, I think what you can probably ‑‑ probably the worst place to assume anything from is my level working at the state as a data person because ‑‑ I mean, I had worked at the local levels.  That helped a little bit but really what you assume at kind of higher level is usually incorrect so being able to go and actually find out what both frontline and what actual people ‑‑ that they're trying to access this ‑‑ this services is absolutely essential.  
>> Michelle:  That's great.  
      Anyone else want to share their addition here?  That we haven't touched upon?  It's interesting because that, Kevan, that you just discussed seems to be somewhat different from ‑‑ I can't see the full ‑‑ it's the pink box on the right, Sam.  Okay.  The scope of what you're trying to govern drives answers to all other questions, right, to find the problem and goal first and the rest will follow, it does seem to be a slightly different approach than what you just described, Kevan, I think.  
>> Kevan:  We kind of started that way but once we started the process we had to do a lot of adapting and redefining.  This was started as a center designed project which, unfortunately, the one we started was much bigger than the other ones and took a lot more time and a lot of relining, you know, who your group was at one point completely changed based on your ‑‑ you know, where you were moving to so yeah, we kind of let the human‑centered design idea dictate the work, which was a really interesting and new way to go from me, but I think it had good results.  
>> Michelle:  Uh‑huh.  
>> Matt:  I'm kind of ‑‑ just to keep on this topic, if you don't mind.  Hit on something which someone else brought up, which is equipping leadership with the, you know ‑‑ both the terminology and the cognitive tooling to ‑‑ to be able to approach things differently, going ‑‑ speaking to your experience and some of the other stakeholders and leaders that you had to work with in Colorado to do that, you know, what was ‑‑ how much of your work was also kind of educational; right?  Training the other folks involved in how to do human‑centered design and how it would feed ‘back into the data mining process?  
>> Kevan:  There's a lot of a lot of there was a big push for the human‑centered design, so it wasn't so bad.  It was more at the other levels that some education was needed, but I think at both the state and local level there was some of that, so I think it was very helpful.  A lot of it was just kind of keeping the process going, you know, and how do you deal with the big picture out in space stuff and did and realities kind of combining ‑‑ I mean, it was just a long time‑consuming process that required a lot of re‑evaluation and changing of things, but I think in the long‑term it turned out very good, and we did come up with a largely‑scale recommendation that was submitted to, basically, the state leadership.
     But the exciting part, it was also then re ‑‑ we were able to reutilize that and what we're doing with WDQI, so a lot of that legwork was already done well, okay, now we actually get a real life chance to utilize this, so that was pretty exciting.  
>> Matt:  That's fantastic.  
>> Michelle:  Why don't we go ahead and move to the next one, Sam.  
>> Sam:  Okay.  
>> Michelle:  We're quickly running out of time.  
>> Sam:  I'm running if we should dig into the key questions or maybe pose one of those discussion questions about collaboration and incentivizing partnerships?  
>> Michelle:  Yeah, Matt, do you want to take that one?  
>> Matt:  Sure, yes.  I think ‑‑ let's dive into answers for this question.  Is it now teed up?  
>> Michelle:  Yes.  
>> Matt:  Awesome.  Do we have to refresh the browser at all?  Nope.  Excellent.  It should be or there.  
      For the last point ‑‑ we want to think about, again, targeting plays that we could add to this playbook or things to the existing checklist that we could add that would help ‑‑ if you could rewind six months, right, to obstacles you ran into and figured a way around and if someone handed you a playbook to say ‑‑ you're going to run into this obstacle try X, Y or Z out or it could have saved you six months, 6 weeks or even 6 days.  All that time is pressure.  I love for example' ideas here around those obstacles and how you successfully incentivized around and across your partners' collaboration among agencies, among agencies and their partners?  And hopefully me rambling the question will give you enough time to hopefully put answers in.  
      I'll ramble about one more thing while we're giving folks time, and that is Adam in the chat brought up a really great point about capacity, right, as a major obstacle, and I'm really curious ‑‑ I'd love to see in the chat or in your responses to this question in Mentimeter how you bridging your gap in the tools and dealing with so much of you and your staff's time does have to go to reporting and compliance, and that's ‑‑ that's must‑do work and a lot of the things we're talking about today, at least historically would be great to do and work ‑‑ and maybe the calculus has changed a little bit with shifting state priorities, which is great, but I'm really curious how folks respond to kind of Adam's obstacle challenge of just capacity within agencies for being able to do this work?  
      Okay.  And Michelle, do you want to go ahead and field the discussion of some folks' answers?  
>> Michelle:  Yeah, I love this first one.  Everyone thinks their data is special.
(Laugh.) 
>> Michelle:  And the idea that people aren't kind of willing to give to get it sounds like this first one attacking about.  
>> Sam:  I'm just going to update very quickly.  Matthew Matthews Kevan brings up a great point here that you have to have ‑‑ you have to have ‑‑ it really helps to have kind of high‑level leadership ‑‑ yeah.  
>> Michelle:  And I think it probably goes to that challenge of also having to socialize with the policymakers and the state leaders, that concern that came up earlier in order to get that covered we have to be able to do that socialization well, so that people understand the importance of this ‑‑ this kind of connectivity sandboxes is also a potential activity there.  
      So a real concern about data ownership and others use of your data ‑‑ data costs, director buy‑in, director, bureau, buy‑in for need.  What can be provided to other areas during the project that they could use or desire once leadership has bought in, and then obstacles from staff are removed from the table?  So again, how do we get the leadership bought in seems to be a recurring theme to facilitate this kind of ‑‑ yeah.  
>> Matt:  I'm curious if thinking of additions to the playbook, have any of you ‑‑ in the internal sales process you have to sell it to leadership.  Have any of you successfully benefited from materials or stories that other states or other exemplars have been able to produce and that has been key or at least helpful in getting your leadership on board?  
>> Sam:  Teams like along those lines are questions of trust coming up, how do you actually build that kind of trust.  I love this line where the problem is almost never a technical problem that keeps the data separated.
>> Adam:  Yeah, that's really well put.  
>> Matt:  It looks like we're coming up occupy time ‑‑ the obstacles are the lawyers.  
>> Michelle:  Did you say the obstacles are the lawyers?
>> 
>> Kevan:  I'm not joking.  If you talk to the federal partners, they'll say the same thing.  They killed us on overregulations and all the ICRs and everything else.  They just ‑‑ we ‑‑ a lot of places have what I would call bad legal culture so there are two ‑‑ there are two ways as a legal ‑‑ as an attorney or a person in a general counsel's office or legal services offices there's two cultures.  The first culture is to get you ‑‑ is that they just want to get a quick answer that covers the agency's butt which, basically, means the answer's always no, and the other culture is the one that says:  It's the attorney's job to try to find a way to get to yes.  Like they are ‑‑ I'm not saying they should be bending stuff or anything else but ‑‑ but if there are many interpretations, if there's a way to clear a path to allow something to happen, if that's what they see their mission as to support the business owners to in the law, and then you have room, and then the problem is, again, as our federal partners will talk about, when you don't have the same culture or the same priorities amongst the attorneys between the different organizations, you end up with cultures ‑‑ you end up with ‑‑ well, let's just say friction and ‑‑ and obstacles.  
>> Matt:  So we may want to have plays targeted at lawyers and convince DOL to create a workforce lawyer data quality initiative?  
>> Kevan:  Cost‑effective well, to publish 70 pages with the Department of Education on ‑‑ I think it was 17 different ways to share data between education and workforce agencies and still I hear about problems out there.  
>> Matt:  This has been great.  I know we're at time.  
      Who's ‑‑ who's sending this off?  
>> Michelle:  Sam knew is that you?  
>> Sam:  I think it's all of you.  
>> Kristin:  Maureen, jump in.  This has been super, everybody.  Thanks so much.  I really, really appreciate your participation today.  I can't wait to read the chat.  
     Eduardo, be sure to grab the chat.  Let's do this again soon.  Everybody have a great rest of your day.