* * * * * *
EHDI Annual Conference
Exploring EHDI Data:  Unveiling Novel Parameters for State-to-State Comparison
March 17, 2024
1:15 P.M. - 4:30 P.M. CDT
* * * * *
This text, document, or file is based on live transcription. Communication Access
Realtime Translation (CART), captioning and/or live transcription are provided in
order to facilitate communication accessibility and may not be a totally verbatim
record of the proceedings. This text, document, or file is not to be distributed or
used in any way that may violate copyright law.
* * * * *
(Captions provided by a live Captioner.)

	>> Is this microphone working okay?  Okay, great.  If everybody could please sign in at the back if you haven't already done so.  I think I had one person who hasn't signed in.  And then I would like to introduce Exploring EHDI Data:  Unveiling Novel Parameters for State-to-State Comparison.  I'm Sonya Richardson I'm the moderator if you need anything during the session please let me know and we have Keisha and Veronica and Marie to present here for us today.  Thank you so much.  
	>> All right just some housekeeping.  I think most of you -- I think there's a couple new people we will do live website exploration and hands-on so if you can -- if you can connect to the hotspot, if you can do that, that would be great.  Does anybody need a device?  I brought on extra iPad.  
	Okay.  Great.  
	Anything else housekeeping-wise?  
	All right.
	>> So welcome.  I'm so excited to be here.  
	This has been the culmination of a lot of work.  So we're finally launching this website.  So it's very exciting for us.  
	My name is Marie Richter.  I'm an audiologist.  I'm the Co-Chair -- no, I'm the Chair for Missouri Newborn Infant Hearing Screening Program Standing Committee.  I used to work at St. Joe's NCID in St. Louis.  I currently just kind of consult in different projects.  So I got pulled into this one to be your EHDI guide.  
	Because Veronica and Keisha know a ton.  But they don't know EHDI.  And I know EHDI.  And I have no clue what they do with websites and data.  
	So I'm going to cover the introduction part of the session.  
	Veronica is going to cover like all the details of definitions and understandings of the website.  
	And then Keisha is going to walk you through it.  And then I'm going to let them introduce themselves.  
	>> I'm a little too short for the mic and for the laptop thing, sorry.  
	Okay.  I'm going to rotate this for a moment.  Can everybody see me okay?  
	Okay.  Awesome, thank you guys for joining us today we're really excited to show you this presentation I'm Veronica I'm a senior data analyst with the Center for Public Health Systems Science at Washington University in St. Louis.  At the center we have three core areas of focus.  
	Policy and prevention, dissemination and implementation and system science some project background I have is in tobacco control and opioid use and prevention and treatment in rural communities.  
	So I really enjoy getting into the data.  And like diving into that data management aspect.  
	So it was a really nice connection with this project.  And then also at the Center we really enjoy exploring the different ways to take the data and transferring it to a variety of different audiences so that was also a really exciting opportunity with this project as you'll see in a little bit.  
	Our goal here that we want to reiterate is to help you visualize the story so we hope to guide you through that today and then love to get your feedback.  
	Nice to meet you all.  
	>> I think I'm visible.  
	Am I audible?  Yes.  
	Okay.  So I'm Keisha Kumar I'm a Master's in public health student graduating this May and with specialization in bio stats I'm working as a graduate research assistant at center of public health system sciences I'm doing this project because this is a project associated with infants and it allows me to apply my learning to a meaningful cause.  
	In addition to this project, I am also involved in translation and implementation research where I led a clinical study to investigate association of amino acid energy and peptides to the breast density while in India as a Medical Officer I supervised a screening program for oral, breast and cervical cancer in a 4 million population.  So excited to see you all.  Nice to be here.  Thank you.  
	>> All right.  
	This is throwing me off.  I don't like being behind podiums.  
	>> Does this pop out?  
	>> Yeah.  
	>> All right.  
	Is it still working?  Bababa.  
	Yeah.  
	Okay.  
	All right.  
	So can we real quick just go around and can you tell us who you are, where you work, like associated with EHDI and we have a prompt.  Do you mind starting.
	>> I'm (inaudible) from Atlanta, Georgia I'm (off microphone).  I'm always trying to learn more about the data and how to collect it.  
	>> I'm Dan I'm the Illinois Chapter Champion I'm a pediatrician also a father of three boys, two of them are Deaf.  One, the oldest one, 21 --
	>> Do you want to restart.
	>> Oldest one 21 years old.  Graduated from college.  He's moderate to severe loss.  
	And my youngest is 17.  He's severe to profound.  And my oldest has hearing aids.  My youngest has one CI and yeah, I've come to this conference since 2015.  And she was my audiologist.  Yeah she was Luke's audiologist.  Hi.  We've been through numerous but yeah.  
	>> Hi, everyone, (inaudible) I'm from Taiwan I work at Children's Hearing Foundation which is a foundation we provide auditory very balancing therapy to children in Taiwan and I'm a research fellow there, thank you.  
	>> Hello my name is Kristin Cruise I'm the newborn screening and birth defects programs manager with the Virginia Department of Health.  
	Why did I sign up for this session?  I signed up for this session because data is a passion of mine.  
	And there's a lot of crossover between my programs, including EHDI.  And we're looking to see how we can enhance that and Parker Parks just stepped out to use the restroom but she's my newborn screening epidemiologist also from Virginia.
	>> Hello I'm Roxanne I'm Director of Michigan Department of Education low incidence outreach we serve blind visually impaired and Deaf and hard of hearing we work with EHDI closely we were awarded the LEAD-K financing to implement that so we're hoping to make some connections today so . . .
	>> I'm Judy Jungle from the Wyoming Department of Health and I'm the Part C coordinator and also the Project Director for the two EHDI grants.  
	>> I'm Jennifer Petri I'm the Part B coordinator for Wyoming.  
	>> I'm Jolene Floors I also work with the Wyoming Department of Health and I'm the data manager for the early intervention and education program.  
	>> I'm Kim Taylor with the same program and I'm the early intervention and Program Manager.
	>> I'm Marsha Forte I'm the EHDI Coordinator in North Carolina.
	>> I'm Linda Hazard I'm the EHDI Coordinator in Vermont as well as the Director for our Part C program and most of our Part B programs for Deaf hard of hearing and deaf-blind.
	>> I'm Tracy I work with the EHDI Program, the data.  And I'm sitting in for Kim Scott which is our EHDI Coordinator for Arkansas.  
	>> Hi my name is Heather I'm the Executive Director at Central Institute for the Deaf in St. Louis, Missouri I'm faculty at Washington University in St. Louis.  I'm always happy to be here in St. Louis I'm here to support colleagues and former colleagues but I'm interested in data related to birth to 3 outcomes.
	>> That's a great answer and that's why we're here, too I'm Jennifer I'm with the Statewide Outreach Center at the Texas School for the Deaf we're legislatively tasked to work with EHDI to make sure that families who have identified children are making sure they get into a Part C.  In addition to that my co-worker here, she does all the things.  
	>> I'm Shelley also at the Statewide Outreach Center I do intervention care coordination so I work in the EHDI center in our TEHDI system to be able to work with families and do data analysis so I'm really excited to hopefully learn something here and work with our Loss to Follow-up Loss to Documentation I also have two sons who are Deaf.
	>> I'm Kristin Birch I'm an audiologist working at Central Michigan University I signed up for the session because I was curious about state-to-state outcomes with EHDI.
	>> Hi, Solomon from Boston, Massachusetts.  I'm the AP epidemiologist for newborn hearing screen and I'm interested in improving quality of data and at the same time we can present visually or otherwise to policymakers and other practitioners and advocates, as well.  
	>> Hello I'm Liz.  I am an audiologist and I am the coordinator, the EHDI Coordinator for Kansas.  
	>> And I'm Kristin.  I am also an audiologist and EHDI Coordinator in Minnesota.  And all things data.  Data drives change.  So . . .
	>> Thank you.  
	>> There's your learning objectives, some learning objectives.
	>> There's a couple that we put up.  So we're here today to help you learn to do them.  You don't have to really pay attention to those, we'll come back and here is just a teaser from the website.  A graph.  And the square is the interaction.  You can hover on a point and see what you're looking at.  
	Sorry; that's all you're getting for now.  You've got to listen to the background stuff, the boring stuff.  
	All right.  So like I said I'm really excited to be here today for this launch we did beta test the site so some of you may have already heard it or seen it.  Probably not.  But maybe you have.  Anyone?  
	Thanks.  
	So before we jump in, just know this is a work in progress.  
	It's a great website.  
	Like it's done.  It looks good.  
	But there's so many aspects to it.  We know it's not complete.  So that's why it's a work in progress.  
	And you can write it down.  You can go to the website.  We'll show you how to do it.  And make comments.  
	If you find any basics like omissions or edits or misspellings, typos but also like what you want to see.  
	Because there's data that's not on here that will get added later.  Because this was huge just getting this data on here.  
	I am really impressed that they got 2021 on here we didn't think -- I didn't think that was going to happen.  Before the conference.  
	So thank you.  
	So why are we here today if the CDC already collects all this data?  
	You can't make informed decisions without being informed.  
	And we're doing this work -- I'm sorry.  
	This is not set up well.  
	So we're here to do the work for you.  Or do a lot of work for you.  
	Because we want to provide you with a tool.  
	So this is the tool to utilize CDC's data.  
	And then we provided some I think once again because I didn't do them so I can say it great graphs and visualizations of the data.  
	As you'll see later, once again, stay for the second half.  
	So when I know a bunch of you are EHDI Directors or related workers.  So when you used to want to do a presentation, you would go to the CDC website.  To data and statistics.  
	Maybe you did, maybe you didn't.  
	And you would have to track down what you wanted to look at.  
	And this is in hopes of just saving you time and energy.  Because I know you all work very hard.  You have limited budgets, limited time.  And hopefully this will save you some time and make your life easy, say to make a presentation to policymakers.  Or whatever you need the data for.  It's really -- we're not here to tell you what to do with the data.  We're not here to make judgments on the data.  We're just here, this is just a tool to present it for you.  
	And then you get to choose what you want to do with it.  
	So one of my colleagues, we did a dry run last week.  
	So she runs the Newborn Hearing Screening at the second largest hospital in Missouri.  And she's not a very animated person usually.  And she was like giddy excited.  By the end.  
	She was like, oh, my gosh.  Oh, my gosh.  Do you know how much time I spend -- like she named a presentation we did a couple of years ago.  
	She's like, my life is going to be so much easier now.  So once again, I didn't say it, so I can tell you that.  
	But I do have a lot of respect for her as an audiologist.  
	All right.  
	So I think we're going on.  
	Next slide.  
	So the EHDI process.  I know, this is review.  Just to make sure we're all on the same page.  
	There's screening, screening by 1 month, right?  Diagnosis.  Hopefully by 6 months.  Nope.  3 months.  Sorry.  Whew, don't kill me.  
	And then intervention by 3 months -- oh my -- screening by 1, diagnosis -- we thought -- you thought you were going to do bad -- diagnosis by 3, intervention by 6.  All right.  
	Now, let's get into some details.  
	So there are many versions of flowcharts.  So way back when, when I was actually still working with peds at CID, I made a flowchart for Missouri because I could not make sense of the numbers we had.  I wanted to understand it.  I wanted to cross-check them so I made a flowchart so I could do that.  
	It was very, very basic.  
	This group, we've gone through, this is what the original version looked like.  
	>> (Off microphone).
	>> That's what it is now.  
	There are different -- this is the best one we're going to use for this conference.  
	There's ones with less information.  There's ones with way more information.  
	So if you have a question, feel free to ask us.  
	Just know there's more lines that go around and do things.  But for this purpose, this is a good in-depth flowchart.  
	All right.  
	All right.  I think the next one, right?  
	All right.  
	So you know the babies are born, those are the gray square or whatever, rectangles at the top.  
	So this is the screening process.  
	So we talk about it like the screening process, like it should be so easy.  But as you all know, I don't need to tell you all the details of why it's not, it's not.  And it kind of goes in order.  Veronica worked really hard on making this actually make sense and I might forget some of it so feel free to jump in.  
	So you go to screens, go to (inaudible) -- then you go down to -- I'll let you talk about it.  You said it nice and succinctly.
	>> Okay.  How we try to visualize it starting with the screening.  Then when you enter the initial, there's summaries of pass or not pass the initial screening.  The very next row is going to be for some reason no initial but you did have an outpatient screening there are two metrics that represent that.  The row beneath will be if you had an initial screening and had an outpatient screening, these are the combinations of all of those results.  The next row is going to be you had some form of initial screening but then for some reason you had no outpatient screening.  Then the new row will be the summaries of passing and not passing outpatient screening then that all flows into these three other metrics of not pass last screening, overall no hearing and clear avid system which we'll chat about later but that's the flow of screening to try to provide some meaning for --
	>> So I forgot to say on the other one but we can do it on this one.  
	So the names are the names of the metrics.  And we go with being consistent with CDC.  So almost everything is CDC what they call it.  
	Underneath is the appreciation of the metric we use in our formulas by our, I'm going to say this one time so I don't have to say it all the time, our is them.  We is them.  I did not do any of this work.  I'm just here to be your guide.  
	So the ones that are in parentheses regular text type are CDC metrics.  
	And then is there any . . . yeah, like no --
	>> Sorry it's so hard to see up here the passed initial not passed initial passed outpatient not passed outpatient if you look at the four those are abbreviations we provide those are in italics we didn't want it to jump out we wanted the whole system to be the main takeaway so it's slightly italics we have something at the bottom of the flowchart so don't think you have to know all of this.
	>> All of this data on this website is from the CDC or derived from the CDC.  
	You might have to go look different places.  
	Some of it is very obvious on the website.  Some of it they really had to dig through.  Then the ones that they made the new metrics, they are all based on CDC information making into that calculation.  Does that make sense?  
	Okay we screen by 1 month.  
	So now this is diagnosis.  
	And you can see that.  
	Then we go to early intervention.  So we know the first two are Newborn Hearing Screening and the last part is intervention.  Or Part C, right?  
	Or maybe not.  
	All right.  
	So duhduh.  
	Do you want to sit here?  
	So the CDC does a great job at selecting this data.  You know from your work that every year you have to submit, fill out the survey of data.  It goes to the CDC.  And then they take it.  
	And make it publicly easily available.  
	So that's like the first page, right?  That's the first page of the CDC.  
	And then you can click on the data or go into the data statistics page.  Maybe many of you have already done this.  
	And they are doing a better job at making it more usable.  But we just thought it could go to another level.  
	I know.  All right.  
	Sorry; this is my first presentation in 11 years.  So I'm a little rusty.  
	All right.  So -- that's why.  That makes sense then.  
	So the CDC presents all the data.  They do a calculation, a rate calculation.  It's called hearing loss prevalence.  Which we all know hearing loss prevalence, we know it's something that the whole country talks about easily like 1 to 3 of every 1,000 births.  So they do a rate calculation to do so you can compare that.  
	So your state can have a hearing loss prevalence.  You kind of look at see how we're doing, how is it going with the national average.  
	So the rates introduce a means of comparing beyond hearing loss prevalence.  
	So that's important to consider.  
	So since we wanted to create normalized metrics, one state to another, we decided to be consistent with the CDC.  
	So hearing loss prevalence is hearing loss divided by total screened and then times 1,000.  
	As the denominator.  
	So also, always know your denominator.  
	We try and point it out in the talk.  
	But also know they will show you there's a place, everything is transparent.  You can actually go and see all the formulas to figure out if you're not sure what's going on with the data point.  
	All right.  
	So what are you doing?  
	All right.  I'm not liking the flow now that I'm looking at it.  What is my problem?  I'm sorry.  So can I just go into total counts?  
	Okay.  Show me what the examples are.  I don't remember.  
	Ah, all right.  Here we go.  
	So all right.  Here is the percentage.  So CDC has total counts.  
	So total count is just a total count like number of babies born.  Number of babies screened.  
	And then it has percentages.  
	So we all know what percentages are.  
	So for example, if I want to look at Missouri for the year 2021, the percent of babies enrolled in early intervention.  
	So the formula would be you would have to look up the number of babies enrolled in EI.  
	And the total with hearing loss.  
	And then do that calculation.  
	So the next page you would find that information on the CDC's website.  
	Somewhere in there.  
	Go on.  
	I'm not going to highlight where.  
	So in Missouri for that year, so we had 84 number of babies enrolled in EI.  
	And then total with hearing loss for that year in Missouri.  
	And then we times 100.  
	So 66.1%.  
	So that's the percentage.  
	So what can you do -- oops.  That's all right.  
	What can you do with that percentage?  
	You can use it to -- it's a good internal check for the state.  Like are we 20%?  80%?  
	You can also use it technically to look state to state.  
	But know that it is not a true comparison on any level.  
	I know I'm bouncing around.  
	All right.  
	So then you say, this is an example of a rate.  So this is the new stuff.  That's why I'm going over.  I don't know if it's easy or it's going to hurt your brain like everybody is different.  Like their stuff hurts my brain but I can follow along with this pretty good.  
	So Missouri, 2021, rate of infants enrolled in early intervention as a rate.  So now you have to go to two different pages and look for this information.  So you have 84 divided by 67,854 times 1,000.  You want 1.24 babies enrolled in EI per 1,000 screened.  
	So not per 1,000 born.  
	Per 1,000 screened.  But still the rate normalizes it.  And then you can compare.  
	I know no comparison ever, no dataset, is perfect.  I get that.  
	But we did -- they did their best of taking everything out of the equation and make it as comparable as possible.  
	Any questions so far?  
	>> The EI rates you're calculating here (off microphone).  
	>> Sorry; if you have a question, we need to talk into the microphone for the captioner.
	>> Okay.  So the EI rate that you are calculating here, just to confirm, it's based on your total infants screened, not those that were actually referred to EI for hearing loss; is that correct?  
	>> Uh-huh.  
	Just to add onto that, what was the rationale for basing it off of screening?  Does that come from CDC?  Or was that a decision you all made?  
	>> I think this is okay for now.  Yeah, so just to kind of jump off of what Marie shared earlier taking that hearing loss prevalence metric example we utilize that formula they are normalizing the population to the screening we're also thinking the screen is the start of the program for all the babies who are entering it and who you're capturing via the EHDI process so utilizing this and wanting to normalize to the population that's getting screened within each state to use that as the parameter to normalize and provide these rates.  So you can compare across.  And like we were mentioning earlier with the percentages the denominator varies from state to state of how many babies have hearing loss and are diagnosed and how many are able to meet the early intervention enrollment, et cetera.  Like the numbers fluctuate depending on the size of the state.  The babies moving through this process.  But if you want to be able to compare across the states let's compare across the baseline total screened for each state.  
	Okay.  Three.  Okay.  Hot topic.
	>> So follow-up question and thank you, that was helpful.  So with what you're saying is that with this tool, your denominator to keep everything consistent across the states is always to use the number of babies screened.
	>> Yeah when we supply the rates we use this as our denominator.
	>> So I understand what you're talking about here but I don't understand what you mean by we can't compare proportions across states.  If Missouri is able to enroll 66% of the children with hearing loss into its Early Intervention Program and Illinois is able to enroll 80% of its children who have hearing loss into their Early Intervention Program, why isn't -- why can't I say Illinois is doing a better job than Missouri.
	>> So you got a sneak preview at our presentation because the next couple of slides we will dive into that if you don't mind waiting a minute.
	>> Of course, happy to.
	>> Okay.  And then one more.  
	>> And I just want to know what -- and this may come later in your presentation.  
	But I think my concern with this using the babies screened is going to lead to a lot of misinterpretation about babies enrolled in EI.  
	Because it does not in any way concentrate on the babies with hearing loss.  This is all babies.  And if I go to my Part C program and say, give them this number, they are going to look at me like I've lost my mind.  Because the number of babies enrolled per thousand screened they are enrolled for all kinds of different reasons not just hearing loss.  
	So have you considered that and what the harm could be from looking at this statistic?  
	>> I'll chime in with a quick start and Marie for later parts we're not saying take this one metric to replace any other metric -- sorry; we're not taking any one metric and saying replace this rate over the percentages you're calculating we're still providing the percentages we're just providing you with some additional rates so you can consider I'm seeing shaking heads so I'm going to pass it over to Marie.  Sorry did you have a follow-up?  
	>> Yeah, like everything with data, we've got to critically think about what the data is saying to us.  What the calculation is saying to us and when to use it so we're not saying this should be used for this.  
	We just did this to provide normalization, some of it makes sense right now, some of it I think will -- this is my own personal opinion, not anybody else here.  
	I think it might shed light later on as we look at it this way.  And it may not.  And maybe we'll take it off the website.  
	Like I said, it's a work in progress.  
	But since CDC presented it this way, we stayed with it.  So like I said, feel free to give us all the feedback you want on all of this these are topics we talked long and hard about debated long and hard about so I'm very curious on all of your thoughts.
	>> I just wanted to add one point over that.  Maybe I'm varying from any one point so I really appreciate your concern.  Because denominators would be one because enrollment in every state is going to vary and this is a flow process of the whole ESDI system so in this program while you go through like we have understood that screening is the one where the ESDI program starts.  
	And when you go downstream, it goes through like hearing loss diagnosis and then intervention.  So it will vary across the state.  To normalize having a consistent parameter.  
	So to having the total population of babies would be nice and better.  To normalize as a denominator.  But being consistent with the CDC what it uses for like hearing loss prevalence, they use total screening.  And that total screen is the denominator to normalize so we didn't want to introduce at this point in time because this is work in progress.  
	So we didn't want to introduce something new normalization parameter so rather we went on to be consistent with the CDC's calculation in terms of normalization while we are calculating rate.  
	So therefore, you will see in the rates and it's coming on our slide, too, we have used the starting parameter that is total screen which is the ESDI process start.  To evaluate whole program.  Not a particular parameter.  
	>> So and the other thing maybe you'll say it later but in the percentages, they are very helpful but also they are not useful when you think of like comparing programs because Missouri is 66%.  And we had 84 babies diagnosed, right?  Or in CDI.  Micronesia has 100% rate because they have 1 child so that's where that falls down so every piece of data has a flaw and it has a place.  So once again we're just trying to provide as much data for you to do your jobs or not.  You don't have to look at the website.  
(CHUCKLES).  
	>> Too many microphones.  So I skipped to the next one as Marie was saying next this just kind of visualizes what she was saying.  All the things that we calculated we understand that you get it at one time point but we wanted you to see how they are different over the years.  This is another snapshot from the website as you mentioned earlier analyzing is important to look at a state to know how your program is doing.  Technically you can compare across everything Marie said she jumped ahead with all the good points so thank you.  No worries.  
	To get to the earlier question of like percentage versus rate, Missouri versus Illinois which is why I thought you had a nice little sneak preview we wanted to also compare just to show you the difference of what you can see in the trends between comparing early enrollment percentage which has the denominator of the hearing loss and the early intervention rate now which has the denominator of the total screened so normalized so you can see it tells you like two different stories.  
	And depending on the question you are asking how is my state doing at this point in time compared to another state in terms of the number of babies that are moving through the process in terms of how many are diagnosed with hearing loss you can see here in 2016 that's highlighted in the upper figure Missouri is 77% of the babies are getting that early intervention enrollment out of those who are diagnosed.  But then you actually see with the early intervention rate Illinois has a higher value because through their whole system 1.248 babies are moving from screening -- let me rephrase that 1.248 per 1,000 screened in Illinois is who is actually getting enrolled and has a higher one than Missouri.  So we're not saying anyone is better than the other but wanted to show you depending on what question you're looking at depending on what data you are looking at, you are able to see this information.  
	>> Let's back up on the charts.  The top is percentage, the top chart.  And orange -- orange, I'm trying -- where did the pointer go?  It was there.
	>> Orange is flagged in the bottom one, that's Illinois.  And then Missouri is flagged in the top in purple.
	>> So the top of the percentage and the bottom is rate, plots.  So you can see depending on what you're looking at, it changes.  And you can see how variable the data is over time.  There's a lot of variability in data.  I've learned that over the years.  There's missing data.  There's all kinds of stuff.  I'll stop getting ahead of myself.
	>> Hi.
	>> (Off microphone).  
	>> Are we good?  We can talk about it more.
	>> I have a question.  
	So the top visualization and comparison lines enrolled in EI by state, is that for those that were just the denominator just screened or those that were diagnosed because I know the bottom -- the bottom visualization the denominator is your total screened for your rate.
	>> So the top one is denominator is going to be --
	>> (Off microphone).
	>> Yeah, so that calculation that we did earlier then we go ahead here and visualize it so this is I believe a calculation that's also supplied by the CDC so this was us replicating it so we could visualize it and then with the rate it will be with the denominator of total screened of the babies with hearing loss what percentage are getting enrolled in early intervention as you would want to help move these babies along into the program.  Does that kind of help clarify that?  Then we do supply all the formulas we use on the website and some data files.  
	We're happy to dive into it more.  We have one more slide for this section.  
	So if we want to do that and if anybody has any other lingering questions it might be a good time and then potentially a break.  
	>> Thanks part of the job of Chapter Champion is to decrease Loss to Follow-up.  So Loss to Follow-up uses the denominator of numbers screened isn't going to do a lot for me thinking about kids with hearing loss.  You know, like the ones with hearing loss, the denominator has to be the ones with hearing loss.  And the ones that are in early intervention.  See, so like was said behind me was, yeah, kids are in enrolled in EI for all sorts of different things then another confounder is kids who have hearing loss as part of a syndrome where they have physical problems, they have occupational therapy needs, et cetera.  And they are going to need EI for that.  
	So it's EI for that.  And tagging along is speech therapy and an audiologist for their speech loss, you know, or speech deficit.  
	Yeah, I see where this is a problem.  I mean we've talked about in Illinois how the denominator is different.  It's different between the CDC and what Illinois uses for some of their numbers.  So I'm sure every state has that problem with the denominator.  
	I even typed in my thing, denominator, denominator, in my little notes here.  But yeah, it's important for Loss to Follow-up.  In that Loss to Follow-up thing that most of us have in our minds.  
	The denominator has to be the kids with hearing loss.  Otherwise the ones with hearing loss are not getting followed up.  
	Yeah; yeah.  
	It's just what you probably dropped down.  It's probably dropped down to this number versus that number.  Then certain numbers are going to be more valuable for certain people you know.
	>> And I would have to double-check what I'm about to say but maybe somebody in the room is more familiar.  But it was Loss to Follow-up to diagnosis I actually thought I saw the CDC's calculation was Loss to Follow-up to diagnosis divided by total passed with not screening not total with hearing loss so even there it's denominator, denominator so keeping that in mind, too, when you're double checking what metric you're looking at.  
	>> (Off microphone).  
	>> Oh, sorry.  
	>> So in general for me, so CDC is focused on 1, 3, 6, which they should.  That's what they are tasked with.  
	This kind of came out more of like ways to look at programs to make sure we're not losing babies.  
	Because there's a lot of talk within Missouri, are we, aren't we?  And we had many meetings about it.  And it ended up because based on the data that we had before this, this was private data that we did.  
	It looked like we were missing one to two kids a year.  Which with our numbers, one to two kids in other things doesn't seem like much.  But we don't want to lose any, right?  So we're just trying to make our program as good as we can.  
	But what we found out in all of those talks, it was a definition of how to categorize the children.  
	So this is just to give you food for thought.  Well, it's not just.  One aspect.  It's to give you food for thought.  To look at programs.  To try and understand the data from different angles.  And to have those talks.  You know.  To make sure we're all on the same page.  
	Because like every state, different population.  
	Every state, different protocols.  Every state different number of personnel.  
	There's a lot of variables.  And so we can't normalize that.  
	So we were just -- we want to present data that maybe can help you say this doesn't look right, was the data reported wrong to the CDC?  Do I need to put a correction in?  Is it right?  I don't like how it looks.  Why isn't it right?  
	You know, like just to question things and cross-check things.  Does that make sense?  And once again, it's not a judgment.  It's purely to help you all run the best program you can.  
	>> I think respectfully, your website is so exceptional, this is not even close to what your website is going to show us today.  So I think if you get right to the good stuff.
	>> Well, I'll be honest with you I think it's hard because we're hanging on some of these details like I'm not going to make use of the bottom one at all and yet what I would love to see is all of the phenomenal visualizations that you have in the website.  And kind of get passed this because it's not -- it's not going to be as useful for this particular population.  Because of what was just said around Loss to Follow-up.  And I do think, though, the whole purpose of this website is to be able to provide some state-to-state comparisons.  
	So I worry a little bit about the threat of constantly saying there are too many variables it's hard to compare no judgment but in the grand scheme of things we do want to compare to say we're not pulling our weight we're not doing as good as Montana what can we do that's as effective as what they are doing so I do think there's something to be said for qualifying everything.  
	But at the same time we are as people who are dedicated to finding as many babies as possible, we do want to find those who are doing it better than we are.
	>> In Missouri we have eight states that touch our borders so we started wanting to look at those states.  Like some states don't have many.  
	And just, like I said, everything is so different.  And you know just looking, is it just a matter of how we're reporting it or are there true differences?  Anyway I could go on and on but thank you, Heather, for what you said.  We have one more slide for my section.  And then I'm done and the fun starts.  
	And here it is.  
	So we/them -- I said I wasn't going to do that.  
	We collected 23 metrics from the CDC's EHDI website.  
	Then we calculated 28 more metrics.  
	So this is that example was just one from the reported data.  
	So we're presenting on this website 51 metrics for you to use in the interactive visualizations.  
	So over on the other side of the slide, on the left side of the slide, that's all -- I don't even know, it's not all, it's an example of how many clicks you would have to make different links to find all this data without this website.  
	>> And to clarify, this is just some of the links from 2007 to 2013 so like we showed before with the screenshot click to the right PDF, get the data you need from there and then rinse and repeat until all of this.  
	So I don't know if we have explicitly said it yet but we have data from 2007 to 2021 that we're going to show you in a little bit.  
	>> So thank you.
	>> Thank you.  
	>> We're going to take a break.  
	>> Yeah, did you guys want a break.
	>> So like a 10-minute break.  Oh, you don't want a break.  Okay.  Keep going.  All right.
	>> Okay.  Well this is the interactive portion.  If anybody hasn't heard yet we understand the WiFi connection issues so we do have some WiFi hotspots available for your laptops.  
	We do apologize, the phone interactivity might not be well suited for the nature of the website so we do recommend on a tablet or a laptop.  And we have one additional iPad if anybody needs a tool.  There we go.  
	And apologies I was made aware that the slides, you can't find them online.  Sorry about that.  
	Oh, did I see a hand for a second?  Okay.  
	I'm going to get my paper then.  
	Okay I'll just give it a minute.  If everybody would like to join me on the EHDI data website.  
	We put some placeholders in for folks just in case we ran into some technical issues but we should be able to dive right in.  
	Does anybody need a minute to get to the website?  
	Awesome.  Okay.  
	And then I'll be demoing through on this so you can look up with this very friendly photo to say welcome to our website.  Thank you for joining us here.  
	We on the first page show a little snapshot of what's happening in the U.S. using the latest 2021 data.  I don't know why I said 2021.  
	So we flag something from screening, diagnosis, early intervention, because these are the three schemes that we try to maintain as we move through the data.  We would like to flag here whenever we say U.S. we do refer to data from the States District of Columbia territories freely associated states it includes all of those when we talk about state summaries such as state average state median that's looking at state data and DC data.  It does not include the territories.  And on this website all the links are clickable and take you to our data hub page.  
	Which you're going to also navigate through the menu.  
	And the first thing we link here is the data description.  Because as you all just heard there's a lot that went into this, a lot of little nuances so we tried to provide as much as we could there.  If there's anything you think is missing from that page, we would love to know that, anything to clarify, any feedback like this now or later because we have an evaluation, we have a contact us page on the website at any point in time if you have thoughts you want to share or questions or concerns please reach out to us we would love to hear from you and make this the best tool for you.  
	And diving into the content of the website, we essentially have two sections.  One explore a metric.  So taking the example of the percentage of early intervention that we talked about before.  If you want to dive deeper into this and provide this metric in several different ways so you can use the explore a metric section to do that.  51 metrics is a lot so we try to kind of group them into different pages to make it easier to digest.  
	So we picked eight key things.  
	If you only have a minute to go look at things we put eight of these on one page so you can scroll through those we have 12 additional metrics of percentages and rates that we supply on the 12 additional metrics page.  Then all the total counts that we talked about so everything that you saw on the flowchart included stuff that's directly reported to the CDC or in the case of Casa patient screening for example that's a summary metric so all of those total counts in no form divisions or calculations all of those if you want to see it are provided on this 31 total counts page and then if you just want to go explore all 51 we provide it in this all 51 metrics page.  Each of these links is clickable and takes you to that specific page.  
	Then in the second section that we'll explore in the later half of our presentation today we take a look more closely at state explorations.  
	So we compiled the data from the 8 key metrics and put it into a sheet so you can see a specific state's trend over time over the 8 key metrics we take state versus U.S. and state versus state I'll stop there because Keisha is going to drive that section later then the other thing we wanted to share with you is the data because we're visualizing it here but we also want you to have it and be able to do what you want to do with it so all the stuff that we have worked on the 23 metrics that we took from the CDC's website the 28 that we have calculated we put all of that into a dataset and you can download it here.  
	If you want to know exactly what all of these things are we provided a codebook to go along with it and again we link over to the description of the data page.  
	So for the next section, we'll dive into the 8 key metrics page.  
	So at the top of all of the pages that we present visualizations on it we've put this banner at the top so it describes the general interactivity of each figure that you're going to see because that's another thing we're really excited about these are all interactive figures.  
	It does take a second to load so if it's blank just give it a minute just for the content that's there, it takes a minute.  
	>> (Speaker off microphone).
	>> And this might be by nature of the display tool that we had to use for the website it's a server.  As you know servers sometimes overload so hopefully we can all get in.
	>> Also we have WiFi that is ours, maybe the speed.  So consider that.
	>> No it's not loading for me.  Sorry, folks, we tested it earlier and it was working.  
	>> (Off microphone).
	>> Oh boy.
	>> I'm using the hotel WiFi.
	>> I think this is WiFi.
	>> Oh, man.  
	>> So how many people are on.
	>> Okay.  
	>> We really tried to be ready for all problems.  
	Do you want them to go off so you can at least pull it up to show them.
	>> I feel bad because the next part is interactive.  
	Let me see if I can reload my page.  Just give it one more minute and I'll see what I can do.  
	>> I feel this is just WiFi.  So just give it a couple seconds so we can ask somebody to help us.  
	>> You can go to a different one -- not the 8 key metrics we're trying to get on.  (Off microphone).  
	>> That's a good point.  I will go -- sorry; that's a good point.  
	I'll try all metrics just because the first half that we're about to dive into now goes into key metrics, additional metrics, all metrics, total counts.  So those are the same sets of visualizations then the later half of the presentation will be the state-to-state comparison.  
	Yeah, I wanted to show the downloading.  This is more helpful.  I appreciate that tip.  Thank you.  Ah, success, thank you.  We'll see how long it keeps me in if I go too fast let me know but I'm excited to show you this so here in hearing loss prevalence the one you're most accustomed to we thought we could walk through this one to show you the different visualizations we have on the website like I mentioned earlier at the top of the page it's going to be interactivity you'll be able to select the metric you want to explore using the drop-down.  You can hover over any plot to get more information.  
	You can click and drag over most plots to zoom into in area.  There's tools for each of the interactive plots you can utilize that shows up in the upper right corner when you are hovering over.  
	And then you can download any of these figures as a PNG so you can use it whenever and wherever you would like.  So once you select a metric in this case HLP you get a brief description at the top.  
	It relates back to the data codebook we mentioned earlier so try to keep the information across the website for you the first we show here states below and above 2021 median by year we used the latest data so 2021.  
	So in 2021 you can see there's 25 states that are higher than the state median.  And 24 states that are lower.  And what this gives you is a quick snapshot of how has hearing loss prevalence or a metric kind of changed over time so you can see towards the start of the years there was more that was occurring beneath this median value of 1.891.  
	And then it kind of increases over time.  So just a quick snapshot of how has this metric changed over time across the states and DC and then to show you some interactivity I hover over it, it tells me what year, the number of states, the percentage.  If I want to just look at 2014 to 2016 I'll go ahead and highlight it zooms in and then it should be whatever portion that you're highlighting, you should be able to click that camera button in the upper right corner.  
	And download it as a PNG.  
	Which is kind of hidden from me.  But here we go.  
	Ta-dah.  So I wanted to focus on this, download it, utilize it as you would like.  
	Oh, click and drag.  
	So I use this cross button then I drag it down like this.  And I believe if I reset my access I can also use the zoom in button in the upper right tools and zoom out.  
	And then even if you zoom in, I think you should be able to pan across.  
	By clicking and dragging.  
	And we do provide some tools that are specific to the interactivity of the figure that you're looking at.  In the bullet points right above each figure.  Then I just hit the home button to reset.  
	Okay.  
	Marie just brought up a good point I'll reiterate it so you can save these and utilize it as you want.  
	Yeah.  So you can put it in a presentation.  Send it in an email.  Just whatever you want to take a look at.  
	>> This is all public.  It's all free.  
	>> Okay.  So the next section are the smooth line plots.  In this we wanted to show you again the variability across the states and variability within the states as you just look at this image.  It's going to be ordered from the highest to lowest latest HLP value for a state so for 2021 Hawaii had the latest highest value as you move down you'll see about New Jersey I think or DC, the value decreases and if there's missing data for 2021 like in the case of Alabama, those will be at the end.  We provide the U.S. also for reference as that very last figure.  
	On each plot you'll see the actual data that's been reported by the state for hearing loss prevalence in that red line.  
	You can see here there's a lot of variation that's happened over the years.  
	But then in the case of Tennessee, there is a trend that you can observe but not as much variability across the years.  Then that blue line just very quickly gives you a snapshot of that trend over the years with the available data.  
	And if you want to just see what that actual data looks like if you click on the legend at the top I'm clicking on smooth right now it takes it away.  Well I double clicked.  So it took it away so now you just see the actual data that's been reported across the states and you can see that snapshot in the variability if you just want to see the smooth line trends over the data reported you can go ahead and take a look tattoo.  
	And -- look at that  too.  And for reference, that black dot will be the state median in 2021 so that value we used up here we're showing you again down here so you can compare it to that state value for each figure.  
	Then the next set of figures, the comparison line plots.  
	Another way for you to kind of take a look at that change in data over the years and across the states.  But this gives you a snapshot in this picture is where do hearing loss prevalence values tend to maybe cluster over the years so you can see by the thickness of the lines in the middle most states follow this trend in values.  
	You can easily see the states that are at the top.  Hover over and you get a picture and you see okay Hawaii has been this line that shows up at the top the state that has lowest HLP  values you hover and see West Virginia you can use the interactivity again to zoom in, zoom out, take a snapshot.  One thing we wanted to provide you here maybe you don't want to see all the states at once we want you to be able to compare anything across that you would want so let's say Alaska wanted to compare themselves to the state average and to the U.S.  
	And then you get this figure showing you the three lines you're most interested in.  
	You can -- I'm sorry?  
	Oh, thank you.  
	Okay.  So the legend is going to show up on the right Alaska will show up in the blue the state average is in the lighter pink shade and the U.S. is in the darker pink red shade when you hover over you get more details on the exact value for that year and we also provide a reference to within 2021 what quartile was this value in and Alaska in 2021 was in a moderate-low quartile of hearing loss prevalence across the states.  
	And one more example of the PNG.  
	And here.  You get this snapshot.  Oh I'm sorry; a question?  
	>> Can you just remind me the difference between state average and U.S. again.
	>> Yeah.  So we noticed some variability in the reporting on the CDC stuff so we decided how they utilize U.S. is that total is across all the states and territories and freely associated states so if that data is available it's applied in the U.S. value then we also wanted to take a look at the state summary so states plus DC.  
	Then we want to keep track of it ourselves too so we put a note to ourselves so we can remember hopefully we put enough information on each plot.  Let us know if we forgot something because we would appreciate that.  
	And then we just want to show you in as many ways as we can how data has changed over the years and across the states so this one will be the heat map of hearing loss prevalence by state.  The colors are going to be related to those quartiles I mentioned before and everything is relative to the latest data we can get so for the 2021 quartiles so essentially if you were grouping the 2021 hearing loss prevalence values into four equal groups, dark blue is going to be that highest or top portion of hearing loss prevalence values.  The lightest blue will be the lowest grouping.  And you're going to be able to see the change in hearing loss prevalence over time.  So you can see again where we got a snapshot of in the first figure hearing loss prevalence values used to be much lower as reported in the data in 2007 and over time it's kind of gotten higher by 2021.  
	It's a lot of different -- maybe Missouri wants to take a look at how they have done over time so if you click on that state it will highlight that state throughout each of the years and then just to look at our Missouri-Illinois reference.  Missouri.  Sorry.  
	Oh, Illinois was not in that year.  
	So let me pick an example.  
	This also gives you a snapshot of if a state has had data available in that year for reporting or for calculations.  And you can see that trend over time.  See how they have done relative to one another or if you want to articulate yourself you can deselect by clicking on a state then you're supposed to be able to double click in any white portion to reset.  I think my system is a little bit slow.  So I apologize I cannot demo.  There we go.  And U.S. is provided for reference in that yellow color.  
	The next metric is annual map comparison because we love seeing the numbers, seeing the states that go with it but what if you just want to track yourself over time visually.  And in Missouri as mentioned earlier we have 8 states -- no.  
	It timed out.  
	Oh, no.  
	Okay.  This was the part we were really excited about, that little play button, I don't know if anybody has it.  But it visually plays out that story over time because each state is going to be colored based on their hearing loss prevalence value.  And it's ranged from the lowest the highest hearing loss prevalence value across 2007 to 2021 so you can see how your state has changed over time.  See how it's changed over all of the states over time.  
	I am going -- I wasn't sure if it would work.  
	Let me try again.  
	And this is why we came prepared with some screenshots.  So I can't show you the pretty interactivity maybe if it comes back.  
	Still going.  Okay.  So the next figure that's underneath the map will be these box plots because again do you want to see the trend over time for hearing loss prevalence.  We want to show you in a different way.  
	The metrics are going to be displayed so within year what's that variability in the data.  If we were looking at -- and I am not able to download with this presentation view.  
	But if you can kind of see this line in the middle is going to be the median value for hearing loss prevalence within a year.  
	The box that kind of goes around it and then the extenders.  If this were a normal distribution of hearing loss prevalence within the year this is what it would look like so the black dots are going to tell you are there outliers to information within hearing loss prevalence within a year.  
	The dots above are going to give you an idea of like how many states are going to have higher values than this and how high do they go compared to the rest of the distribution of the states.  
	It gives you a picture in 2007 there were a lot of states with higher hearing loss prevalence values relative to the trend within all the other states.  And then if you see in the year 2017 so this blue line down here there were even some states that had really low hearing loss prevalence values that were reported.  And we're back.  
	Okay.  So I'm going to do the play button before I lose this again.  Although I accidentally zoomed out.  
	There we go.  Okay.  And play.  Ta-dah.  Thank you for your patience.  
	Oh, yes, thank you.  
	White if you're wondering means no data was available for that year we're trying to make that distinction between the light gray and the white.  Then these box plots I forgot to say hover over you get more information about the state and the hearing loss prevalence value for it.  Click to the different year that you want.  And then you can see, Texas for 2013 all of these interactivities you can download for the specific year you're interested in using the download PNG button then box plots similar information hover over to get what I was talking about before 2020 that's highlighted there were no outliers of hearing loss prevalence in 2020 but in this year you can see the max HLP reported is an outlier to the distribution as high as 3.86 compared to the median which is 1.75 across the states.  
		Then we want to give you all the data here on the website.  
	Let's say you wanted to see Alabama-California you can click to keep track of the state over time it stays highlighted until you deselect it then you want to say okay what is the highest for 2021.  
	If it let's me click it.  
	We promised this worked in all our practices.  It's supposed to let you sort it so you can go high and low and click it over the years and it maintains that selection of the state so it's nice because you can track your state over time.  
	And all this data is provided.  
	Oh it's loading.  That's why.  
	Okay.  And then this is basically the walk-through of this metric exploration page.  As you can see we tried to show you each metric in as many ways as we can.  Let you see which story is best to your purpose at that point in time maybe the heat map is what you want at that point maybe you want to focus on a couple different states compare it to the state average compare it to the U.S. maybe you want to show the distribution of hearing loss prevalence via this way over time.  Again just because we're really excited you -- you can download any of these and use them as you would like to and we hope they are helpful to your work.  
	I will pause there for any questions.  
	>> First of all, thank you for showing the visualizations and all the different possible ways we might want to use them.  I think that's really helpful and valuable.  
	I'm just curious why this project.  Why was this even put into place?  What was the impetus for the whole project?  
	>> Do you want to speak, do you want me to speak.
	>> No 30-second elevator pitch I'm just curious because this is something that's coming from a university this is coming from individuals who aren't in hearing loss and it's something that I think is going to be extremely useful for a lot of people in this audience.
	>> So I think the story goes it started with Missouri EHDI and questions.  And there was a stakeholder that used to attend the meetings.  And he was heavy into data.  He was a physicist.  
	He was a CODA so he was very attached to the topic.  
	So we -- within Missouri we did a bunch of stuff like I said flowcharts and data.  
	And then the conversations kept happening.  The stakeholder then talked to CDC directly.  Went to Atlanta.  He talked to HRSA.  It's been like years of -- and I'm not totally sure.  Somehow the project got started at WashU.  I got brought in, like I said, as the EHDI guide.  
	But I think that was out of a suggestion from some docs at WashU saying the Brown School is really good at, you know, taking public data and doing these things.  
	So that's how it started I think.  
	Do you guys have comments?  
	>> I guess from the Brown School side at least at the Center like I mentioned in the intro, we work on several projects, depending on the need for the project, maybe it's an evaluation of a state policy program.  
	Just kind of depending on how that aligns so they reached out with the project and this was something that we were interested in.  And yes, this is data that is in PDFs in many different forms.  And to give credit to the CDC, they did give us I think it was like 2016 to 2020 data.  And some Excel and access files but the rest we did have to kind of pull from the PDFs that were available on their website.  And at the Center we do love finding different ways to help translate the data.  
	So even if we are a research university if we write a paper we still try to synthesize that in a shareable format that's easily digestible we want to make sure the data is here the data is accessible.  The data is in a form that people want.  
	And is helpful to them.  That's why we keep saying give us the feedback that you have because we want to make sure this is something that's useful to you.  It's a tool that helps you and not just you know to the earlier conversation we're giving you a rate that you don't really want that's okay.  We are giving you all the other metrics and we want to give you the metrics that work for you.  So . . . 
	Oh, yes, thank you.  
	So this is an ongoing project it should be happening for at least a couple more years so we are taking any suggestions that are given to us for revising the content on the website.  The next time the data becomes available we want to add in the next year's data, too.  
	So we hope to continue helping for a while.  
	Two questions.  Sorry; I think you get the mic over there first and then we'll come back.
	>> The cleared out of system rate.
	>> Oh you got the one.  Okay.  
	>> What the heck?  
	>> So it's a work in progress.  Because we made some discoveries right before the presentation that we want to dive into a little more but I can give you the intent of this metric.  
	Essentially babies who do not pass their last screening you would want them to move on to their diagnostic evaluation.  
	-- this doesn't always happen but they are captured in Loss to Follow-up to diagnosis for maybe parents cannot be contacted or in some cases the CDC even has it documented as no reason documented so there were three different variations they have.  And they summarized.  So these numbers are there.  
	So let's say maybe the numbers don't add up, that's the babies we're hoping to capture with this cleared out of system.  Like is there a way that babies are lost to the flow.  And we have since realized there are some babies that are captured in the CDC's data that are not screened and go on to the diagnosis.  
	So we have based the premise as you've heard like screening is the start of this program.  Screening is how you move through the flow but now we're learning there are some babies who are not screened and go into diagnosis the reason we need more time to dive into it when the CDC does provide it on the website their denominator is of babies screened.  
	Like you are not screened, go straight to diagnosis divided by babies screened.  So we just needed to wrap our minds around that a little bit more to try to figure out the proper flow for seeing this is how the babies flow through the system why is the CDC using this other denominator because like we talked about denominators are important and want to get a better understanding of that clear is something we have calculated based on the formula we have provided on the website in the dataset but it's a work in progress once we can screen out -- look at not screened babies we would like to incorporate them.
	>> Also adding to that -- oh.  
	>> Are you still confused by what it is?  
	Okay.  Basically diagnostic evaluation plus Loss to Follow-up should equal total not passed last screening.  But it might not necessarily actually add up.  So we calculated this metric to be able to figure out what is that number if it doesn't match up and in some cases if you'll see -- you're probably looking at the images there's negative numbers and then really large numbers or something like that and we have discovered it's a discrepancy or negative numbers are coming from this understanding that there are numbers in the system related to these not screened babies because we based everything in this flow and all the counts and calculations off of the screened babies.  
	So . . . 
	Not passed last screening.
	>> Okay.  So you're talking about like babies that could be in this category could be infants that have died?  Infants where parents declined.  That's what's in that group?  
	>> I just want to say because this helped me a little to understand things.  So Loss to Follow-up, the number that they are using the denominator for Loss to Follow-up is Loss to Follow-up of diagnosis.  
	So I know I'm getting off your point a little.  I know there's Loss to Follow-up before the screening even happens.  That's not in it at all.  But yeah, all those things.  The baby died.  They moved.  On and on.  And I'll give it back to her.  I didn't answer.  I was just trying to provide more information.  
	>> I wanted to add that while we were making flowcharts, so we were retrospectively going from the data to making sense of like how the babies are flowing through the system so sometimes we find that, okay, the number at early intervention do match with the previous system.  And then we add those at each step and goes to the total screening.  
	But for some states we find that after adding each bits and pieces still there is some figure that's messing up.  
	So we are like okay there is something that is not matching or not reported.  
	So still the work is in progress.  We are figuring out what those variables are.  How those babies maybe the syndromic patients are coming to the ESDI system that we are still not aware of or we are still working on.  
	So I think the clear is that parameter where the missing numbers after even going through all the variables still we find that some of the parameters are missing.  
	So clear is that system I think.  And we are still working on that.  Recently we found that one variable can add.  Though still we are not clear about that clear.  
(CHUCKLES).  
	>> Hi, I just wanted to let you know this has probably saved me thousands of hours of work that I've been doing over the past five years so I know sometimes maybe working on a big project like that you may not understand the impact it's going to have eventually.  One of the things I'm really passionate about in Georgia we created a language in literacy dashboard for Deaf and hard of hearing children we have about 18% of our children who are Deaf in special ed reading on grade level by 3rd Grade I think this is so important no matter what numerator or denominator you're looking at I've had to slice that and dice it a million different ways to maybe select five data points to include in this dashboard and report and now I can just sort of do this a million different ways, see what it's telling me.  In Georgia we have hospital reporting issues.  
	So that's affecting the data that even goes into a system like this.  
	So we're really starting --
	>> (Off microphone).
	>> Exactly; exactly.  So this is very exciting to me because I think there's the potential to create change, behavioral change, systems change, from this data and measure it so we get better outcomes not just looking at data and numbers and reporting but actually improving the outcomes so I really applaud you guys.  
	How long did it take for you to build this?  
	>> Let's just say we had a timeline and the data synthesization steps like where you're talking about getting the different data slice the different data maybe four times the time we estimated we would need at least.  
	So one thing we discovered I don't think we spoke to it yet but we wanted to provide a dataset you can trace back to the source on the CDC and I give credit to Keisha for this.  Keisha combed through -- sorry I'm speaking on your behalf I'm just really impressed by the work that went into it combed through all the PDFs and datasets we were given by the CDC other stuff that was on the website we wanted to say consistently you can get this hearing loss variable across each year you can consistently get this total screened variable across each year and I'm using this because it's the examples from before.  
	But if in one year like here in hearing loss prevalence was not calculated for example we calculated it for you because we wanted you to have A + B = C we're giving you A and B if you can get C every year yourself essentially.  
	So a long time is my simplified answer.  
	>> (Off microphone).  
	>> Yeah.  
	>> (Speaker off microphone).
	>> The captioner is not picking up people without the microphone.
	>> Oh, the microphone.
	>> Sorry; sorry.  
	>> So the stakeholder Jim Cesar, the idea he pushed he met with the CDC and HRSA and it was all good.  He just had a lot of discussions.  And he just wanted it to be more visual.  And like I said I don't know the details of how it like you know got to you guys.  But it got suggested and then they just ran with it.  Like he did basic stuff when we were working with Missouri.  Or when he would come to the meetings.  He has thrown all of that stuff out.  Like this is way beyond anything he's done.  
	And he doesn't want any credit.  
	So that's why I was trying not to say his name.  
(CHUCKLES).  
	>> Sorry; I forgot one more page here I wanted to flag during my portion of it.  We will keep you updated whenever we update the dataset, make any data additions.  Or if we perform any new calculations, we'll put that date on here.  We give credit to the CDC for the data they shared but we also link to you if we put anything from the PDF we put that in here for you so you could track that down.  
	And then we do provide a copy of the data which was what I was going to show because slicing and dicing we understand that about the data we wanted you to be able to have that so thought we would just give you a show.  So you have this dataset.  
	Oh okay yes that's a good point one more thing then we'll take a question something you said flagged it for both of us here in addition to the outcomes we also thought it could be interesting to the data that is being reported on the website.  There's like more details on the description page.  But in essence if a 0 was reported for a metric we utilize that 0 as reported.  If data was not reported that stays missing on the website so if you see a 0 in there and you think it looks weird maybe it is weird but that's what's on the CDC's website as reported to them by a state in a year we wanted this in addition to being a tool for presentations you have also for a state to be able to go look and say why is there this variability here I think that number is actually wrong it should be a different number it helps you flag and keep track of your data being reported by your state over time help you flag that, identify that and see the changes over time.  
	Is that good, Marie?  
	Yeah.  
	>> Thanks I'm Linda Hazard from Vermont my question I'm going to go directly to my state because I think there's a lot of really great work in here and it's really interesting and things that I know moving forward I can look at.  
	But Vermont is -- we're the smallest birth state in the U.S.  We have a small diagnosis rate every year.  But if you look at the data, it's consistent.  
	So we always run about 6 to 9 babies identified.  
	And then every once in a while we have a blip where we see 14.  And that's typical of this year.  
	But because there's the low, the moderate and then the high diagnosis rates, how does that not get perceived as our state missing babies?  Because our state isn't missing -- I mean I joke we have a fully integrated database system at the Health Department so we have electronic birth and death records, immunizations, lead, developmental screening, newborn screening and Newborn Hearing Screening so escaping us is really challenging and we also follow our babies out.  I also oversee Part C so I see those that come in with a quiet and progressive losses.  So it's perception that I worry about a little bit.
	>> On this website, you can look through and this is what you can do.  You can pick any metric and follow it through and look.  
	And like the overall, Vermont comes out well in all of this.  You can do -- I know I probably shouldn't be saying this but I've looked at the data so much pull up New Hampshire and pull up Vermont and you'll see how different things look in how it plays out.  Does that help you?  I don't know how to --
	>> I understand what you're saying with like also from the groupings flagged in the presentation because I flagged you can see where states are low or you can see where states are high we kind of intend this with the distribution of values across the states.  
	So we don't mean it to mean low is bad or high is bad we just want to present from lowest value to highest value here is how it's spread out across the states if we wanted to group it here is how the grouping would look based on the values not based on any meaning of low being good or low being bad or anything like that we just waned to take a step back -- wanted to take a step back and give the numbers and the trends -- and allow you to take a look and give additional meaning so maybe in addition to I forget the metric you just said but maybe in addition to looking at that you look at the figure of the total counts that are there and then you get the reference you see it's because the base number that's leading to this calculation is already so low so that's how it ends up there.  
	Hopefully many different ways to approach it.  Look at it.  
	But we don't mean any meaning for like this is the lowest value this is too low or anything like this.  This is what's reported.  And this is how it's showing up across the whole spread of data.  I think there was one more there.
	>> I'm Tracy Mastery in regarding in evaluating data, I was going to ask I know each state has their own legislator and some states don't have mandatory like -- like they don't have the states supporting certain screenings.  So my question is, is that being taken in consideration in doing the data?  
	So would that be something that you guys might want to examine?  
	>> I think if -- sorry.  Yeah, we would love that feedback.  
	With some light context and some knowledge of where to get some of that information we could try to incorporate it with this first go of this working in progress website we just wanted to say like these are the data that the states report to the CDC survey.  We get this as reported on the CDC's website.  And then we just try to present the visualizations as they are available on the CDC website so we don't take any additional context or meaning.  It's like if this number was submitted this number is visualized we don't have additional contacts like the Legislature.  I'm struggling.  Legislature.
	>> So currently that's not on the website but you can you could take like -- you could highlight all the state funded sites or state parameters versus the non-funded states if you're wanting to present to somebody or a policymaker saying look they are so much better than us or we're lagging behind or whatever the issue is, and you can make visualizations of that and use that in that if that helps.  
	>> Just going back to my original question this is Linda Hazard again, I think it would be good if you can put a statement up there that describes that low versus you know medium versus high is not -- you know is not making a statement about a state doing well or poorly.  I think that's important.  
	And then, also, just making sure that you reach out to you know the various -- while you're doing this work, you may want to reach out to the various EHDI Coordinators just to get more insight from us, as well.  
	Because I think that can help.  
	And then the last thing I'll say is Tracy, right, you mentioned states that don't have legislation.  Vermont doesn't have legislation.  But we do have administrative rules, which guide.  And so other states may have other things in place if they don't have legislation.  
	>> I'm going to swing the mic over here.  
	>> Okay.
	>> Thank you.
	>> (Off microphone).  
	(Unable to hear speaker).
	>> Just for the words up there into the microphone so we technically have about 25 more minutes for the session that's scheduled but we believe we have the room until maybe 3 so we're not saying we have to cut it at 2:30 I'm getting a nod of approval so just wanted to let you guys know that.  
	>> Okay.  I'm Kristin from Virginia.  And I oversee all of our different Newborn Hearing Screening programs and I unfortunately have to spend a lot of my time putting out fires with media, with legislators, from misinterpreting data and manipulating data to suit their agenda.  Especially across multiple states.  We don't have eight states bordering us but we do have a few.  
	So something I would suggest including, if you haven't already, is contact information for the EHDI Coordinators for the states as well as some kind of statement for encouragement to contact the state as far as with assistance to interpret the data.
	>> Can I ask a follow-up question?  So for tobacco control there's a tobacco control network that they update like who is the person to be contacted for the state.  Is there a similar page that synthesizes this for this work that we could link to or direct people to go take a look at if it changes over time?  
	>> So I'm -- so I know that the -- and like the Newborn Hearing Screening space the HPL new steps does have all of the Newborn Hearing Screening contacts including EHDI so there might be additional resources.  But that's one that I use for all my programs.
	>> Okay.  I appreciate that.  Thank you.  
	One more.  And then I'm going to -- I stole both from Keisha so I'll pass one mic back for his portion of the presentation.
	>> So the National Center for Hearing Assessment and Management does have a listing of state EHDI Coordinators that does not give you the state EHDI Coordinator's direct email address.  
	So APHL newborn screening has some of them, not all of them but I think we could certainly work with you to get a way to get in contact directly with the EHDI Coordinators.  Karl says yes.  
	>> And I just said to Linda on the way up so maybe I missed it, I'm sorry, but we do want to like involve all the EHDI -- like we're open to making anything better, working with EHDI Coordinators.  But it was such an overwhelming task just to get here to launch this that we couldn't bring in a single other person's input.  
	So just so that you know.  
	I don't know what you guys were talking about when I was talking.
	>> No it's right here so infanthearing.org.  It's NCHAM's website.  
	>> It has the link to them.
	>> It has the link to them but it has all the phone numbers.
	>> I sent an email through NCHAM and there's no direct email (inaudible) at this time.  
	>> Hold on.  
	>> Yeah.  
	>> So this is Karl White and I'm with the National Center for Hearing Assessment and Management.  We probably won't put a list of email addresses on the website because we want to protect peoples' privacy and don't want them to get a whole bunch of spam but if you were to contact us directly and say can you give us a list of direct emails for all the EHDI Coordinators for this purpose, we would be happy to do that.  
	>> Yeah before I pass it off I just really wanted to say thank you for all of the really helpful interesting feedback and questions we're taking notes.  We appreciate it.  Again if anything comes up please reach out to us we want to make sure we incorporate any ideas and any other ideas that come up after this session so again a thank you for me.
	>> Thank you, everyone.  It was insightful up until now.  So I think everyone is very excited about the state profile, which I am going to cover with you.  
	So we have these prompts similar to the -- just wait a second.  
	Similar to the previous ones.  
	So I'm not going to repeat that.  
	And another thing, moving on to -- moving next, in the left corner you can see that you can select a state.  
	I'm selecting Ohio for example.  
	And here you'll get the comprehensive summary of state data along with the state map in a state of red.  
	Then moving on to the next if you see the first one I will highlight here is the year as mentioned you may be curious about what is this 2021 this shows that we have the latest data available and the summary is generated out of the latest year.  
	And if that generated one is missing, you can find there will be like 2020 or 2019 or whatever the best possible data available to us.  It's reported on the CDC website.  
	So we have total births of that particular year, 2021.  And then screened percentage, 97.2%.  
		And another thing I want to flag that you had questions before that this rank does not mean that it's your ranking of your state.  We know -- by no means we wanted to rank any state with the other or this data does not provide you or judge you any state because everyone is doing a great job at their own pace.  
	So this is basically we are saying the distribution of your state in particular data we considered.  
	So this highlights you the distribution of your state data out of those 49.  
	And we have rates.  Like diagnosed hearing loss out of 1,000 screened.  And similarly we have enrolled in EI.  
	Moving on to the next hearing loss prevalence.  That is our interactive plot we have a couple of metrics here.  
	So that it gives a comprehensive summary of a state profile.  
	So hearing loss prevalence before I dive into, I'll say how we calculated this.  This is calculated as number of permanent hearing loss per 1,000 total screened.  This was our formula to calculate this.  
	And you can see that hearing loss prevalence trend of Ohio is close to the U.S. trend.  As some years are above, some are below.  And you can find out there are remarkable years.  You can see that the trend is crossing U.S. trend.  
	So you can find that you can interpret it in different ways.  
	So I'm keeping this ball on your court.  
	And one thing I want to mention that U.S. trend here is like it is from the data of state territories and freely associated states.  
	So this is the U.S. trend.  
	Moving on to the next this is the total screen percent, total screen percent is calculated as number of screened infants divided by number of births times 100.  
	And you can see the plot -- the plot in this parallel to U.S.  In recent years you can see in the past year you can see there was variability.  And the sets of gray is like the other states.  
	And you can see those variables, too, from -- the variabilities, too, from your state.  
	And I wanted to highlight one of our tools that is Lasso.  
	So it gives you a point -- sorry.  
	I just . . . 
	I'm not well versed with this mouse plot.  
	Okay.  
	So this is -- okay.  I'm resetting the access.  It's taking time.  
	So Lasso gives you access to you can select a particular area of your plot.  
	And it will highlight the points in that particular area.  
	That determines if you have an objective like okay I want to see 2015 to 2020.  So you select that particular area.  All the points of that particular area will be highlighted.  And the rest will be like dimmed out.  
	So that gives you an access to being specific to your project.  
	And so with that, moving on to the next is the not passed rate.  Calculated as counts of babies who did not pass their final -- okay.  
	>> I have a quick question.
	>> Yeah.
	>> Is there a way if we wanted to remove those gray areas to do that?  Or is that something that maybe you could put on your notes to make an option?  
	>> Yeah at this point in time we are going to note that down to remove those gray areas.  Thank you for highlighting that.  
	And okay.  Continuing on to the not passed rate.  This is calculated as counts of babies who did not pass their final or most recent screening.  Maybe inpatient or outpatient per 1,000 screened.  
	And you can see the rate of not passing follows the U.S. trend until recent year.  When it has started increasing.  The year 2021 can be seen as breakpoint of the trend.  And you can like get more interpretation based on particular things you are looking at.  So it's on you.  
	And we have this pan tab.  I just want to highlight that you can just slide it down or particularly if you have zoomed in, so you want to select particular year, so this gives you access to you scroll it to your particular highlighted state.  
	Okay.  Always don't forget to double click so that it will reset your access.  
	Or the home button.  It gives you the same access.  
	Next is the --
	>> Can I make one comment.
	>> Yeah, sure.
	>> Okay.  So for this just to give you like how would you use this or how would you look at it, there's a bunch of ways.  But this is saying Ohio  state brings a lot of babies into their -- has this screen moved.
	>> This is diagnosed.
	>> Just making sure.  They bring a lot of babies to see their audiologist for diagnostics in the state.  And most -- and they are saying the average in the United States is less.  It's not -- you know, it doesn't make a comment on the effectiveness of their program.  But then you know if they don't have the staff, overwhelmed, like let me look at some of the states that don't bring as many babies in but still find all the babies with hearing loss.  
	Does that make sense?  So that's just one way you could use that.  
	>> Yeah, go ahead.
	>> I have a question.
	>> Yes.
	>> Just I've been -- I'm a nerd here so I've been playing a lot on this.  And if you go to not pass rate if you go to the individual metric and you pick two states I think it gives it to you without the gray.
	>> Oh perfect.
	>> Yeah.  So if you go to the individual.  Because I did that for Loss to Follow-up.  I was comparing us and our neighbors I'll say.  
(CHUCKLES).
	>> One thing I want to note that this is not pass rate is most recent or this is a screening.  Yeah, go ahead.
	>> Sorry.  This is like the abbreviated metric name.  This is the rate of not passing last screening.  So not just a diagnosis.  It's okay.  I just wanted to clarify.  
	>> Thank you.  
	So moving on to the next, this is Loss to Follow-up rate.  
	And I do want to specify, this is for the diagnosis.  
	So it is calculated as counts of Loss to Follow-up at diagnosis per 1,000 total screened infants.  And you can see the trend in recent years we see the state trend is going to follow the U.S. trend.  
	And if you see there is a box so if you want to highlight a particular area of the plot, just select a particular area.  
	It will highlight for you.  And it gives you like access to -- I mean takeaways the points of the -- outside of the box.  And this is like pretty more convenient for you to dive into your particular intent if you are working through particular year, not the whole dataset.  
	So this is one thing.  
	And moving on to the next is the diagnosis rate.  
	Diagnosis rate is counted -- calculated as counts of total diagnosis that includes no hearing loss, transient hearing loss or permanent hearing loss.  Per 1,000 screened.  
	And in recent year, you can see that the trend has increased as compared to the U.S. trend.  
	And taking onto also you have these tabs.  And this gives you a little screenshot of this particular plot.  
	So that is very convenient for you.  I can just go . . . 
	Okay.  Here you get the screenshot of that particular plot.  
	Moving on to the next is the enrolled in EI percentage.  So we have heard about EI percentage, especially percentages with rate.  
	So this is total enrolled in early intervention Part C, non-Part C divided by total with permanent hearing loss times 100.  
	And that trend line in the recent year has gone down below the U.S. trend.  
	You can mark it from here.  And it is going down.  
	So this is enrolled EI percentage for -- okay.  Go ahead.  
	>> So I just want to clarify, so this is number of children with hearing loss enrolled in early intervention divided by the number of children diagnosed with hearing loss.  
	>> Yes.
	>> So that would be something really important to put on the graph.  Because if you scroll way up to the very top, one of the things you make note of is right there, the number of children enrolled in EI out of 1,000 screened so my assumption was then that's going to be what I see below when I look of rate of enrollment in EI so that would just be when you're clarifying add to your list to clarify because if I see that then I'm going to assume because it's on your homepage, your rate divided by screening denominator then I'm going to assume that that rate is as well but it's not and that's a really important thing for EHDI Coordinators to have access to.
	>> Okay.  
	So at this point in time we are going to note that.  And we'll work on this to highlight that we are very clear about what the metric is, what the plot is.  
	And since -- and moving on to the next early intervention rate.  
	So that is calculated as counts of infants enrolled in early intervention Part C and non-Part C per 1,000 screened and the rate trend line had fluctuations but it has been near to the U.S. trend in the last two years.  Here you can see.  
	And this is the menu that gives you access to download all the plots altogether.  I'll quickly show you.  
	So hit this download screenshot.  
	And it gives you very nice view of like getting all the plots in one page so that you can make sense or talk about all different metrics with one page, slide.  
	So that was state profile.  
	Next is the state-to-state comparison.  
	So I need to reload.  
	So I'm selecting Missouri from where I belong.  And I'm surrounded by eight neighbors.  
	But I'll select Illinois, closest to me.  
	So we'll just wait for a couple of seconds.  
	Okay.  
	Just give me one sec.  I need to select state from here.  
	And I'm selecting Missouri.  
	And then a comparison state, Illinois.  
	And it will appear side by side.  
	And as different -- has a different color for easy visibility or comparison.  Missouri in sorts of red.  And Illinois in the sorts of blue.  
	So you can walk through.  We have the comprehensive summary as like the state profile.  And again, I'll highlight the year shows you the most recent data we have available with us.  
	And these are comprehensive summary generated out of the latest data suppose Missouri has 2021 latest reported data and Illinois has 2019 recent available data so we do highlight here 2019.  
	So that people do not confuse about like we are comparing in two different states and one is doing better.  So in a you have a couple of eyes there that is from 2019.  
	So moving on to the next, we have hearing loss prevalence.  So here one thing again I will flag that we don't have U.S. trend here.  
	So you won't find U.S. trend.  Instead of that you'll have Missouri and Illinois trend in all the plots so you can compare one state to another.  
	And moving to the hearing loss prevalence we observe the trend closely mirrors Missouri as the neighbor except for the recent year where Illinois friend is increasing compared to Missouri.  And the total screen percentage we are replicating each of these trends, not pass rate we had pretty similar trend in the past.  
	But in recent year rates have been the same.  
	And Loss to Follow-up rate diagnosis is we can visualize that in the past we had a couple of variability or the trends were different from each of those but recent year you can see we are very close to each other.  
	And again we have a couple of like two maps.  I can try Lasso for you.  
	Here I go.  Okay.  
	I have selected -- this is pretty wonky.  So I don't know.  
	Select.  Okay.  
	So it gives you -- it highlights you that particular area.  I'm selecting here.  It takes out all of the dots outside of the selected area and when you get the print you will have that particular area.  This is what I was talking about that Lasso do.  Moving on to the next.  That is diagnosis rate.  Both the trend -- both the trend you can see are similar except for a few points which is also close to each other I'll say.  But you can see that our trends are crossing each of those.  You can signify that particular year that was 2019.  
	So you can derive as many points as you can.  It's on you.  And again we have enrolled EI percentage.  I do take your comment.  We will highlight here also that it is not based out of total screened.  Thank you for that comment.  
	And we are going to -- this is work in progress.  So we are going to improve this.  
	So Missouri trend is in the early intervention percentage has increased in recent year compared to our neighbor, Illinois.  
	But it has remained close to each other.  
	And early intervention rate there have been a couple of fluctuations in the trend.  Missouri in recent years.  Whereas Illinois had fluctuations in the past.  
	And again, we have this download screenshot of this whole page.  You will get this downloaded.  And I'll quickly try to show it to you.  Okay.  There you go.  
	Yeah.  
	So this is the plot you got.  And you can print it like control P or something.  
	>> Fit to paper.
	>> Just remember fit to paper.  Thank you.  So make sure -- because otherwise it's going to print out on several different pages but if you do fit to paper that whole thing will print out really nicely in one page.  
	>> This was our backup.  We know -- we never know what we're going to miss with it.  
	So now we have -- we are left with very less time so we are going to skip this.  And -- yeah.  
	Okay.  Questions.  
	>> Hi this is Liz from Kansas.  And I have to say I'm the first one -- or I have to say that I am not a data person.  I like data.  I know it drives us.  But when I look at this and I see us comparing to states and I know you say that the ranking isn't top three/bottom I think it's really -- there has to be some way to identify that and make sure either you take out the ranking, I don't know.  
	But I can tell you that if my boss looked at this, and he would say, oh, you're ranked 3rd but Missouri is ranked 49 we're doing a dang good job compared to that.  So when we're looking at the standard state, how do we compare that?  Or maybe I need a definition as to what I can tell him what the ranking actually means or does it not mean anything when we compare two states.
	>> Yeah.  That is a really nice point.  
	So we are going to flag that ranking is the distribution of data, not the ranking state as good or bad.
	>> What does that mean, distribution?  
	>> Out of 49 states what we took the data to calculate all of these plots.  
	So this is like that particular calculation or that particular metric where it lies out of all those 49.  But we think that we need to be more clear about that.
	>> Okay.  So -- okay.  So let's say that -- let's say Loss to Follow-up.  And I'm ranked 3 out of 49.  Is that a good ranking?  Or --
	>> No comment.  
(CHUCKLES).
	>> So that specific question we're not trying to say if anybody is good or bad.  
	And --
	>> It looks like it.  It looks like I'm like 3rd in line to be in the top.
	>> Uh-huh.  Yeah, so we try to find the balance between like what's the right amount of text in here because we didn't want you to get a handout with too much information and we fell short of what to put on this page versus other pages so we will account for that.  But the process we try to take into looking at the distribution of data is basically from the highest to lowest values that are available in a year.  The highest rank is going to be the highest value.  We only look from high to low we don't take meanings if high is good or low is good we apply it across all of the metrics so high values of a metric is created the same across we didn't take meaning to you want to have a lower Loss to Follow-up rate versus a higher Loss to Follow-up -- I'll say a lower Loss to Follow-up number versus a higher one I'm frame it that way this is Step 1 we just wanted to take the distribution of the values as they appear and then just put that ranking.  
	This kind of relates back to the heat map we saw.  We pulled the ranking order from the visibility.  So if you looked at the distribution across those years you'll see if you hover over it this one is ranked 49 out of 49 because it has the highest value.  And I get tripped up, too.  But it has the highest value at that point in time.  
	So we will add some more notes here.  But we have tried to take a step back from taking any stance on like this is good or this is bad.  We just wanted to say if you looked at the values, this is what's there.  And then if you looked at the order of the values, like this is where you kind of fall relative to everybody else in terms of like the values that appear in that year for that metric.
	>> Okay.  So just one last comment.  
	So if you're looking at Loss to Follow-up you want to be 49 out of 49.  I mean the lowest Loss to Follow-up.
	>> So every metric is going to be different.  Some you want to be high.  Some you want to be low.  Some you really want to fall in the middle.  So this is the critical thinking part.  
	So it's easy -- as easy as we made this you still have to think about what am I trying to show, what do I want to show, who am I showing it to.  You still need to answer that stuff.  But you can pull that all out to make it work.  So there is still some work that you guys have to do obviously.  You guys do a lot.  
	But we can't -- like I said we're just trying to -- like they said, present the data consistently and you've got to figure out or you can black it out and not use it.  
	>> (Off microphone).  
	(Speakers off microphone).
	>> These are the decisions we made.  These are the decisions we made.  
	You know, we can keep discussing.  
	>> So this is Marsha from North Carolina I'm going to tag onto Liz in Kansas.  But what you have done is amazing.  And I can think of the amount of time this can save me when I'm trying to put things together for my Advisory Committee and other things.  
	I do share the concern about rank.  Because you've got rank highest to lowest or lowest to highest everywhere and it's not the same.  To be -- and I can tell you that the people that make decisions in my state about where money is going to go, that's the thing they are going to see on this website.  Period.  Rank.  And that's not good when Loss to Follow-up we really want to be 49 out of 49.  
	And we don't want to have to explain.  
	But I do -- we've talked a little back here in the corner.  I think there will be more discussion with EHDI Coordinators and I think we can work that out.  But the work you have done here is phenomenal.  So don't let that go by even though we still have concerns.  You've really done great work.
	>> I wanted to add that I did check on the fact that the slides aren't on the app.  And I will -- what?  They are not on the app for this presentation.  
	But I'll -- we're going to try to get that fixed.  
	>> Thank you.
	>> It's part of the process.
	>> And if you still don't get them, just at the end is our emails.  You can also just email us.  
	Should we pick one person or . . .
	>> You can email EHDIdata@(inaudible) we have it up and running so it's a quick and easy way to reach us.
	>> Yeah.  
	>> We tried to make it easy but also really confusing because the only difference you see in the website and the email address is that really period and  @ sign and yes I have written down the wrong thing before just because I write it so fast.  
	Oh, it is on the handout at the bottom.  
	Uh-huh.  I like tiny print.  Sorry.  But it's at the bottom of the flowchart that you got it says you can visit the website and email I believe both are there, sorry, I forgot we did that.  
	And then quick housekeeping, we appreciate all the feedback that you guys have shared with us verbally, we've written down as many as we can I'm sure we're going to debrief after but we had a very quick evaluation if you don't mind taking it and filling in any thoughts maybe you didn't get a chance to share or contact us page reaches out to that EHDIdata@wuso (phonetic) email so we'll follow up with that but if you had a couple of minutes to fill out the evaluation there's a QR code you can scan or a tiny URL/EHDI data.
	>> That has nothing to do with the course evaluation.  So there's different feedbacks.
	>> Thank you.
	>> Did we share where you contact us and drop feedback on the website can we show that real quick.
	>> Sure the contact us button at the top of the page but they are scanning so I want to track.
	>> Also you're going to get an evaluation form at the end of the conference for each session you attended so don't expect to have it right now.
	>> Yeah, sorry; this was a different one because we -- at the Center for Public Health Systems Science we like evaluations we don't do a session without having our own evaluation form.  
	One more tidbit we have a poster tomorrow if you would like to come join us for lunch.  Please stop by, say hello.  If you have any further thoughts you want to chat about, we would love to see you guys again.
	>> If you guys would like to pick our brains or have questions when you start playing with the website, while you're here, feel free.  If we have time we can sit down together and go through that.  
	>> Thank you so much to all of the presenters and people who worked so hard on this.
	>> Thank you, guys.  Do we have the room for a few more minutes.
	>> Yes we do.
	>> So we'll linger if anybody wants to chat about anything too if that's okay.
	>> I checked I don't think there are any other sessions after this.
	>> Thank you, thank you so much for attending I'm sorry we didn't get to say it to the other folks but appreciate you guys so much.  Thank you.  

(Session ended at 3:37 p.m. CDT)