The name of the agency that you will be evaluating in this course?NOTE:?If the agency prefers, you may use a pseudonym for confidentiality

I teach her on an adjunct basis. There we go. We're fully recording. But I have been at college and university teaching for 30 years now. So this is something I've long done. I am centered in the Midwest now, but all of our class times and due dates and so on will be posted in Eastern time. So no matter where you're located in the country, please note that it'll always be Eastern time. My background is that I am a social personality psychologist. That's the area my Ph .D. is in. But I also have a concentration in I.O. psychology. So I spent plenty of time in business school as well while I was a graduate student taking courses there and earning my concentration. I'm excited to teach program evaluation. I'll hold off on a few minutes in terms of exactly what this course is because it's run differently than your other research courses. from what I understand. And so it'll require you to do a project and to be thoughtful in what you're examining, but you won't actually collect the data. So we'll talk about that. It's only an eight-week course, so we can only do what the college thinks is reasonable for that eight-week period. And program evaluation is usually a multi-month, if not multi -year, kind of endeavor. so my next slide just is one that says introductions the last time I taught this course as I said there was no one here and now there's a good number of you here so we won't have a lot of time to delve into like which projects you're thinking about and so forth yet later tonight we might have a little bit of time but I want to make sure I get through all the material. So I am just going to ask you if you're on camera and on microphone. I think everyone is in this class. Not Julio. A couple other people might not be either. I'll have Julio and anybody else who's not on microphone just introduce themselves in the chat room. Say who you are, if you'll be attending residency this summer, and maybe where you are in the country. When you say who you are, please just keep it to your name. Again, we won't have a lot of time tonight to get into fun stuff. And that would be your preferred name. What do you want me to call you? We were just talking about this as class was opening, so I'll hope to get it straight as we move along. So I'm going to call out names because I don't want to call out names because I'm going to mispronounce them. So I'll just let you each jump in, just your name, where you're located, and if you'll be at residency this summer. I'm the first one to log in. So that's my name that you didn't want to mispronounce. You are one of them. Yes. My name is Liz Nasing. I used to not like my name because my family, you used to call me Lizzie, but now that I'm growing up and everybody calls me Lizzie, I got it. I'm used to it now, Lizzie. And I'm in South Florida, and I am done with residency. I have been through residency twice already. Welcome. Thanks. Hi, my name is- Well, I guess I'll go. I'll go next. Hi. Welcome to see you again, Dr. Wade. I'm Kevin Tucker, and I reside in Stone Mountain, Georgia, and I will be attending residency. Okay. Thank you. Latasha, I think you were coming in. Yes. Hi. My name is Latasha. I'm in Rhode Island, and I will be attending residency this summer. Good. Hi, my name is Juliana Green. I'm in Middletown, Delaware, and I'll be at Residency 2 in July. Hi, I'm Farron, and I'm at West Central Florida in Clearwater, and I'll be at residency too this July. Great. And we've got Adelina. I believe I'm saying her name correctly. Checking in as well as Julie. Okay. Who else wants to let us know who you are? I'll go. Hello, good evening. I'm Esther Champon. I live in New York. I'll be attending residency one and two this year. Great. J.C. Alonzo. I'm in Fort Lauderdale. I've been to both residencies already. I should have remembered it was J.C. Thank you for reminding me. no worries it's okay who else hello my name is roxanna my camera is not working unfortunately um i'm from chicago but i just moved to um the treasure coast area and i will be our residency this summer very good i i know i saw your uh introduction this afternoon because i had to look up where where treasure coast was so thank you hi i'm julie um it's everyone somehow your um your audio is not coming through julie okay very faint yes so julie did mention her uh residency is completed as well uh susana you may not have her microphone okay i think they got a good feel for the oh valerie go right ahead sorry my name is valerie i am attending residency one and i live in dallas texas great all right i wanted to get a feel and also for those of you who may be attending residency to um be able to recognize each other look out for each other there if you want to connect in terms of various interests. Susanna just came in as well. She'll be at residency too and is in Florida herself. Okay, very brief introductions, I know, but I appreciate having a sense of who everyone is tonight. I'm going to jump in and talk a little bit about what we're going to get done this semester. It's an ambitious course. Although you won't be collecting data. You'll still be designing a program evaluation, evaluating mock data, and creating a full-fledged program evaluation report by the end of the semester. You'll be turning in pieces as we go through each week as your assignment. Sometimes it's incredibly brief. Portions will be a single sentence, but other times it's a lengthier section of the report. And this week it's just a logic model that you'll turn in. But what you will do in your discussions, we'll talk more about in a few minutes, is actually think through your assignment each week. So what is program evaluation and what is its purpose? Essentially, what you want to do with program evaluation is evaluate the effectiveness of some program that already exists. This is not like designing a research study where you manipulate the variables you have to take what you get when it comes to a program and evaluate what's in place so again you're going to assess the effectiveness of the program we'll do that on a single measure this won't be as a as involved as a typical program evaluation will be um but what's nice about this is this is an area of opportunity for our students who finish PhDs. In the field, down the road, you may be asked to do this kind of work. You may apply for this kind of work. You're not trained as clinicians, but you are trained to really understand data and how to evaluate and assess that. So when it comes to programs that are usually out there and running, clinicians are doing the day-to-day work they're meeting with, doing the treatments, etc. But they're not often as trained in data analysis. They need to bring in outside consultants to assess the overall effectiveness of the program. And that's where you all can come in and have work opportunities in those settings. The difference between, as I mentioned, running a research project and evaluating a program is that you cannot change the program at all. We'll talk, of course, about ethics this week as well. But just from a practical standpoint, the goal is not to go in and change things. The goal is to go in and evaluate what's already being done to see how well it's being done. That is, is it effectively creating good outcomes for the clients who come through that program? We're going to ask you to focus on non -profit agencies. You're going to need to select a non-profit agency this week. They measure the effectiveness on something other than profit. So, of course, you can always evaluate a profit program for whether or not it's making money, and that's how some things are evaluated in terms of whether they'll continue. But for our nonprofit agencies, we're not interested in that. We're interested in knowing did the psychological outcomes that were intended to be changed, change, and did they change in the correct direction. The reason we do a program evaluation in the first place um is to make sure that the service being offered the clientele or the community is effective right so we're going to assess for changes that are needed in services offered or whether we can hold the course and tell pretend potential uh grantors that uh the program is effective enough they should continue to give monies give granting money for that agency to continue the work it's doing and of course to assess whether or not anything needs to be fixed if something's going poorly we need to address that yes so you want to look at programs that are up and running you want to look at non-profit programs at the end of the whole report when you're done you'll make recommendations but again it's not your job to go in and manipulate anything. You can't randomly assign clients to different services in the program. You have to look at what's already occurring and measure its effectiveness. That's the end goal of program evaluation. So this is not like a clinical trial where you're designing the study from the outset, arranging and organizing and manipulating the variables you're taking what's given and you're evaluating its effectiveness um when it comes to consent the good news here is that clients coming into these agencies generally sign a consent for treatment and evaluation um so uh They may even sign on for training consent. So maybe a new clinician working side-by-side with a veteran clinician. But in that, they will agree to having the program evaluated. The only time you need consent, will not apply to this course, is if you're going to later publish research based on the program evaluation. That is rarely the case. Usually you're not conducting it as a scientific endeavor looking for scholarly publication, but instead looking at whether this is effective or not, being able to report that back to a board of directors, to granting agencies, to folks like that. In that case, you do not need additional consent. When clients come into these agencies, they say not sign off on that when they're agreeing to their treatments. Just to sort of finalize this slide, kind of underscoring the idea here. Yes, your question here is can a nonprofit be your place of work? Yes, that's fine, because you're not actually going to conduct the research. you're not going to actually collect data there's no conflict of interest there and so you may use your own agency and we'll make this first couple of weeks easier in terms of when you're assessing what the agency does and what program you'd want to look at for its effectiveness so good question Roxanna again essentially what we're talking about with program evaluation is programs where there are already psychologists in place doing the work of therapy or training or counseling. And then people like us who like the data end of things are brought in to assess the effectiveness of that service, treatment, counseling that's being offered. This is not to say that every clinician isn't a data person, but in general, they don't have as much training in these areas as we do. So they're usually master's level counselors, sometimes PhDs, but usually with a counseling emphasis, not so much the clinical emphasis, which might have more data experience. Okay, we're kind of going to jump into the discussion because it's actually the work of the week, and it also helps you figure out what we're doing this semester with the course. So all of the discussions and written assignments in this course will focus on a program evaluation of one specific nonprofit agency. Each week's discussion thread will focus on the issues that will be reflected and relevant in the week's written assignment. So you'll use the discussion as a means to clarify your understanding on the issues related to the topics each week. This week you're going to begin your work in the course By selecting an agency, interviewing key personnel at the agencies regarding the programs they provide, or, and this is a very big or, reviewing their website, okay? Again, eight weeks, not everybody's going to have a contact at a nonprofit agency or have time to contact someone who could meet with them within the first five days of the course and so on. So since the pandemic, this course has been revised. You used to have to contact and complete the interview the first week. Now you can review a very good website. You can't choose an agency that doesn't have a very information-filled website if you choose the route of reviewing the website. And what you'll do from either that interview or that review of their website is you'll develop a logic model that provides a graphic overview of the program's purpose, services, inputs, activities, outputs, and outcomes that will be evaluated. I know that sounds like a lot, but when you see what the model actually is, I'll show you an example. You'll be Googling this week. Google is your friend. but if you use one much like I'll present, you'll see that there's just about three columns to complete with bullet points. We're not going to need long paragraphs. We're not going to need involved sentences. In this logic model, we just want bullet points. In your discussion, you of course have to write in EPA style and think through some of these things. So again, please note that due to the very short nature of this course the duration of this course it will not be possible to actually collect and analyze data from that agency you'll create data files using hypothetical data in future weeks five and six is when we'll get to this and there's even some discussion behind the scenes again i deliver the course teach the course there's a supervising professor, given how large this course is, they may create a data bank for you. So we're still discussing that. Won't come up till week five. It'll be pretty straightforward how to create data because you'll only create 10 lines of data if you do it yourself. And then you will replicate that 10 lines 10 times, so it feels as though you have 100 people, clients, whom you're analyzing on various data points. The truth of the matter is in this course, you will only assess a single outcome. So we're going to talk about a lot of possibilities for what you can look at tonight, but essentially it's going to be the best measure of effectiveness for what that program is trying to do. So you don't have to take in, you know, this data point, that data point, a third data point, create all these outcome results, one result on one variable, one effectiveness variable. I very much recommend this week that you review the PowerPoint presentation of the program evaluation conducted by a previous professor of this course at the Broward Partnership for the Homeless. So it'll be a PowerPoint presentation that takes you through a program evaluation. By reviewing that, there are not PowerPoints for this book, but by reviewing that PowerPoint, you'll get a nice hefty sense of what a program evaluation would look like if it were delivered in the real world. Okay, in the discussion, you'll address the following. Items one, two, and three are due by Wednesday night. You know, discussions at Kaiser, you need to post your first discussion by Wednesday and your follow-up by Sunday, you're actually only going to have to finish the first three points by Wednesday night. Then you'll make another post by Sunday in which you address item four. So what will you be addressing in your discussion thread? the name of the agency that you'll be evaluating in this course if the agency prefers if you actually interview someone as opposed to use their website you can even use a pseudonym for confidentiality purposes again we're gathering no data but some folks may prefer that their agency isn't sort of scrutinized too much in a class like this so if that's asked of you because you really do an interview then use a pseudonym because the vast majority of you are just going to evaluate a website for a nonprofit agency don't have to worry about changing the pseudonym to a pseudonym because that won't have been asked of you by anybody on the agency should provide psychological and or social services and be a not-for-profit agency. The next thing you're going to do is discuss the ethical issues involved with being a consultant for a social services agency. How will the confidentiality of the clients be maintained throughout your evaluation project? Well, you could be smart and kind of obnoxiously say it'll be protected because there is no real data in this study. But we want you to pretend that there is real data. And so you need to think through things like, where would you store the data? Where would you keep consents for being in the study? And how would you keep those two things separately? The data from the consent forms. You need to keep in mind And any legal issues that could come up, anything that needs to be considered at all. Because you're not going to manipulate who gets what services, you're avoiding some of the ethical conundrums that clinicians face when people come into their programs. and they're assigning them sometimes to various treatment options. You won't be assigning anyone to those things. So you don't have that ethical concern of, oh, do we think one treatment actually is going to show out better than another treatment? And how do I put someone in that secondary treatment if we really think one is better and so on? Of course, you can't know until you do the research. um so it's it's not something that should enter a clinician's mind but it might so again you're not going to collect the real data here but because you're going to model it um you want to discuss in your discussion and you'll read this in the first chapter of the book the ethical implications that could arise. I already advise you to look at that. Yep. When you're looking at the websites, you want to look at what it is they do, for whom they do it, what kind of facility they keep. If it's a inpatient, inpatient situation, maybe you could say how many beds they have, how many units they have, how many people they can house at any given time for those treatments. You'll want to notice, you'll want to report what it is they do in these settings. I should move on from the ethics here to the next slide because the third thing you'll want to be laying out here is the personnel you interviewed or the website you reviewed. Some of these questions you won't have to answer if you review the website, like who can provide the best information about the services delivered by the agency. But let's say Roxana, using her own workplace, might be identifying not the person by name, but the person by position within the agency and saying whether that was a good person to have made these assessments and given you the information. You want to determine who can give the best information about how the agency currently measures outcomes related to effectiveness of their programs. So maybe this nonprofit's already in the business of program evaluation. What do they look at? What do they evaluate? And then you're going to review the components that are required for your logic model, right? So you're going to be asking yourself, what will I assess? Will I assess services, individual counseling outcomes, group therapy outcomes? I'm going to encourage you to stay away from agencies that work on case management. If you were to deal with a case management program evaluation, that gets really hairy. because they're sort of a centralized place that's sending out people, various cases to be treated in other programs. Then you'd be having to collect data from multiple agencies and so on. So you want to stay away from case management. Look for people that are doing, agencies that are doing maybe alcohol or drug treatment, treatment, treatments for depression, eating disorders. You might look at a homelessness environment, just like the Broward County. But then you have to really think through what I'm going to measure. What is an effective way to measure what I'm interested in? But the first thing is to figure out what you're going to assess. Are you going to assess certain services like counseling, group therapy. What are the outcomes that you're interested in? You must have measurable outcomes that are non-religious in their outcome. So we're not interested in looking at measures of religiosity here. We're interested in looking at psychological and sociological outcomes. So any outcome that's going to give you a mean, so it can't be a yes, no question. Do you have housing? Do you not have housing? We don't want that. We want something that will give you a mean. How are they doing pre -services? How are they doing post-services? You're mostly looking at a pre-post -test on some variable, like treatment outcomes. How is their depression going in? How is their depression upon finishing six months of treatment? Upon finishing 12 months of treatment? Something that will provide a mean, an average. Not a yes, no question, and not a question of religiosity. Let's see, give you a couple examples here. You can look at mental health symptoms, job training outcomes, trauma symptoms. You want to look at primary symptoms, things like reduction of symptoms, not secondary measures like peer involvement or family connection or those sorts of things we want to see. given this service offered, what is the outcome of that service? So a very primary outcome, not a secondary one. What you'll do here, we're going to get to the logic model in a minute, but in your discussion is you'll again review those components. The person you interview or the website you look at should be able to explain the inputs, the activities, the outputs, desired short-term intermediate term and long-term outcomes you're not going to evaluate all of those but in describing the agency well you want to have a sense of each of these reading your chapter this week will get you more deeply into what each of those components are then you'll summarize the results of your interview what is the key information you learned and what will you use from that in your logic model. So again, because you'll be working on your logic model and turning that in Sunday, this fourth part of the discussion can also be turned in in a separate post on Sunday. First three parts of the discussion should be completed by Wednesday night. The fourth one, because you'll be kind of working through that as you do your logic puzzle, logic model, you won't need to turn in until Sunday. Okay. These were some of the alternatives that used to be offered, so I'm mentioning them again. If you couldn't go meet in person, of course, you can do it by telephone. And now we're even allowing just that you review the website. Again, be sure you pick a website, an agency that has a website where there's sufficient information on the website to determine the services provided the staffing available and the treatments and services provided um we're a big class so i won't be able to talk about everyone here but if some people want to throw out an agency idea they're considering um we can talk through where you might turn for information hopefully their website And think about, you know, what exactly is the service offered that you want to look at pre-taking that service and post -completing that service? Anybody want to throw out an example of a nonprofit, even if you're just thinking about it at this point? I work with my job that I work with, Tides Family Services in Rhode Island. Okay, LaTasha speaking. It took me a minute there because I thought someone else was coming in. So what was the name of the service again? Tides Family Services. Okay, and what services do they provide? So it's a nonprofit service they provide to the community for mental health for the youth. So they serve the youth. I'm really glad you brought up youth. It's coming up on a couple notes later. Be sure you're aware of what age range the agency serves. Because, Latasha, in your case, you'll be wanting to look for a measure that's been validated and is reliable for youth, for the people, the age that your agency treats. So we don't want to grab a measure out of nowhere, an instrument out of nowhere. We want to make sure that it's been validated for the youth that are served there. Is it all mental health problems? Is it specific to depression or anxiety or eating disability disorders? I want to say at risk. So, yes, depression, anxiety, eating disorders. I think the age range is between 10 and 18. So most likely kids that are at risk to be placed in inpatient hospitalizations. So we kind of create that block to kind of go into the home and help the kids and come up with some strategies to prevent inpatient. So you're not the inpatient agency. You're the agency that's hoping to get them maybe treated well enough in advance that they don't even need the inpatient. So you're reaching them when they're at risk. Yes. And do you know how they're identified? Well, we have referrals. Usually we have crisis lines. We get referrals coming in from that. We have a mobile crisis team that goes out and assesses the families and then decides what program they would, you know, benefit. Excellent. Excellent. You're probably going to have to look for a measure that doesn't target any particular mental health disorder then. So you wouldn't just look at a depression instrument, but you'd look at a youth mental health instrument. And what you'll want to see is at first um contact how do they do on a test of mental health and then after your services which are what is it a do they come for therapy on a weekly basis do they meet in groups individually do they just meet with people in their homes who are sort of outreach to um circumvent any treatment yeah so i'm looking to measure the ohio scales that we do in the beginning when we start with a youth um we have the first 29 days to do a youth ohio assessment which identifies any risky behaviors and then it also scores them on their functioning score like how well do they function at home at the community in school and then every 60 days that assessment is re about like redone with the clients. And I think the program evaluator, she collects all that data, but I'm looking to look into those assessments to see how. Yeah, well, you don't actually want to look at what the program evaluator is doing. You want to look at what the agency is doing. After they give the Ohio instrument, what do they give in the 60 days in between to hopefully see an improvement at 60 days. So they receive clinical and behavioral support weekly. So they have a clinician who comes out weekly and then a BA who also comes out weekly. Outstanding. Okay. I'm excited about this program. I hope it's as effective as it sounds like it should be in my head. Okay. So what you're going to be evaluating then is either a 60-day window or a six-month window, which would be three 60-day windows. You'll have to decide. And then you'll want to do a single pre-test and a single post-test on the instrument you think is strongest. Could be the one they use already, as you say, the Ohio, since it's reliable and valid for youth. It could be another one. If you decide that you want to see independent of their assessments, are we seeing a better post-test result than the pre-test situation? And all of these things have to be means, averages. So the first thing you mentioned, I think, was like a behavioral risky behavior kind of thing. And risky behavior may either be in place or not in place, occurring or not occurring. If it's a yes-no answer, that's not the kind of assessment you want to do. Make sure it's going to leave room for a mean result. So we don't want yes-nos, and we want to make sure that it has reliable, valid, mean outcomes for the pre-test and the post -test. There was one more thing I was going to say about that. Oh, I know. We want to make sure that the instrument isn't a purely screening instrument. A screening instrument might ask, have you ever considered suicide, for example? Or have you ever engaged in ex-whisky behavior? Instead, we want to say, over the last two weeks, how often have you? Because have you ever questions, a screening question, may have the very same results at the beginning and at the end, right? Just a yes, no, yes, no, or yes, I have, no, I have not. But assessment questions will really get at over the last 60 days, over the last six months, to what degree? We're asking more to what degree than have you ever type questions. We don't have a lot of time tonight. Do we have anyone else? Thank you for that example. Who wants to mention? J.C. is asking, would a counseling department of a college university fulfill the requirements? Probably not if the outcomes of a university department are just successful completion of coursework. If you're talking about a service that's being offered, you would have to have incoming clients to that counseling department who have something measurable at the outset that you think would have changed by the time they finish with that counseling in the department. But just measuring sort of educational outcomes won't be enough for this course. One more example. Anybody able to give me an example? And you'll see lots more examples brought up in the book. And you can look at the Broward Partnership for Homelessness. My agency is New Horizons of the Treasure Coast. We are a Baker Act facility. we house children outpatient children inpatient and adult inpatient as well as outpatient services and we also have a substance ***** unit and a group home unit as well okay so you're going to want to focus on a single service right so you're going to have to choose outpatient or inpatient I was thinking more in the, because I'm really interested in knowing why we have such a reoccurring admissions of past clients and both our children and adult units. That's a research question. So you don't want to ask a research question like a clinical question. Why do we have reoccurrence? We're looking at program effectiveness. so it's not just whether they come to us and they come to us again but it's some measure that has a mean an average and we see how they are when they arrive and how they are it would be ideal if it was a discharge but sometimes we can just measure it six months or 12 months sometimes it can be at discharge but it that can be so lengthy that as a program evaluator we usually have to put a time frame on it so we can report to our granting agencies. So you don't want to ask the question why something is occurring. That's a research question. You're thinking more like a clinician. You have to think like a program evaluator. When they arrive, are their scores less good than when they leave or when they finish or when they've had six months of treatment? So then the focus would be on, because we assess them on the PHQ-9 in the Columbia when they come in. And then in three months, we do the same, depending. Some are, not three months, six weeks is how long they stay with us. Okay, so the first thing you're going to have to choose is whether you want to focus on the inpatient services or the outpatient services. Inpatient. Okay, go ahead. So then we do the assessment, the PHQ-9 and the Columbia. In the beginning when they come in, and the reason why they were Baker Act, usually it's by police or, you know, they come from our outpatient department

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.