Ed Zerylnick (a current DPA customer) and Rob Mandeville, SolarWinds product marketing manager, share the secret to being proactive with DPA alerts.
Okay, thanks everyone for joining us today for our DPA Beyond the Basics. Looks like this is Episode 2: The Secret to Being Proactive with Alerts. The episode two kind of reminds me of a Star Wars theme. That's kind of cool. Who am I? My name is Rob Mandeville; I am the Product Marketing Manager for Database Performance Analyzer, which I will just call DPA, because that's kind of a mouthful. But I'm not just on the marketing side. So, I've been a production DBA, developer, data modeler, working with databases for quite some time over the past years. The last five of which have really been focused on performance. Thus, the product that I support. Just a little bit of house cleaning up front. If anyone has any questions, let's go ahead and use the Q&A section. That way, everybody can get the benefit of seeing the questions. And by the way, go ahead and ask them, and when we get to a logical break or something like that, I'll try to monitor that as much as possible, and we'll try to get those addressed in real time. Okay, fantastic.
So yeah, like I mentioned, this is our second in a series of Beyond the Basics, where we try to go deeper into maybe a given feature or a set of features, or specific functionality within DPA, and today we're here to talk about alerts. So, in my personal role as a production DBA, I had a fairly strong opinion about when to configure alerts and how often I wanted to get notified and stuff. Really, I didn't want to necessarily get woken up at night just for information, right. I always wanted—if there was some kind of action that I could or would take from a specific alert notification, no problem. If it was just informational, you know, maybe that should just end up in an email so that I'm aware of it the next morning. Something like that. So I did want to be very careful on how I personally configured my alerting and notification. For this reason, it was fairly intentional that when we created DPA, that it comes out of the box not pre-configured with any active alerts. So again, that was definitely done by design. However, we have gotten some feedback from some of our end-users that would like to have seen at least some basic kind of alerting pre-configured.
So, toward this end, I wanted to get a sampling of all of you today, and see what you think about this. So, let me see if I can get this poll open, and this is the first time I'm doing the polling, so bear with me. There we go. So, you'll see on your screen that there's a poll up there, if everyone wants to go ahead and vote that up. I'll give you just a minute to vote on that. Okay, so interesting. It looks at this point, there's a few more that still have to chime in, but looks at this point like the eyes have it. Like, that people would like to see at least some of the basic alerts pre-defined or pre-configured out of the box. Okay, fantastic. Thank you all very much for the feedback. You can still vote on that for the next minute, maybe.
Okay, so while you vote on that if you haven't already, let's go ahead and move on with our webinar here. So, I'm going to turn it over to Ed now, to introduce himself and tell us a little bit about his DPA story. Ed?
Yeah, thanks Rob. This is Ed Zerylnick, and I've worked with databases for a really long time. First as a 4GL developer, and more recently as a database developer in DBA. I am working for Wescom Credit Union, it's a large Southern California credit union. And we also have a for-profit arm called WRG, Wescom Resources Group. We make software for other credit unions. In particular, online banking and mobile banking, and for many of our clients, we also host the software that we sell to them on our own servers, so we end up—myself and the others on the DBA team, we end up taking care of about 100 database servers for both the credit union and for WRG. And I'm now scheduled to go into a story, and this'll be the longest of my contributions here, and it's about moving from reactive to proactive using alerting. And when I joined WRG, that was about seven years ago, the software maturity was not—well, they were a fairly young organization in terms of software maturity, and there was no automated monitoring of any sort. In fact, every day there'd be some client that would have one of our generated, generator- connected services just disconnect, and the only way we were finding out was when the client would complain to us, and that was actually every day.
So, one of my first assignments was to write an alert system that would report on these particular outages, and that allowed us to reconnect the services before the clients even noticed. That, of course, improved our reputation for reliable software quite a bit. But that was our only automated alerting at that time, and other things would happen. Servers would go down, clients would not be able to use their product, and at that point, they'd always say, "Oh, it must be the database!" And, in fact, DBA at that point stood for Default Blame Acceptor, because that was always the assumption. And we spent a lot of time when it wasn't the database proving that it wasn't the database, but there were plenty of times when it was the database. And it was total chaos when this would happen. There'd be 15 people on a call and you'd be on the call, but then they'd be calling you aside from that, calling you directly, and IMing, and emailing and wondering what's going on. And this is like the worst environment that you could possibly have to be troubleshooting a problem, trying to figure out what's going on and then what to do about it. But this is what was happening.
During a serious intermittent problem, my boss's boss was hearing from me, basically, information that he wasn't getting from anyone else. And I could tell when the problem was about to happen, what the amplitude of the problem was, how many times we had it, et cetera, and he came over and he said, "Well, what are you doing? How do you know this information?" And what I had done is, I had downloaded a free version of the predecessor to DPA, and it was telling me about this problem. It wasn't a database problem, but the database was having issues as a result of a network problem, which was totally undefined, but he started looking at the screens of this product, and he's a very visual person and there's all sorts of charts and graphs. And this really, really turned him on and he wanted to know about it and if the full version of the product had more features. And yes, it did, and he just said, "Okay, we're buying this." "We're buying the full version and we're putting it on every one of our servers." And that made a huge difference, because then we could do automated monitoring, automated reporting, automated alerts, and this really started making a difference along with other best practices, of course. And we've moved from being the default blame acceptor to a much more balanced position where the database isn't always the problem.
The DBA group is known for having our act together because we actually, very often, know what's going on before other parts of the organization, before the customers know. Using alerts, we're often able to head off problems from even starting. Other times, of course, you can't always avoid problems, but we know when they're happening. Just the other day, there was an issue, and the alerts came in and we started working on it. When that phone call came with 15 people on it, we told them, at that moment, "Oh we just fixed the problem, please retest." And this has been just a really wonderful change of pace from the reactive chaotic mode of the past. So, we get to troubleshoot and resolve issues right away without having 20 people breathing down our necks and second guessing us. Life is good, and life with alerting is much better, not only for the DBAs, but for the customers, the customer service reps, and everyone else. And that's why when SolarWinds was looking for someone to participate in this webinar, I was glad to volunteer to do that. And that's the end of my story. Back to you, Rob.
All right, thank you so much for sharing that, that's pretty awesome. I like the default blame acceptor. [laughing] It's a good acronym for DBA. So, real quick before we move on. We do have a question from the audience, and it says, "Will you have the ability to set up DPA alerting from the NPM, like from the Orion side, if DPA is integrated with NPM?" The answer currently is "no." However, that is on our roadmap. That is something that we definitely want to expand further. Just so everyone is aware in this call, the Orion Platform is a .NET. That's what it was developed on. The DPA platform is a Java app, so as you could imagine, those two don't necessarily integrate fully, but we are working on what we call JSWIS. So, it's a Java SolarWinds— what's the IS, Information Service called? Basically a web service call, that we can start to leverage some of the alerting functionality within the Orion side, but pulling the data from DPA. So, that is definitely on our roadmap. Look for future good stuff to come from that, but right now that integration for alerting is not there yet.
Okay. Thank you very much for the question though, and let me go ahead and share my screen. Let's get started in the actual dive into alerting. Bear with me just a moment while I get set up. Okay, so hopefully everybody can see my screen. This is DPA proper. And, well, let's just jump right into alerts. So before we dive into specific alerts, I wanted to explain the tabs across the top here a little bit, just so everyone's aware of what they do, how you can use them, things like that. So, the first one here is Alert Status. This is just a quick snapshot of the status of any alerts that you have defined and that are active, or running against any of your monitored instances. So, you can click on the alert name here, and it'll bring up the definition of the alert, in case you want to edit it or change anything, you can do that. We can get the current value. So, if I click on more here, it'll show me the current value, the last time it was sampled. And you'll see here the last time this alert actually fired, or went to get data, you'll see the date on that. Or sorry, that was the last change for the alert, my bad. But also, we can check on the history of it. So, if I check on the history I can see kind of when it ran, what the current status was, looking back in time.
So, some of these I have set to only run really once a day. But they can be much more aggressive if you guys— depending on how you set up that interval for the alert. Okay, we have Manage Alerts, and I'm actually going to come back to this one, because this is going to be kind of the meat of the webinar here. Definitely more into deeper dive than that discussion. So, Alert Groups. Alert Groups were created to help manage or maintain alerts to instances. So, especially for those of you— if anybody's monitoring a really large environment with a lot of instances, sometimes when you set up the alerts it's like, oh now I have to go define this alert for each instance, right? And you have to go into each alert, you have to do that mapping. So what this allows you to do, and I'll just go in this AG1 as an example. It allows me to have some kind of subset of alerts, and I could bring all of them over if I wanted to, mapped to a specific set of database instances. So again, it was really created to help you more easily maintain and administer the instances that you're monitoring within your environment.
All right, and then the last tab here is called Alert Blackouts. Now, this would be if you want to set up a time when you don't want to receive notifications, the alert may fire, but it won't send anything out. So, we're still watching, we're just not going to notify you. So, during this time, if I want to click on let's say this first one here, I just created this earlier today. But let's say this is— maybe peak time is not over the weekend or we really don't have end-users that rely on the database during that time. So, me as a DBA, I may not need to know if anything is going awry during that time. So I can go ahead and set up a blackout for Friday, starting at seven o'clock or something like that, and then waking up again starting Monday morning.
Okay. Oh, one thing to mention in the alert groups here. This can be really helpful from a standpoint of maybe you want to have different thresholds set, or maybe different behaviour, based on the grouping that you're going to define here, and probably a really good example of that is if you're going to have a production environment versus pre-prod, or non-prod, right? You may want to have your alerting behaviour different depending on if it's a mission-critical server versus just something in my test sandbox area. Okay, so let's move on to our Manage Alerts here. And when defining alerts, we give you these four main categories, so you'll see here the alert categories. We've got Wait Time, we've got Resources, we've got Administrative, and we have Custom. So starting with the Wait Time, this is— If I do this drop down, you'll see it really focuses on the dimensions within DPA. So, if I were to go back to, let's say our home screen. Let's flip back here. Let's dive into an instance. We'll pick on this 2016 SQL instance, and you'll see these dimensions across the top where we gather wait time statistics on. So we've got the SQL Statement, we've got the waits, we've got the programs, we've got the database, we've got the client machine, DB user, things like that. That's really what's going to map to the wait time alerts here.
So, hopefully that makes sense. And this is all the data that we're collecting wait time statistics for during our quick pull. So, during our sampling that we kick off once every second by default. Okay, so at this point with the wait time alerts, I'm going to kick it back over to Ed for maybe how he utilizes it in a real-world example.
Yeah, thanks. One of the really critical ones we use in this particular category is Total Database Instance Wait Time. And when wait times go really high and if you're supporting, for instance, a web app as we do, wait times go high and timeouts occur. Clients or customers can't log in and things time out. So that's really important, but another thing that a total database instance wait time gives you a warning about would be if you are a victim of a distributed denial of service attack, then you will see the wait times ramping up, and this alert can put you right on top of that sort of thing. And also blocking wait time, I find to be something that we want to know about, because in many instances blocking is something you can do something about, and actually you can even kill processes inside of DPA. So, it's handy that way as well. All right, back to you Rob.
Alright, excellent. And one thing I should tell you about in this one is we do have some custom ones as well. So, if we do want to define a custom one for, let's say SQL Server, let's go ahead and choose that one, then it lets you define exactly what you want to be alerted based on. So you can filter a little bit on the wait events, like as Ed was saying, if you have a locking and blocking problem or something like that. Whoops, guess I have to select one. So I could alert based on any of these kind of conditions. So especially if you notice, when looking at your performance story or picture of your environment, if there's something that tends to happen a lot in your environment and you want to know—maybe it happens at like eight o'clock in the morning, or maybe month-end or something like that, and you know that there's a specific behavior or pattern to it. When that stats to occur, now you can get ahead of the game, right, and be alerted based on that. Okay, cool, cancel out of there.
So, the next category is going to be Resources. Now, these are going to be based on the resources, as defined in DPA. So, out of the box, we have some pre-defined— let me try that again, pre-defined thresholds. Say that ten times, which can be modified either globally or maybe for a specific instance, stuff like that. So the resource alerts here within our alert manager is— they're actually inherited from how those resources are defined within DPA. So as an example, let's jump over here to resources, and you can see here that I'm within this instance. We're still picking on this 2016 SQL instance, and I'm on the resources view. So down here I can see the Signal Waits Percent, or maybe the VM CPU Ready Time. By the way, if you're running on a virtual machine, and we kind of discussed this in the last episode, but we pull a lot of metrics if you're using the VM option, from the vCenter API. So, not only can you alert on database instance-specific things, but now you can alert on physical host kind of resource metrics as well, or maybe guest operating system, or storage latency, or really, whatever the case may be. But if I click into here under settings, I can see how I have it defined as a default, so it looks like we have critical set at 20 milliseconds. So if I click okay, we'll go back in there. But that also—oh, not milliseconds, sorry, that was 20%, so if my CPU Ready Time is 20% or greater, that's kind of in the critical category. But, I can change those thresholds just for this instance, or I can actually change the default here as well for any of these.
Okay, so based on the thresholds, how they're defined within resources, now that is going to be how we can define the alert based on that resource metric. So if I do it just on a single resource metric, let's still pick on that 2016 guy. I can pick a category. Let's just do CPU. Nah, let's do Ready Time again. And then what this means, just so you know to understand the rest of the alert parameters here. When I say percent meaning metric alarm criteria, we actually do our alert interval once every 10 minutes, so our sampling, though, of that resource metric is more often. It's like once every two minutes, let's say. So, we'll have five samples within that 10-minute interval that the alert kicks off. So this is just saying that the number of times that we exceeded the threshold has to be 75% of the time. So, about four times during that 10 minute period, it has to have been over that 20% in order to meet the critical alert level. So hopefully that makes sense to everyone. If there's any questions about that, let us know or you can follow up after the call. So at this point I'm going to turn it back over to Ed, and he can talk about how he may use the resource metrics in the real world.
Well, as a matter of fact, I told a story about how the CIO noticed that I could tell some things about what was going on with the network, and in this particular case, the network was affecting signal waits. So, I was keeping track of signal waits, and once we had the full product, I created an alert based on signal waits. And that's basically telling you your CPU's are stressing, and if you get this often from a server, then we start investigating. One thing I might mention about the product is that if you have a very lightly used instance, occasionally you'll get false positive high signal waits, and using the settings as Rob was showing, you can actually basically turn off— for particular lightly used servers where you get false positives, you can just turn off the accumulation of signal waits. We found that useful. All right, that's it for my story.
All right, perfect. And yeah, so this was one that Ed had defined as a custom alert for looking at signal wait specifically. We also have one within the resources itself, so if I wanted to— Let's see, I think that's going to be under CPU.
Let's see. Yeah, I'm not seeing it. Maybe all categories, let's see if I can find. Okay, I'm not sure why I'm not seeing it. [laughing] But the signal waits as a resource metric is certainly in there, so it's signal waits percent, and yeah. So, we do have one that's pre-defined for this specific resource metric, or you can use a custom alert as well. Now what this one is, let me see if I can go into settings here, it's not going to define it here. But basically, this does a— it's a delta, so what it does is it looks at our signal waits from, let's say time zero, waits looks like one minute, takes another sample at time one, does a difference, and then divides by the value that was time zero. So it creates this percentage of signal waits, but as a delta between two timeframes. So, that can be beneficial. I'm sorry, it doesn't divide by the value at T-zero, my bad. It divides by total wait. So, it takes the signal waits time, as a difference between two points in time, divided by the overall waits between those two points of time as well.
So I'm probably not describing that very well, but basically, what can happen is the denominator can go down, so your overall waits can actually decrease, which can drive up your signal waits as a percent, right? So, it can actually be, even if you're doing the right behavior, like trying to reduce overall waits within your environment, it can actually drive this metric up. So, it can cause maybe some false alarms there a little bit. I've actually created a custom signal waits, not a percent, just as a raw metric, which I've used in the past as well and I've found great success with. So it's really just an indicator overall, am I experiencing signal waits, period, end of story, right. I'm not doing any kind of comparison or ratio or anything like that. Okay.
Rob, have you shared that particular wait stat that you created in THWACK, by any chance?
Good question. I think I have, but actually, before we move on to that, because that's kind of a cheat there, but I did want to ask another poll question. So this is maybe a good time to throw that out there. Let me go to polls, bear with me one sec. So, I'm going to throw another question out there, and I would like people to chime in on this and let me know what you think about it. But basically, it's are you aware that we've got this community out there, this user community out there, called THWACK? Where there is content available for all kinds of stuff: custom alerts, custom reports, custom metrics.
So folks can vote and you want to give them another half a minute or so?
Yeah, yeah, and it looks like initially, most of you have, although I wouldn't say it's overwhelming. So, having said that, let's make sure that next time, everybody can answer yes. So, at least on this call, at least if you attended this webinar. So, I'm going to jump over here to THWACK, which I handily already have open, but if I go to Product Forums, drop down to Database Performance Analyzer, and there's a quick link here called Content. Now there's a lot of other material out here and content out here, but this content exchange was created specifically for DBAs to share information or ideas, or scripts, or whatever the case may be, with other people that are using DPA, so it's our own little community here. If I go to SQL Server Custom Alerts— maybe it's a metric. Let's see I think I defined it as a metric. Did I share it? There we go. Is it this one? No, that's not it. Bear with me a second. Let me look at documents.
You know what? I think you caught me. I don't think I've shared it out there. So that is something that I will for sure do after this call. Sometimes I do it because somebody has a question about maybe that specific resource, so the Signal Waits Percent. It's one that's a little bit controversial, meaning that some people find it helpful, some people don't want it as a percent. So, when I was working with support, and this goes back even to the Confio days prior to SolarWinds, before we were acquired, but yeah, we would actually help customers kind of make sure they get what they need, whether it be a percent or just the raw, stuff like that. Yeah, so I'll be sure to— I'm taking that as a note, Ed called me out on that one. I will definitely share that content after this call.
Yeah, that wasn't scripted.
Right, exactly. But, these are some good ones. And these are custom metrics, so we've also got some specific custom alerts. And if anybody is not aware of this site, I would strongly encourage you to go out there and check it out, especially if you're thinking of coming up with some kind of custom alert, you may not have to reinvent the wheel. So it may have been posted out here already. So, or you can use these kind of alerts, these custom alerts, to basically be a starting point. It's not to say that it's going to meet your need exactly, but it might be a good place to start and come out here and grab the SQL, grab the code, and tweak it for your own purposes.
All right, awesome. So, let's continue with our next category, which was Administrative. So, in the Administrative-type category, we have some basic alerting definitions for, kind of generic purposes, but also for specific database platforms. The more generic ones are Instance Up Down, if we detect that we can't get to it while we're monitoring it, we can notify you on that. We can also let you know about disc space, like Database Freespace or Tablespace Freespace, if you're on Oracle. We have some specific ones. We can parse through the alert log entries on the Oracle side, or the Airlog on SQL Server. Whoops, sorry, I'm going to use the scroll bar, kind of flipping around a little bit there. Yep, so we have lots of different ones that come pre-configured out of the box for your disposal.
The one thing to mention here: we do have one, if we do experience errors when we're trying to get the data collection with our monitoring to notify you about that one— that one can be a little bit sensitive, though, and let me describe that a little bit. So, we do our quick pull, which is a sample once every second, of your instance to collect those wait statistics. When we do that, every once in a while those quick pulls will fail. As long as we get, I would say like 80 to 85% of success rate during those quick pulls, I would call that statistically valid, so I wouldn't worry about that other, 10, 15%, whatever that number is. But, if you have this alert set up, we will tell you every time that a quick pull fails. [laughing] So, you may be getting some noise if you do set that one up, so that's just a little bit of warning. Okay, so Ed, back to you on maybe some ways that you've implemented some of these administrative alerts on your side.
Yeah, this is a very basic set of alerts, and I have a feeling that if DPA decides to put a set of alerts out of the box, some of these would be right there. Instance availability. Boy, that's a big one. You want to know if your servers are up or down. Job failures, are your SQL Agent jobs all running successfully? You know, if you don't have some sort of alert and if you aren't checking every server you have every day, you don't know. These are some basic really good ones for every DBA.
Yeah, and I think you're spot on. Like, these probably would be the ones that we would come pre-configured out of the box with. I think that's right.
And then what I'd like to point out that we use and get a lot of use out of is the SQL Server Error Log Alert, and that's just a really— it's really easy to create an alert that, yeah, pick one of those, all of them.
Maybe we'll just stick with the 2016 one.
All right, well anyway, if you put in anything that you're interested in here, it'll just search for all of them, and the one that I use is just the word fail. What I'll get with the word fail, is log-in failures, backup failures, and also fail overs. So that's really simple to set up, and effective.
Looks like I already—let's see, we'll do two. So, here's the results that we get back, so it looks like this one's okay. No messages with the word fail were found within the error log for this instance, but again, good to check. It's good to have eyes on it and you want notification if it does pop up.
Absolutely, all right, back to you for custom alerts.
All right, very nice. So, custom alerts are probably my favorite amongst all the alert categories, because it's extremely flexible. So, this is if you want to roll your own. A lot of our current customers and prospects and stuff, they say, "You know what, yes, DPA is a wonderful tool, it really helps us out to kind of watch our environment, to really do that deep dive performance analysis. But me as a DBA, I've created all these scripts, I've created all these custom code in SQL that I run against my environment to tell me very certain specific things." So this was created to let them integrate those. We definitely don't want to say, "Leave those behind, just use whatever DPA provides," we want to be very inclusive, very flexible.
So, I would say that the primary limitation of the custom alerts is going to be the format of the return data set. So just keep that in mind as you're configuring these. So we can handle the single numeric, we can handle— it's called a multiple numeric return, but it's really one alphanumeric field plus one numeric field. So we can also handle some kind of true false, or boolean kind of condition. We can call or execute a SPROC, a stored proc, within your monitored instance, so that can be very, very powerful. Now, the thing with that is that that SPROC has to exist on that monitored instance, so we won't go and create it for you, but we can execute it from DPA.
So, the other thing with these, and let me go ahead and click on maybe just a single numeric. We can go against either the monitored database, so we can run SQL against the monitored database, or we can actually pull data from the repository itself. Now, the advantage of going against the repository is, if it's something that we've already collected the data for, why put additional overhead or load on your monitored instance when we already have the data? Or, if you don't just need some kind of real-time status from a query, and you want to look back at some kind of historical trending or something like that, or do some kind of summation over a period of time, then you can pull that data from the repository, if we have collected it. So that's kind of nice. But it really depends on what you're trying to achieve with these specific alerts. Okay, and now I'm going to turn it back over to Ed to talk about maybe some things that he's done with custom alerting in his environment.
Thanks, yeah, if you don't mind, why don't we go over to the databases on the C drive alert that you've put in there? And in our environments, we want to separate the operating system from the data, and so we don't want actual database files on the C drive. And this is a way that we can monitor that across many many servers, really simply.
This is a select from SQL Server metadata, and one of the nice things you can do with custom code is you can put in exceptions. You can make it report on the things that you want and not on the things you don't want, so you'll notice in the SQL here, we're not worried if the report server is on the C drive, and you can also exclude particular servers where it's okay. Maybe there is only a C drive on a few of ours that were set up prior to our current system of dividing out the drives. So, @@SERVERNAME not in, and you can put in whatever servers you want. So, this is an example of a simple one, but very useful for us. Rob, would you mind going over to the Discspace Free alert? Excuse me, Disc Freespace alert. That's an example that shows the extreme flexibility of what you can put into an alert. Yeah, the box with the SQL, I mean, you're able to create a temp table, and then it's filled. And here we have another section where we can exclude certain servers, where we can use Office automation, go against Windows Performance Monitor, PerfMon, run a cursor. [laughing] There's not a lot of limitations here, there's a lot you can do.
Right, if you can imagine it, chances are you can probably roll it into an alert.
Yeah, DPA has made it extremely flexible. The sky's the limit if you're good with code, if you're good with using the metadata. So you can use metadata, you can use things from Windows, you can use the repository, which has tremendous information and pull it out the way you want. You can pick up code from the web. It doesn't have to be something that somebody else has defined as an alert. You can put a whole lot in here. I've been extremely happy to have this flexibility whereas you could create a SQL Agent job, or you could create a PowerShell script to go against multiple servers. This actually puts a lot of that infrastructure into your hands with very little work on your part. We've found it very useful. And that's all I have to say about it, Rob.
Excellent, no, fantastic, thank you. So real quick, I'm going to jump to the Q and A section, it looks like we had a couple of questions there while we were walking through the alerts. One was, can the alerts be integrated to a ticketing system like ServiceNow? So, currently DPA can send notifications in one of two different ways. One is an email, and so if you have something that can ingest emails and parse it and then take action according to that, like maybe open up a ticket or something like that, then yes. But we also support SNMP Traps. So, an alert can be sent out as an SNMP Trap, and we've got the MIB OID, which kind of helps you define, or see, what's being sent in that SNMP Trap, how we define that, what information is being sent over. So if you do have something that can receive SNMP Traps, we can certainly send it that way as well.
So currently those are the two ways that we can send out notifications. Another question: don't you need to correlate signal waits with signal wait percent, see if there's really anything worth worrying about? I would say probably yes, and it's probably worth looking at both, right? You want to see if signal waits as a percentage of your overall waits is increasing or exceeding some kind of threshold, but looking at the raw signal waits data itself can tell you if you have an absolute problem or if it's just something that is maybe a little bit of noise in the background. So if signal waits are tending to increase over time, that means something on your system is probably not completely healthy, so worth checking into. And it could be, you could be running into a condition of the frog in the boiling water, I don't know if you guys have heard this analogy, but basically if you slowly crank up the heat on the frog, it will not jump out of the pot. I know that's kind of morbid. [laughing] But if you throw a frog into boiling water, it'll jump out right away. So, the reason I say that is that your signal waits might be going up, but your overall waits might be increasing as well, so you could be having systemic problems with your system that are hidden by the fact that while overall waits are going up, so my percentage remains the same roughly, as far as the ratio goes. So I'd say watching both is good.
Another question here: does alert grouping allow you to put alerts you've already created for one instance into a group, and apply them to other instances? Absolutely. So, if you're going through the alert grouping tab there, even if you have alerts that are in instances and stuff, we won't overlap. We suggest you, to even that same instance or some kind of grouping of instances that you want to have that alert run against. Okay, so one other thing to mention here with our alerts. I think most of it is fairly self-explanatory within these, but one thing that may be a little bit cryptic is this notification policy. So we've got some choices here. One is notify when level not visited since normal, when level is not normal, when level changes and is not normal, [laughing] when level not visited since normal. When I first read those, I'm like, it's a little bit cryptic and, you know, what do each of those mean? So if you guys do a search out there on Google or something like that, you'll likely find a hit on THWACK that talks about it.
So, somebody had asked a question like, hey what is the notification policy out there, and what do they mean? So, this comes from our success center, so we have a Knowledge Base article on this that you can find as well, but this one I posted just to help them understand what each of those means and when that notification will fire. So I just wanted to point that out. If anybody can't find that, you can reach out to us and I can definitely point you in the right direction as well. Okay, so at this point, I would invite anybody to ask any other questions that you might have, or go ahead and put those in the Q& A. Okay, if no questions, then let's kind of wrap this up. So, definitely look for an invite to the next webinar in our series, we're going to be doing a deep dive into the Orion integration. We're going to run through the integration wizard, discuss application-to-database mapping, and that's more from a Server & Application Monitor, SAM, to DPA. Plus, we've added, in this most recent release, we've put some storage to database mapping, so that's taking data from SRM, Storage Resource Monitor, to DPA as well. During this webinar— sorry, this webinar will be available for viewing after the fact, so look for a follow-up email on that, and that's in case maybe some of your colleagues weren't able to make it today, or maybe they didn't even register for this, but after attending you think they might be interested in this specific topic. You can send that on and they can view it after the fact.
A huge thanks to Ed for joining me in co-presenting on DPA alerting. This was fantastic and Ed, I loved hearing about the real world ways that you've implemented to help achieve some of that proactiveness within your environment, and your company. By the way, if anybody else is interested in joining me on one of these, [laughing] it's fun. I don't know, I think it's fun. Definitely reach out to me and let me know if you're interested and also which topic or feature or functionality that you think you'd like to cover. All right, with that, I would like to— oh, wait a minute, I think we have one more question, let's see. Let's see, you can, give me one sec while I read through this. Oh okay, it looks like it was primarily just a comment.
So, all right, with that, I would like to sincerely thank all of you for stepping away with us today for a few minutes, and hopefully we will "v-see" you, virtually see you, next time. You know, somebody really needs to coin a term, or come up some kind of terminology that's catchy that kind of encapsulates that, right? It's like, I'm not going to see you next time, but we will get together. Anyway, thank you all for attending, and I hope you have a wonderful rest of the day.