In our typical VSA implementation, we create four distinct access levels, and several sub-types for MSP employees, plus three roles for customer access. No user has Master role rights in our deployment configuration. These roles should have a “NOC” or “MSP” prefix to designate them as internal roles.

Level 0 – Support A role designed for support staff to access VSA for running reports, getting agent counts, or checking use and available licensing. No access to automation is available, but these users can view agents and have virtually unlimited access to the reporting functions.

 Level 1 – Technician This role, which we name “NOC-1-Tech”, grants the ability to perform basic agent administration, view audit and other configuration settings, and access remote control features. This provides the ability to perform about 80% of what a technician would do on a daily basis for end-user support.

Level 2 – Administrator Named “NOC-2-Admin” in our system, it grants additional capabilities to run procedures, deploy AV and AM, and perform most agent configurations. Neither of the above roles permit changing the configuration of VSA-wide settings.

Level 5 – Specialist These roles grant VSA administration rights to specific features, distributing the administration tasks among multiple users. In our practice, we use the following specialist types:

• Security – provides the ability to perform all Auth Anvil configuration and management tasks.
• AV-Malware – grants access to administer the Antivirus and Malware components, including definition of profiles, policies, and assigning them to customers.
• Updating – allows administration of all Patch Management and Software Management components. It may also allow access to other application updating components.
• Backup – Allows configuration of all VSA settings related to backup operations.
• Manager – grants a combination of roles, usually assigned to the Dispatch, helpdesk or Technical Manager(s).

Implementing these security roles will allow for better security and organization within your VSA infrastructure. To learn more, schedule a demo for MSP Builder’s RMM Suite!

There were some surprises, like when Mike Puglia shared a photo from an early career path. At least, that's what I was told - I had stepped out of the room at that moment. At least I can see where he gets his determination and focus from!

Of course, there were regular gatherings at the dispensers of the magical Elixir of Life, especially during the morning breaks. Many of us fought for space at tables close to these oasis'. I even tipped one of the urns to allow a fellow Kaseyan to enjoy one more cup. Squeezing the urn did not help, however. These sessions were a great place to meet and share experiences and ideas.

A personal highlight of this event was my presentation, which - thanks to confusion surrounding both the title and the description - became known as "Automate or Die!". Thanks to Lauren, a last-minute post provided clarity about the topic and content, and we had far more people attend than the handful I expected.

The session covered many of the best practices related to using the VSA that allow you to use automation effectively.

One topic that we covered was how to use Custom Fields effectively. These can be used for reporting as well as to control the automation in your VSA. Using these effectively, with proper planning will pay huge automation dividends.

Some of the best moments of the event came during the awards ceremony. The MSP practice (Baroan Technologies) that spawned MSP Builder was nominated for the MSP Efficiency Award. Shortly thereafter, we were awarded the Community Award - which is awarded to "companies that have demonstrated thought leadership within the IT Services industry by sharing best practices with peers to shape a better technical community". Here, Dimitri Miaoulis looks on as I accept the award. We're all honored to be recognized, and are happy to continue to share our knowledge and experience with the MSP and IT communities.

So – how many endpoints do you have? Around 600, including roughly 80 servers. That’s across 26 clients, and four of them are break-fix – we use the agents just for remote support.

What do you charge for basic support package – monitoring, patching, AV, AM, and maintenance?
 Charge? We only charge for the actual time we spend fixing stuff. We do charge $5 for the AV license.

Other answers range from “$25 for the basic package of patching and AV/AM" to "$150 for everything, including call-in support".

Well, with the initial cost of the VSA agent and the monthly costs for agent and AV/AM licensing, if you monitor and respond to alerts, you need to generate $250-400 per year on each agent just to break even. That means you need to charge at least $28 for just the basic monitoring and AV/AM, or find a way to reduce costs. (gulp!) I didn’t think of it that way.

How many monitoring alerts do you get each day? Too many (laughs). We used to get over 200 each day, so we turned off the monitors.

How many techs do you have on the help desk? Four – two L-1, an L-2, and an L-3.

How’s that working out – is their time fully utilized? It is – we could add another tech if we could afford it! The L-3 engineer is pretty busy with escalations and some manual maintenance tasks.

What’s your rough payroll for your four techs? Around $300,000, plus benefits and such.

What about customer projects – how do you handle those? We have one dedicated engineer for projects, and press the other techs into service  as time permits, but then the tickets pile up.

Do you do any ongoing system maintenance? We tried, but the procedures are difficult, don’t always work, and we’re too busy to spend more time on it.

Don’t you think that regular system maintenance might reduce the calls and alerts? It might, but that’s complicated to set up, and we just don’t have that expertise.

Well, I think that if you took a look at the results that our tools provide, you might change your mind. Here’s an example from an MSP that we first developed these tools for.

When we started working with them, they had around 1200 managed agents (only 16 were break-fix). They had two L-2 and three L-3 engineers on the helpdesk, with base tech salaries hitting just over $500,000. Another two engineers working on projects pushed tech salaries to $720,000. Those endpoints generated over 200 alerts per day – that’s one in six machines generating alerts through monitoring. There was one engineer per 240 endpoints, so each endpoint had an “employee cost” of $421.

During the first year, we developed standards that helped streamline their operations, we removed the sample monitors and deployed ones that were engineered to their customer requirements, and added some basic automation to VSA to deploy monitors based on detected services and schedule patching and updating. This is what we now sell as our Core Automation Suite.

After the $1500 investment in this solution, they were able to increase their managed endpoints to just over 1800. They eliminated the L-2 helpdesk positions and promoted “one call and done” help-desk services to the clients using only L-3 engineers. Alerts dropped to 75 per day – one in twenty-four. With one engineer per 600 endpoints, the “employee cost” of an endpoint was now down to $192 – a savings per endpoint of $229.

The following year, we helped them implement the EMM Suite, which includes Smart Monitors and Daily Maintenance. The Smart Monitors reduced alerts by auto-remediating many common conditions and dynamically setting reasonable alert thresholds. Daily Maintenance improved the operation of the endpoints and further reduced the help-desk load. By now, they had grown to almost 3000 managed endpoints, yet alerts dropped to under 16 per day (one in every 188 agents)! Customer calls had also dropped since they were proactively maintaining the systems. One help-desk engineer now supported almost 1000 endpoints, dropping the “employee cost” of an endpoint to just $110 per year. After a year using the EMM suite, they reported that the help desk team spent nearly 50% less time on tickets because the basic remediation tasks were completed through automation.

Thus – a $1500 investment in automation allowed the elimination of two L-2 help desk seats, freeing those employees for project work and still supported a 50% increase in managed endpoints. Using our automation for patching and daily maintenance, end-user calls dropped, and customer satisfaction increased. The $0.50 cost for these improvements was easily absorbed by the increased profitability, and our end-user interface that reported what was being done eliminated the “I pay you all this money – what do you do for me?” question that so many customers ask! We tell them every day what we’ve done.

Here’s some facts based on clients that we’ve helped.

Typical “small” MSP has around 750 managed endpoints, and a few hundred “break-fix” systems. They have 3 techs on help desk at L-1 to L-3, plus another 2-3 on staff for project work.

Without automation:

• 3 Help Desk techs base salaries (major city region) cost $250,000
• About 125 alerts per day are generated – 42 per tech, and hard to review and close all of them.
• Each managed endpoint requires $334 per year to break even on salaries alone.

With automation and maintenance:

• 2 Help Desk techs (L-2 and L-3) base salaries are $195,000. $55,000 per year savings.
• $4,500 per year in EMM licensing, still a $50,000+ annual savings (not counting that the 3rd tech can now be billing for project work with no additional salary cost!)
• Fewer than 5 alerts per day – 2-3 per tech, easily handled, allows direct support of client calls.
• Each managed endpoint requires $266 per year to break even on salaries and the $6 EMM license costs. (Yes, EMM costs just $6 per agent per year.)
• At the industry average of 500 agents per help-desk agent, the break-even cost is around $150, and we’ve seen 1100-1200 endpoints per agent possible once the environment has been patched and maintained for a few months.

The question is – what’s more expensive? $3000 plus $6 per agent per year or the salary of another tech? Plus benefits, payroll taxes, insurance… Other considerations include:

• What about your employee satisfaction vs. the frustration of fighting a losing battle against alerts?
• Then there’s client satisfaction – their network gets “quiet”, the “fires” stop, and their employees become more productive.
• Can you do this yourself? Sure, but at what cost? Can you dedicate an engineer to build the monitors and the tools and then maintain it? Or will they get pulled into customer support?
• Then there’s the extra billing capacity for new work by employees that aren’t tied to the help desk!

NOTE: Salary costs were based on averages in major metropolitan areas like NYC, Boston, and LA. Level 1 tech salary is $55,000, Level 2 is $80,000, and Level 3 is $110,000. EMM costs are based on an average distribution of 12% servers and 88% workstations.

We regularly run comparisons of current to prior quarter ticket levels, and sometimes run the comparison of current quarter against the last quarter of our old Kaseya platform, before we implemented the RMM Suite. By developing smart monitors to handle specific alerts, we have seen tremendous reduction in the number of tickets. To be clear, the reduction in tickets is noted at the help desk/PSA. We still get alerts, but the auto-remediation built into the Service Desk and Smart Monitors reduces the number of alerts that result in actual tickets seen by the help desk.

Looking at just the Antivirus Definitions Outdated alerts, the smart monitor initiates a definition update instead of immediately triggering an alert. This has reduced that alert from an average of 30 per day to less than 10 per week. This is an outstanding 95.5% reduction in this alert, which was one of the most time-wasting alerts we ever responded to. Alert – assign – connect – run AV definition update – update and close ticket – REPEAT! Our Antivirus Smart Monitor knows how to force an update for nine different AV products, and more can be added in just minutes.

Another time-consuming alert was the Low Disk Space monitor. Replacing the standard Kaseya monitors with smart monitors provided several advantages:

1. The low space warning and alert threshold values are calculated automatically based on the disk volume size. This eliminates false alerts from a “one size fits all” monitor that alerts on 10% free. It also eliminates the need to create and deploy many custom monitors with unique thresholds to avoid the “one size” type of monitor. The calculated value is also adjusted across 10 distinct volume sizes, so thresholds for small drives are higher than for huge drives.
2. Alerts are suppressed until the alert condition persists for a specific period of time – 3 days by default. This prevents firing the alert due to transient conditions like deploying applications or restoring a folder from backup to a temporary location.
3. Crossing a space threshold can trigger a remediation process, invoking a cleanup task and requesting that it perform a more aggressive cleanup.
4. Trend analysis monitors usage and predicts when the threshold might be crossed, providing a 30-day period in which to review utilization and possibly deploy additional storage.

While these two monitors contributed significantly to a reduction in alerts, overall, we have seen an average of 62% fewer alerts since moving from a Kaseya “default” configuration with minimal optimization to the highly optimized alert pack in the RMM Suite. We have seen a steady decline in the number of alerts by using the RMM Suite over the past year as well. As new remediation opportunities are identified and implemented, fewer alerts will make it to the help desk. Given the considerable research that we did and having a staff member dedicated to developing appropriate monitor sets, smart monitors, and the automation in our Service Desk, we expected this.

What really surprised us, however, was the reduction in man-hours allocated to responding to the alert tickets. We spent over 50% less time on alert tickets in the past 12 months compared the prior 12 months. This alone could be significant, but during that time we also added over 600 managed endpoints. All but sixteen of our current 2500 agents are fully managed, have AV and AM, perform application updates and patching, and run maintenance and audit tasks daily. The reduction in time needed to respond to alert tickets has translated directly into the ability to work on more projects and the billable time that it creates. It also allows us to manage the help desk – handling both alert tickets and real-time calls – with just 3 engineers.

The use of automation and quality alerting have had direct and substantial benefits to our bottom line.

• Elimination of false alerts reduces time to manually review;
• Use of Smart Monitors reduces alerts by eliminating transient events and use of inappropriate thresholds;
• Automation remediates common problems, eliminating the need for help-desk staff to get involved;
• Flat-rate monitoring revenue increases by reducing or eliminating manual actions;
• Project revenue can increase when less time is needed to respond to alerts.

As we developed new tools for our MSP business, we began to offer those under the MSP Builder product line. These are the Tool Chest products that allow an MSP to develop their own advanced tools for Kaseya Service Desk as well as several PSA-Agnostic tools that provide consistency of administration and methods for common administrative tasks. These tools are sold for a one-time license fee and allow unlimited use at any client managed through a PSA, including Kaseya and LabTech. Alternate versions are available for organizations that are unmanaged by an MSP, ranging in cost from free for small businesses to under $1000 for an unrestricted enterprise environment.

Our flagship product is an RMM Suite for Kaseya version 9 and above. It's a turnkey implementation that includes everything an MSP needs to proficiently use a Kaseya VSA platform:

• A polished and focused set of monitors that eliminate the "noise" and allow engineers to target the real issues.
• "Smart Monitors" that can self-remeidate certain conditions, auto-adjust to host-specific configurations, and alert only for multiple consecutive detections. These simplify deployment and further reduce false alerts.
• Baseline configuration sets for AV and AM profiles and patching profiles, which allow an MSP to deploy customer-specific profiles that are based on standards, reducing effort to deploy while maintaining consistency.
• Pre-configured System Policies that automate the administration and deployment of monitors, patch schedules, AV/AM profiles, application updates, and even auto-remediation of common conditions like outdated agents.
• A fully automated Ticket Processor, with automatic deployment of remediation procedures, escalation of alerts, notification by email, pager, or phone for high-priority events when the help desk is closed.
• An autonomous daily maintenance tool suite that performs day and evening tasks to maintain the health and quality of operation of both workstations and servers in a client environment. Tasks can easily be customized by customer, site, or even a specific machine.

Best of all, the entire package is affordably priced, and installation & training are available and take less than 3 days.