×

350
351

TechnologyI'm a IT Engineer and Manager at a Managed Service Provider (MSP) doing outsourced IT throughout the pandemic AMA! (self.IAmA)

submitted by DarkAlman

My short bio:

IT Engineer with over 15 years experience in industry, specializing in Networks, Firewalls, Windows Servers, Storage, and Server Performance. 8 years working at MSPs.

Have been the manager of an MSP throughout much of the pandemic, doing outsourced IT, Sales, and helping everyone work from home.

My Proof:

Proof removed at AMA completion

Thanks everyone!

all 182 comments
sorted by:
best
topnewcontroversialoldq&a

[–]SirTobyGirl 48 points49 points  (58 children)

I'm 30 and still doing entry level IT roles, never went to university.

I started in IT when I was 19 and left 3 years later to travel and such and entered back into IT 2 years ago.

What field of IT should I specialise in for the moneys and job security etc, Netowroking? coding ? Security ?

[–]Abrham_Smith 46 points47 points  (8 children)

Cloud Technology

AWS Certs and 100k+ in 2 years.

[–]_Killua_Zoldyck_ 6 points7 points  (3 children)

Where would you start?

[–]Veldimare 15 points16 points  (0 children)

I just passed my AZ-104 with Udemy and free azure account in a little over 30 days. I had a little bit of experience with it before hand. Whizlabs for practice tests. Got everything on sale so not including the actual exam fee, about 80 bucks.

[–]Hellow0rld 9 points10 points  (1 child)

For AWS certs, if you have some IT / aws experience start with Sys ops or solutions architect. After that you can look into Solutions Architect pro or one of the specialty tests like networking or security. If your starting with no AWS / cloud experience check out the cloud practitioner test to get a base understanding of the services.

I used some courses on udemy for sys ops/ solutions architect. Tutorial Dojo helped me with the aws security cert. I’d also recommend making an AWS account and play with some of the free services.

[–]SirJumbles 0 points1 point  (0 children)

Saving this for later.

[–]CambriaKilgannonn 2 points3 points  (0 children)

I'm doing an IT degree at a community college and just went through a class revolving around AWS, it seems pretty easy to use compared to Azure. Is it pretty easy to get into once you have the certs?

[–]AMasonJar 2 points3 points  (1 child)

Can you elaborate some more? For context, I'm still in school, recently switched from software engineering to comp sci with a cybersecurity focus, but I'm still not really sure what direction to take things other than that I don't love programming enough to want to make a career out of specifically that. My school doesn't have a whole lot to cover regarding the cloud though, likely because of how it's such a recent field.

Where can I check out more info about cloud technology and how to dive into that?

[–]superkewldood -1 points0 points  (0 children)

Definitely this

[–]sandiegoking 10 points11 points  (5 children)

Security, we can't find qualified people.

[–]LowestKey 8 points9 points  (2 children)

And when we do we ignore their resumes.

[–]sandiegoking 4 points5 points  (1 child)

I can tell you..resumes lie. I've seen amazing resumes and when we ask simple security questions they can't answer them.

[–]LowestKey 0 points1 point  (0 children)

Sure, but you won't know until you call them back. No way of knowing if you never contact them.

[–]DarkAlman[S] 5 points6 points  (0 children)

Or dismiss their resumes because they fail the drug test

[–]8andahalfby11 0 points1 point  (0 children)

I keep hearing this, but then I keep hearing that formal degrees, certs, and even experience aren't enough to be considered 'qualified' from other redditors. What do you people want?

[–]Kierkin 27 points28 points  (8 children)

Data science. I’ve been in the IT industry for 25 years. I’ve done it all. Started help desk and am now a cloud services architect. Infrastructure is being abstracted by SDDC even more now than before. Big money is in data work and Machine Learning. That’s the way I’m moving my career and it’s working.

[–]superkewldood 7 points8 points  (7 children)

Sounds like your a cloud architect not a data scientist (which is better due to more money and more jobs)

[–]Kierkin 5 points6 points  (6 children)

Yes but I’m moving into data science. Machine learning has always been a hobby and I’m focusing on unsupervised models.

[–]Flamekeks 5 points6 points  (5 children)

how would you rate AWS/Azure architect/developer in terms of salary, job prospects etc?

[–]AcceptableAnswer7 0 points1 point  (4 children)

Plentiful and highly paid. Go look at Glassdoor for specific roles/locations if you want salary specifics. But there are always job openings for that sort of work.

[–]Kierkin 0 points1 point  (3 children)

Agree. I’m approaching 200k/yr.

[–]KingOfCruel 0 points1 point  (1 child)

Experience and education/location?

[–]Kierkin 2 points3 points  (0 children)

GED, learning by doing, AZ900, AZ303. Mid Atlantic.

[–]Hannicho 11 points12 points  (1 child)

Cybersecurity. Sounds sexy really tedious.

[–]DarkAlman[S] 9 points10 points  (0 children)

"60 hours worth of meetings next week to make sure we pass our PCI"

[–]CptSaySin 6 points7 points  (2 children)

Cloud computing: Azure, AWS, GCP. Choose one and dedicate yourself to it as a specialist, or choose all and go as a generalist.

Entry level Cloud Engineer/Architect jobs are 80k+. A few years experience and you'll be over 100k.

[–][deleted]  (1 child)

[deleted]

    [–]gatman12 -1 points0 points  (0 children)

    There's a save button. And you don't have to tell people when you use it.

    [–]DarkAlman[S] 31 points32 points  (25 children)

    If you're in it for the money and job security I suggest you learn Mainframes... like the old IBM type.

    I'm not joking, mainframe guys are super in demand and it's garaunteed job security. You'll be the mainframe guy at a major insurance company or whatever because you'll be the only one that knows have any of it works.

    Otherwise IT security is lucrative, but very very stressful.

    [–]pikachu8090 8 points9 points  (0 children)

    i mean one of the companies i've worked for they've been replacing their mainframe with cloud services so that they doin't need to pay people/ company to maintain/host their mainframe

    [–]CptSaySin 57 points58 points  (9 children)

    mainframe guys are super in demand and it's garaunteed job security

    No. No, no, no. This is like telling someone in the 2000s to become a specialist in Novell.

    Mainframes are dying, that's why currently, jobs might be in demand. Because everyone has moved on. There's no future in mainframes or any vertical scaling.

    [–]PazDak 9 points10 points  (2 children)

    Being have been saying learning cobol is a dead end for 30 years. But it probably is still the easiest to make bank off.. especially if you can get a DoD clearance.

    [–]PhDamnit 4 points5 points  (1 child)

    I mean, aren't 90% of cobol experts like... 65+? Even without government work, I'm sure there are enough old private banks needing to transition out that someone ambitious enough could make a niche career doing consulting work.

    [–]ripvannwinkler 1 point2 points  (0 children)

    IBM mainframes aren't going anywhere any time soon. The technology is cheap, readily available, and there's no value to upgrade in many cases. You can slap a web service in front of a mainframe app and be "modern" with very little investment.

    [–]beemoe 2 points3 points  (0 children)

    Because everyone has moved on.

    So you must be new to the industry. That's not even remotely the case.

    The world still runs on mainframe, deep in the bowels of every sector of the economy. Tons of lovely abstractions sprinkled on top, but a lot of places still use it. There is a massive shortage of people who know how to program on it because businesses are terrible at planning for the future. Mainframes are nearly bulletproof, and most importantly are probably paid for.

    They are now panicking and paying top dollar to get people to either maintain or take part in migrations.

    Tiny companies like Lowes for example, currently need COBOL, TCL, JCL. Nearly every major bank in existence, the airlines and...well anything outside boom/bust trend/buzzword software.

    Certainly don't put all your eggs in that basket, but if you can pick up the skill, and be a resource for companies to migrate (knowing old and new) you can pretty much write your own paychecks. Completely disagree with it dying. Maybe it should be, but it is most certainly not.

    The Uni where I got my CS from spun up a specific curriculum, just to service the need.

    [–]more_beans_mrtaggart 2 points3 points  (0 children)

    I’ve got a customer who is 70 and trying to retire, but he’s getting calls from banks, supermarkets, etc and having to drive into London to work on mainframe.

    So he put his hourly rate up to £250/hr. They still called.

    In the end he just said no. It ended up with a bank executive stood in his doorway pleading with him to go and fix a problem. There was literally no-one else they could call.

    [–]ronin_cse 0 points1 point  (0 children)

    I was just going to say this, you might be in demand right this minute but that probably won’t last more than a few years

    [–]IntellegentIdiot 0 points1 point  (0 children)

    I assume those companies eventually will migrate to newer technology once the pool of employees gets to a point where it's uneconomical or impossible to proceed

    [–]LeopardBernstein 0 points1 point  (0 children)

    They said the same thing 30 years ago. Guess what they'll probably be saying in another 30 years.

    The older you get the more you realize, upgrading a system frequently isn't worth the cost if it's maintainable.

    [–]2gig 6 points7 points  (1 child)

    Any particular certs/programs you'd recommend for mainframes?

    [–]mlambie 15 points16 points  (0 children)

    IBM have an upcoming global conference on mainframes. Z Day in a few weeks. Full student track. I’m gonna check it out.

    [–]OneAndOnlyJackSchitt 3 points4 points  (2 children)

    Learn mainframes but also learn cloud services (Amazon Web Services, Google Cloud Platform, Microsoft Azure)

    The big thing is migrating stuff that used to exist only on mainframes to the cloud. So you'd need to know how to do that. [Edit: People who work for companies which do this] Companies which do this type of migration make fucking bank. Mid-six figures.

    [–]mata_dan 0 points1 point  (1 child)

    Oi, you missed IBM Cloud :)

    [–]Kierkin 4 points5 points  (0 children)

    No one misses the IBM cloud.

    [–]TheLittlestLegend 2 points3 points  (0 children)

    I got my first tech job out of college as a mainframe application developer for IBM. It can be frustrating to work from a mainframe terminal when there are nice tools like VSC for nearly every other programming position. Other than that I’ve really enjoyed it.

    [–]thor561 1 point2 points  (0 children)

    Ehh… it’s in demand NOW because all the old programmers are retiring and large businesses haven’t migrated off them yet. So for the next 5-10 years there will be a demand, but I feel like that will wane significantly. My company currently has both MVS and VSE mainframes still (don’t ask) and just had our main systems engineer retire. I don’t see them replacing him with anything besides a managed service provider to do break/fix and upgrades for the short term until we transition off. I wouldn’t recommend anyone under the age of 40 suddenly get into COBOL programming, I don’t think it will be there in enough demand for the long term.

    [–]superkewldood 2 points3 points  (0 children)

    Until the job gets outsourced to Asia.

    [–]heard_enough_crap 0 points1 point  (0 children)

    everyone thinks I'm crazy at work. I'm in charge of moving our company in into the cloud, but I still have a DEC20 in my home office.

    [–]subscribemenot 0 points1 point  (0 children)

    Umm sure but it’s not a long term career. Your dealing with decades old tech and protocols

    [–]mata_dan 0 points1 point  (0 children)

    I'm not so sure about that, 5 years ago I was teaching someone JS so they could get out of that position at one of the worlds largest finance companies because they needed job security moving on.

    [–]KingOfCruel 0 points1 point  (0 children)

    What's the least stressful cloud security/aws/ azure job you can recommend to an Ism major? I run an online business on the side.

    [–]Kinser9 0 points1 point  (0 children)

    New Jersey was scrambling for COBAL programmers during the beginning of the pandemic. The system for filing for unemployment was antiquated and crashing all over the place.

    [–]mattl33 0 points1 point  (0 children)

    I guess depending on your professional network and your proximity to major bank and federal govt locations this might be sound advice but otherwise I can't imagine this being fulfilling work. The govt agencies I worked for moved away from mainframes years ago, it's clearly dead end work but if the only criteria is good pay for the mid term then sure why not.

    [–]count023 1 point2 points  (0 children)

    Network security. Get a cisco certificate, enter into a SOC team, you'll be able to move laterally to cloud security, network engineering or devsecops within 18 months.

    [–]Irishiron28 0 points1 point  (0 children)

    35 with a masters. 15 years in cyber security generally can get in with a few certs. Great salary and mostly remote work. 100k easy.

    [–]mattl33 0 points1 point  (0 children)

    Learn python, and then learn how to use that to automate things like checking parts of router/server configs to find things needing an update etc. I'm a network engineer for big tech companies and automation is basically the core of my job these days. Knowing how the network works is almost secondary.

    Also recommend https://nrelabs.io/ for the practicing the above topics.

    [–]IveBeenJaped 77 points78 points  (16 children)

    How do you feel about msp’s exploiting their salaried workers putting them on call and not paying them appropriately?

    [–]DarkAlman[S] 45 points46 points  (2 children)

    It's a nasty practice that I've previously been on the receiving end of. It's not just MSPs either, there's a lot of industries that do that.

    At a previous employer I was disciplined for not checking my emails at 3am, because apparently I wasn't allowed to sleep while oncall?

    When they threatened to fire me for refusing to comply I quoted them the labor law and showed them how much overtime they legally owed me and they changed their tune pretty fast.

    You shouldn't be offering 24/7 service if you aren't willing to have 24/7 shifts.

    I've tried real hard to avoid these practices while being a manager. You can't avoid some after hours work in IT, but you can automate away a lot of things, be flexible with schedules, pay OT, and set it up so your staff have infrequent oncall rotations instead of 'all the time' or every couple of weeks.

    [–]Prayer_Worrior 1 point2 points  (1 child)

    If I was on an on-call rotation at a previous employer could I sue them for back pay? I was salary and didn't know about overtime for on-call. State was Colorado.

    [–]DarkAlman[S] 2 points3 points  (0 children)

    \r\legaladvice

    [–]bp24416 11 points12 points  (5 children)

    Oh so I'm not the only one that happens to? My co-workers and I complained so much but it did nothing. We just started fucking up while on-call purposely so that they would see the problem. They eventually decided they would allow us to claim anything over 4 hours while on-call as overtime. And they would pay us a flat $20 per hour for it with no regard to salary (and $20 per hour is less than all of our regular rates). Every time I try to submit the OT hours (which they require we fill out a special form for) the main boss sends an email passive aggressively saying they will check to make sure I actually worked that time.

    Edit: I forgot to mention when a person is on-call they require us to do it for a week solid, while working the normal 40 hours Monday through Friday. 24/7 for 7 days. They do not consider any time you aren't actually working directly with a client as actual work. So you can only claim billed hours. Also there is only one person on call at a time. So basically it feels like being tortured by not being allowed to sleep for a week. And getting paid pennies for it.

    [–]Saffa_NZ 3 points4 points  (2 children)

    I have an identical on call rotation but we get a flat 600 pre tax for a week of on call.

    [–]bp24416 2 points3 points  (1 child)

    Management where I am at considered doing something similar, decided it was too much money and went with what we have now. They regularly dispute hours to keep the number of claimed hours low. From speaking with co-workers most people aren't even bothering to fill out the form to get paid because they have made it so difficult.

    [–]Pacifistering 0 points1 point  (0 children)

    Get the fuck out of there. Do not award them for their dirty practices.

    [–]ripvannwinkler 1 point2 points  (1 child)

    They do not consider any time you aren't actually working directly with a client as actual work. So you can only claim billed hours. Also there is only one person on call at a time. So basically it feels like being tortured by not being allowed to sleep for a week. And getting paid pennies for it.

    Just say no, fren.

    [–]LetMeGuessYourAlts 1 point2 points  (0 children)

    If you say no, you're being "insubordinate". They don't care that what they're doing is abusive and probably illegal. The "easier" method is to just say you slept through the calls because you're a human being who has already been kept up all week. Better yet: Get a doctor's note saying it's affecting your heart or health being woken up so much.

    [–]Kierkin 24 points25 points  (1 child)

    This right here. I’ll never work for an MSP again.

    [–]berrattack 17 points18 points  (0 children)

    Yep on call weekends and nights. Putting your life on hold while only getting paid peanuts if you are called, but expected to pick up the phone, be sober and in town. Able to drop everything in a moment’s notice.

    [–]skrshawk 16 points17 points  (0 children)

    This exploitation is very illegal. There is a provision for those whose primary job duties involve the design and architecture of computer systems and networks to be salaried exempt, but that is not the vast majority of people at MSPs. Operation and maintenance roles are specifically not included and these people must be paid properly for being on-call and overtime when expected.

    My state says that all hours in which a person is on-call and expected to be available to work (cannot use the time productively for their own purposes) must be paid at least at minimum wage. As they aren't actual "hours worked", they aren't subject to overtime.

    Workplaces ignore this all the time in New York because they'd be shelling out a lot of labor expenses. But MSPs do this all the fucking time, usually with 24/7 on-call requirements for a week or two at a time. With that being the case, that should be your straight 40, plus another 128 of minimum wage, except for those hours actually called in which are paid at 1.5x normal rate.

    I'm actually hoping to see an Attorney General action crack down on this because a lot of IT people are getting royally boned here.

    [–]megasxl264 2 points3 points  (1 child)

    MSP employee here: This completely depends on the company and where you live, and it's more of an issue of the tech industry in general.

    We're required to track all of our hours and tasks completed each day for each client. We also get overtime pay and our clients aren't supposed to contact any of us directly.

    Typically, non-contracted clients don't bother us during off-hours because they get billed each time we work with them. They also make up the blunt of clients as they're usually individuals or small businesses that simply don't have the need or capital for dedicated IT. For our larger clients, we're basically just a typical IT team, so it's regular business hours, and the rule of no commits or updates on Friday stands. Again, when it comes to after-hours that usually involves one of our bosses bringing it to our attention and paying us for the time worked.

    [–]GoMoriartyOnPlanets 0 points1 point  (0 children)

    Worked for an msp like that. It was remote, but it was oncall at random times. It was kinda hellish. They fired me after a year. I begged them for an extra week of severence because I needed the extra money.

    Actually I didn't need the extra money. I never gave my notice to the previous job. I was earning 2 full time paychecks the whole time. Paid off all my debt with that extra money and more. Will never regret that sweet sweet time. Every time the bosses or clients clients yelled at me for not working extra, I laughed inside. Suck it Doug.

    [–]mata_dan 0 points1 point  (0 children)

    Not OP, but hilarious because that's how they can't retail actual skills and leave themselves open to err, all sorts of issues.

    [–]Exact_Sport 18 points19 points  (3 children)

    When companies decide to outsource infrastructure services, what’s their common complaint with MSPs. That is, what do they wish their MSPs should do better? In your experience, What should MSPs focus on to keep a client “happy”?

    [–]DarkAlman[S] 16 points17 points  (2 children)

    TLDR: If you have good people, you pay them well, and they don't get burned out then they'll pay more attention, care about the customers, and won't be as prone to mistakes.

    The most consistent complaints I hear from existing MSP customers wanting to switchover to us is not being able to get hold of techs on demand, delays in getting support, pushy sales people + constant upselling, regular maintenance not getting done, and straight up incompetence.

    For the first two staff are often used to having a dedicated IT person on staff that can show up and teach you to use excel at the drop of a hat. When you're with an MSP you have to deal with the fact that we have a lot of customers and we have to prioritize who gets service when.

    We have a servicedesk to provide quick service, but staff often hate calling for support so it's give and take. Having high quality people and good training for the service desk helps, but you also need to have a good escalation path so that people don't waste time on the phone with issues the tech on the line can't fix. Being able to do regular onsite visits helps with the touchy-feely stuff.

    We also offer fulltime onsite techs if a company wants, but they have to pay extra for that.

    For the pushy sales people, that's a cultural problem within the MSP. We have sales people dedicated to MSP accounts that aren't on the same commission structure as our enterprise sales guys. That helps a lot. The MSP Sales guys get commission based on new contracts and retention so they are more interested in keeping customers happy than upselling them on something they don't need to make their sales targets.

    As for incompetence and not getting stuff done... have good people, good managers, and pay your techs what they are worth. Give your techs a chance to learn, listen to their ideas, and innovate. The biggest complaints I get from MSP techs I talk too is dealing with burn out, refusal to pay industry average wages, and not enough time for hands on training. Too many MSPs are only interested in the bottom line rather than developing people, so they become meat grinder shops that go through a revolving door of techs. I've worked very hard not to be like that with my teams...

    [–]Wasabicannon 6 points7 points  (1 child)

    For the first two staff are often used to having a dedicated IT person on staff that can show up and teach you to use excel at the drop of a hat. When you're with an MSP you have to deal with the fact that we have a lot of customers and we have to prioritize who gets service when.

    Work in a MSP and this is the big one. Whenever we onboard a new client it is always the same where they had internal IT and were used to just seeing the IT dude walking around and asking him a question.

    On the flip side you also have those that came from another MSP that was ran awful. First call with the department manager and you get a list of everything that was ignored by the previous MSP and you get it all resolved and then fix some other shit you noticed while fixing it and you have that department manager loving you for life.

    The biggest complaints I get from MSP techs I talk too is dealing with burn out, refusal to pay industry average wages, and not enough time for hands on training.

    This has been my biggest complaint. MSPs tend to focus on if you can't fix it asap just escalate and move on to the next call/email instead of do some research and figure it out or reach out to a tier 2 to get some ideas and learn how to fix it in the future.

    [–]DarkAlman[S] 2 points3 points  (0 children)

    yup, too much focus on grinding away on tickets and not nearly enough on training and developing your techs.

    "Write your certs if you want to get promoted"

    Yeah and btw we have no training budget and study on your own time.

    I'm glad my current employer has a very different approach.

    [–]infinitevalence 10 points11 points  (4 children)

    Are you going to DattoCon?

    [–]DarkAlman[S] 11 points12 points  (2 children)

    Nope, Veeam all the way baby

    [–]yaroto98 3 points4 points  (1 child)

    Can I ask why Veeam over Datto?

    [–]SFHalfling 0 points1 point  (0 children)

    Not OP but in our case it's because Veeam works, and in the cases where it doesn't the support is good. Average response for a P1 ticket for us is within 30 mins, even the P3/4 I've raised are answered with 2-3 hours.

    [–]tellemurius 1 point2 points  (0 children)

    More of a Rubrik guy myself.

    [–]TehCuddler 9 points10 points  (2 children)

    I'm a small fry Network Engineer and really need to up my experience. What kind of home lab setup would you suggest to increase knowledge?

    [–]DarkAlman[S] 10 points11 points  (1 child)

    \r\homelab

    You can easily find cisco 3750's switches and 2800 series routers on Ebay. Older stuff, but good stuff for a CCNA lab that won't cost you much.

    [–]Dirty_Pee_Pants 1 point2 points  (0 children)

    ...or build a server and run eve or gns3. Much quicker time to lab and the added benefit of but having to reconfigure your hardware every time you change your design.

    [–]madmax_br5 7 points8 points  (3 children)

    If you could wave a magic wand to automate a particularly annoying routine task, what would it be?

    [–]DarkAlman[S] 35 points36 points  (2 children)

    If I had a magic wand I would wave printers out of existence.

    [–]akulbe 4 points5 points  (0 children)

    DAMMMIT. I cannot upvote this comment enough!! 🤣

    [–]ForeverYonge 0 points1 point  (0 children)

    AWS Printer Prime - print it in the cloud, delivered same day :)

    [–]AutoModerator[M] 2 points3 points  (0 children)

    Users, please be wary of proof. You are welcome to ask for more proof if you find it insufficient.

    OP, if you need any help, please message the mods here.

    Thank you!

    I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

    [–]WardenWolf 2 points3 points  (1 child)

    Were you impacted by the recent Kaseya vulnerability? I used to work for a large MSP that used Kaseya (wonderful tool, but super-dangerous if compromised) so I'm well aware of what could happen. On that note, I absolutely love what Kaseya can do if you have the necessary voodoo-level knowledge of command line operations. I'm an old-school command line wizard so I appreciate that.

    As for me, I got out of MSP work and into the data center side of things, and honestly never want to work for an MSP again. I'm just glad I no longer have to deal with constantly ringing phones and irate customers.

    [–]DarkAlman[S] 0 points1 point  (0 children)

    I left the consulting world for the private sector for nearly a year.

    Like many other career consultants I needed a break from the grind, but after months at a company I started to hate it, couldn't deal with the rigid team structure, got bored and went back to consulting and I'm much happier now.

    Just more mindful of burnout now.

    [–]IAmTheKingOfNoPants 1 point2 points  (1 child)

    How do you feel all the changes made to accommodate the pandemic and work from home has affected cybersecurity?

    [–]DarkAlman[S] 10 points11 points  (0 children)

    AAAAAAAAAAAAAHHHHHHHHHHHHHHHH

    A lot of companies weren't caught off guard and just had to spin up additional VPN and Teams/Zoom licenses, and buy a ton of laptops. Working from home was just an extension of the existing work from home policy and various IT departments handled it easily enough despite the initial panic buying of hardware + licensing and setups.

    But many organizations have done crazy things like enable RDP servers open to the web, do BYOD (get users to use their own PCs etc), let users take home data on USB drives, no centralized monitoring of Anti-Virus, just utter madness.

    Zoom was another problem with all the various security holes that cropped up shortly after it became the defacto work-from-home conference calling platform on a budget.

    It's no wonder ransomware and phishing attacks went up by a significant percentage during the pandemic.

    The upside of having an MSP like ours is we were able to use our experience to transition over our customers to work from home relatively easily (although we did work 3x 60 hours weeks at the start of the pandemic) because most of what was required was either already in place or copy+paste to implement.

    [–]thegreatpablo 1 point2 points  (5 children)

    I was at Build IT and Kyle Hanslovan asked a seemingly innocuous question that resonated with me and I think is much more nuanced than people are giving it credit for. The question was "Are you increasing the security risk of your clients simply by being their MSP?"

    I'm curious what you do to instill a security first mind set at all levels of your company to ensure that you are increasing your clients' security posture and not compromising it in any way?

    EDIT: When I say "all levels of your company" I mean all levels including ownership, sales, marketing, etc.

    [–]DarkAlman[S] 4 points5 points  (2 children)

    Our CEO reinforces a mentality of being pretty risk adverse. In other words "don't do anything that will get us sued"

    With that in mind it's really up to the senior techs and the managers to reinforce a risk adverse mentality in terms of IT security.

    Another key thing pointed out by my operations manager is to be willing to stop doing business with customers that repeatedly take risks, or don't listen to your advice. You have to avoid the bad press.

    EDIT: You should also define and enforce certain basic standards on all your customers like Strong passwords, MFA, etc. If they can't even comply with those basic things then your shouldn't be doing business with them.

    [–]PM_your_randomthing 1 point2 points  (0 children)

    I've been in MSPs for several years now. And when a client is unwilling to do what it takes to be secure it's hard not to fire them on the spot.

    [–]JuniorPositive88 0 points1 point  (0 children)

    being pretty risk adverse. In other words "don't do anything that will get us sued"

    I fail to understand how you're phrasing this. Isn't "not being sued" the bare minimum in IT security? How could this fall under being "risk adverse"?

    [–]goinginforguns 0 points1 point  (1 child)

    … aaand when the client asks their MSP that question, it’s time for the MSP to bring in the MSSP heh.

    [–]DarkAlman[S] 0 points1 point  (0 children)

    lol, yes

    [–]thekarmabum 1 point2 points  (1 child)

    Do you work from home? I haven't been to the office since covid started. I'm just a network engineer, no management experience.

    [–]DarkAlman[S] 2 points3 points  (0 children)

    It's been 50:50 for us. Several of my techs work from home, others prefer to be in the office.

    We were 100% working from home for several months last year and moved back to a hybrid environment as the restrictions were laxed.

    We can't avoid onsite visits for some things as a number of my customers are required to stay open.

    [–]UnicornGasm 1 point2 points  (2 children)

    How did you get into the industry? Is there anything you would change in your approach/strategy? Any other tips appreciated, looking to start my career in IT through an apprenticeship.

    [–]DarkAlman[S] 7 points8 points  (1 child)

    I always wanted to work with computers. I went to University for a degree in Computer Science but programming wasn't for me, so I got a trade school degree instead.

    From there I just worked my way through the ranks.

    If your the type that prefers fixed hours and looking for benefits and pension try to get in with a large company or multinational. It will take you longer to move up, and you will have to specialize but you'll get benefits over time.

    Where-as if you're the type to think on your feet like me and prefer to learn on the job MSPs and SMBs will probably be more to your liking. You'll have more freedom, and you'll learn more, but the pay and benefits won't be as good. Every major pay raise I've ever gotten in this sector has been from changing jobs, so don't be afraid to apply for stuff if you're feeling stuck.

    [–]UnicornGasm 0 points1 point  (0 children)

    Thanks! Really helps.

    [–]Syncite 1 point2 points  (3 children)

    Recent CS graduate and I've been applying to many Security positions. Dont have professional certs but I've been doing Tryhackme learning pathways. Any further advice to increase the chances of a job in the field?

    [–]DarkAlman[S] 2 points3 points  (1 child)

    Get on the CISSP track, when COVID allows go to hacker conferences to learn and network.

    Don't smoke Marijuana if you want any chance of working for the US govt.

    [–]Syncite 1 point2 points  (0 children)

    Alright. I was planning to get into CCSP instead but CISSP might be better because it seems to be more generalised? Don't think I'd be able to work for the US government since I'm not a US citizen anyways haha

    [–]kaine904 1 point2 points  (1 child)

    If you were running your own business, what would you insource vs outsource to an MSP, given your experience?

    [–]DarkAlman[S] 2 points3 points  (0 children)

    Depends on the size of your business and the complexity of your infrastructure.

    We deal primarily with SMB's and non-profits that either can't afford their own dedicated IT people, or can only afford a full time Tier 1. So having an MSP behind them means they have access to a much larger and more diverse team.

    Some customers use us as a full IT department, while others use us to supplement 1 or 2 in house techs so they have an escalation point, or just outsource the daily riff raff to our servicedesk.

    Once your environment reaches a certain amount of complexity and number of users + devices it makes sense to have in house people. At that point an MSP can switch to more of a consulting role or take over certain IT jobs like Service Desk, helping with automation and planning, or handling Tier 3 + 4 escalations.

    But to answer your question, if I started a small business (being a tech) I would do my own IT until it became too much for me to handle then I'd hire an MSP to take over. Once the business grew to a size where hiring a dedicated tech made sense, I'd hire a dedicate tech and scale down the MSP contract.

    We do that with customers all the time.

    [–]Wasabicannon 1 point2 points  (1 child)

    How do you guys handle your internal documentation?

    Do you guys deal with any self-resolution systems like user facing documentation?

    With being a remote MSP do you guys do anything to vet the person calling in is who they say they are for things like password resets?

    [–]DarkAlman[S] 0 points1 point  (0 children)

    We have canned documentation platform for most things including network info, contracts, contacts, and procedures. Plus the usual mix of Calendars, Teams, and Visio.

    We have a repo of pre-made PDF docs that we can send to customers on demand for self-resolution stuff like "how to setup active sync on your phone" which we tend to lean on during bigger projects and migrations.

    We have a verification process for confirm user identity, and we enforce MFA for customers as well so even if a password gets stolen there's an extra layer of protection

    [–]beldict 1 point2 points  (1 child)

    Are you hiring?

    [–]Individual_Forever70 1 point2 points  (2 children)

    How do you feel about people saying IT is not engineer?

    [–]DarkAlman[S] 0 points1 point  (1 child)

    In my career I've been called an Engineer and an Architect.

    What does that have to do with IT? I have no clue, it just goes on my business card and translates to my pay grade.

    [–]Individual_Forever70 0 points1 point  (0 children)

    While I am not saying One must have an engineer degree (specifically CS/EE) in order to become an engineer, but typical IT guys don’t have it and it is common to find that IT guys don’t have the mindset of average engineers.

    Not only do I feel it, it is very commonly agreed from people I talk to.

    But of course, for people do have engineer mindset and skills, there is no problem calling themselves engineer and get paid as such.

    [–]eraser215 1 point2 points  (10 children)

    What's your automation strategy? And if you have one, which tools/tech are you using, and are you paying for them?

    [–]DarkAlman[S] 10 points11 points  (9 children)

    Automation is key, the more you can automate the more your technicians can do on a daily basis because they aren't busy dealing with the little things. The other side is more automation means you need less eyes paying attention to certain things.

    Alerting, reporting, performance monitoring, patch management, etc is mostly automated like 90%+

    We use a dedicated suite for most of the automation. I won't say which one.

    [–]Dat_Steve 0 points1 point  (7 children)

    What do you mean by alerting, reporting, performance, and patch management is mostly automated? Specifically what does that look like? Of course event notifications are automated, but do you automate response? What types of reporting is automated?

    [–]ErikTheAngry 5 points6 points  (2 children)

    Not OP obviously.

    Reports are generally based on what users like to see. "Your server's uptime is so many 9s, your patching is up to date/x days behind, there have been X alerts which we have responded to".

    Some events are conducive to automated responses. My personal favourite is a local user creation. If a local user is created, it's automagically disabled. The account used to create it, if possible, is also automagically disabled.

    Local user creation being rare and always something that happens through a planned change, it can be whitelisted in advance when it's intended. It's otherwise virtually always a very significant event, so an automated response is always the best way to slow them down while you look into it.

    Another is file events. When a file is saved, I look at the file's headers (simple hex dump) and compare them to known extension/header mappings. If the headers don't match, odds are something encrypted the file... ransomware. If it happens twice in a 2 minute period, user account gets disabled and I start investigating.

    And lastly, network connections. Any unexpected network traffic to well-defined destinations (say a file server, workstations have exactly one reason to talk to it), the automated response is to disable the switchport that workstation's plugged into. Bam, like that, a potentially compromised workstation's off the network and immediately zero threat.

    [–]canondocre 0 points1 point  (1 child)

    That ransonware catchall is smart, I've never heard that idea.

    [–]ErikTheAngry 0 points1 point  (0 children)

    It's by no means definitive.. just another layer of protection. And you have to be very careful to avoid false positives. They do happen, thankfully not super often, and when they happen the user is quite pissed off.

    [–]gray527 2 points3 points  (0 children)

    SIEM, patch management, network monitoring, ticketing, etc systems all have an API now. Windows has Powershell. Python is very powerful at calling APIs and organizing. Smart responses can open tickets, disable accounts, disable network nodes, deploy patches. etc, etc. Responses can be automatic or await human interaction.

    [–]RazoRReeseR 2 points3 points  (2 children)

    The MSP I work for uses the ConnectWise suite of applications for those tasks. Works pretty well.

    [–]StarMech -1 points0 points  (1 child)

    Just wish fucking automate control center would run on Linux. The web portal is annoying.

    [–]Ucla_The_Mok -1 points0 points  (0 children)

    Windows VM from within Linux would surely be an option.

    [–]mata_dan 0 points1 point  (0 children)

    Does it end in '-gage'? xD

    [–]ssharma123 0 points1 point  (4 children)

    What's the biggest IT mess up you have had to deal with?

    [–]DarkAlman[S] 2 points3 points  (3 children)

    I am not at liberty to discus the details

    REDACTED

    I've cleaned up after a lot of Ransomware over the years. That's never fun.

    [–]affixqc -4 points-3 points  (2 children)

    If you regularly have to clean up ransomware from networks you manage, you've got some problems to address...

    [–]DarkAlman[S] 1 point2 points  (1 child)

    I was waste deep in it when Ransomware first appeared years ago and I still have PTSD about it.

    Nowadays security practices have come a very long way and ransomware prevention is part of our daily lives, but we still get a fair number of walk-up clients begging us for help with ransomware out of the blue.

    [–]affixqc -1 points0 points  (0 children)

    That's good to hear! My first IT job we dealt with it regularly. Wasn't until I joined my current MSP that I learned what proper preventive measures looked like and it opened my eyes :)

    [–][deleted]  (8 children)

    [removed]

      [–]Choralone 6 points7 points  (2 children)

      As a hiring manager, who's been in this field for 25 years....

      Tech certs in general mean very little. I know far more people who learned on their own / on the job than I do people who learned through certification. If you're young and come in with some basic tech certs, it hints that you might be really interested in this field... but that's all. I'll know more about how you'll perform from a 30 minute interview than I will looking at your certs.

      The first thing I look at on a CV are your technical skills - what kind of stuff do you know how to do, and how relevant is it to me.

      I've hired enough CS grads that were absolutely useless that it's almost a net negative if it shows on your CV - give me the kid who jumped into tech right out of highschool and loves it.

      • I work on the operations side, not the programming side. Fundamentals for software development ARE important.

      [–][deleted]  (1 child)

      [deleted]

        [–]Choralone 0 points1 point  (0 children)

        Thanks… what you just wrote is 1000% exactly how the real world works, at least from what I’ve seen.

        The tech world is not academia.. it’s still the Wild West.

        [–]DarkAlman[S] 11 points12 points  (4 children)

        TLDR: I've got a learning disability (ADHD) and it's extremely difficult for me to get through the mandatory courses. I'm honestly just really terrible at studying and writing exams.

        I hold a ton of Networking, Storage, Veeam, Sales certs etc that I don't bother to put on my profile fyi. Mostly because they're expired.

        Oh I've GONE for certs... but actually passing is a different problem.

        I've got a learning disability and I can't learn in classrooms, I actually tend to fall asleep or lose attention with 30 minutes. So any exam that has a mandatory classroom phase or e-learning like Vmware, cisco, lots of HP+Dell stuff, etc I have no hope in hell of passing. I often need multiple attempts at the exam to pass, so my employers learned to focus on the techs more likely to pass on the first try and let me focus on fixing things instead.

        I have to learn hands on. Give me a new product and I'll figure out the ins and outs in an hour, but go to a class? snore

        [–]Kozm 2 points3 points  (2 children)

        When we’re you diagnosed with ADHD? I feel like I may have ADHD but I don’t want to self diagnose

        [–]BroughtTheRuckus 4 points5 points  (0 children)

        Not OP but also in the IT industry with ADHD. I got diagnosed at 26 with moderate ADHD and was put on to Concerta which was fairly life changing. Happy to answer any questions if you have any 👍

        [–]PM_your_randomthing 0 points1 point  (0 children)

        Dude, it's never too late. I'm 37 and just started things. I wish I would've done this for college. Maybe would've been done in 4 instead of 7 years

        [–]tellemurius 0 points1 point  (0 children)

        Those required VMware classes are a joke for anyone thats got onhand experience. Even my fellow classmates, each of us have already blown up and rebuilt clusters. The exams toss mostly easy questions you would know just from fiddling vcenter. I'm on track to complete my VCAP in a couple of months. Just gotta convince my boss to toss another 3k for a class.

        [–]dasdas90 0 points1 point  (1 child)

        How do you find clients?

        [–]DarkAlman[S] 0 points1 point  (0 children)

        We have an office filled with Type A sales people that take care of that for us.

        The usual mix of networking, cold calls, trade shows (when not dealing with COVID), follow ups, and references.

        [–]iama_triceratops 0 points1 point  (1 child)

        Are you one of those MSPs I hear stories about on Darknet Diaries? Like having a client with a domain controller with RDP open to the internet for all the users to log into?

        [–]DarkAlman[S] 0 points1 point  (0 children)

        I like to think of us as the guys that steal customers from those MSPs and fix them.

        [–]dadofbimbim 0 points1 point  (1 child)

        What is your current workstation setup? What tools are you using daily for work? And personal thanks!

        [–]DarkAlman[S] 0 points1 point  (0 children)

        I have a mediocre laptop with a docking station at the office with 2x additional monitors.

        Nothing too fancy, just a Core i5 with 16gb of RAM and an SSD. Adequate, light weight, and good battery life.

        I have a gaming laptop for personal use, my work laptop doesn't have to be a powerhouse.

        Tool wise in my bag I carry an electric screw driver, bits, colored tape, my console cable, USB ethernet adapter (for a 2nd LAN plug), USB hub, a purple patch cord (so people don't steal it), flashlight, and a Patroit Supersonic thumbdrive.

        [–]fake_it_till_umakeit 0 points1 point  (2 children)

        Hi, As a MSP do you provide the product with the same price the vendor are offering? Or you are more expensive than the vendor, The second question let's say you manage EPP (endpoint security) for 100 clients how many people you need to manage them ?

        Thanks

        [–]paradox1108 1 point2 points  (0 children)

        MSP markup on products from other vendors is typically 20-30% from what I’ve seen. Direct profit margin for an MSP on services (i.e. labor and support, etc.) ranges depending on product offering but a less sophisticated MSP will see in the 30-40% range and a more sophisticated MSP will be 40-50%

        [–]DarkAlman[S] 0 points1 point  (0 children)

        We are the vendor/reseller

        For endpoint security it depends on the platform, the number of endpoints, and your incident count. Is your endpoint security person only doing monitoring? or remediation as well?

        1 person can monitor thousands of endpoints easily enough, but remediation depends on the daily ticket volume.

        [–]dj_bpayne 0 points1 point  (3 children)

        Is it possible to progress your IT career at an MSP? Or should you be looking to job hop to an internal position within a year or two of experience?

        [–][deleted]  (1 child)

        [deleted]

          [–]DarkAlman[S] 0 points1 point  (0 children)

          I've have companies reject to interview me in the past because I worked for MSPs and they didn't believe anyone that's worked for an MSP would want to go onto a note relaxed environment. Crazy...

          A lot of more senior techs in the consulting world (myself included) end up burning out on occasion and go into private sector. We end up hating it and going back into consulting before the end of the year.

          Companies end up not wanting to hire former consultants because we don't last that long.

          Dealing with siloed job roles, unions, change process, etc drives us absolutely crazy. Honestly we get bored!

          But I'm the type that the stress doesn't bother me 99% of the time, I thrive under pressure.

          [–]DarkAlman[S] 0 points1 point  (0 children)

          Depends on the MSP, how big it is, and what the culture is like.

          If it's a big team and they emphasize training and give you chances to learn, get certs, and shadow other techs then yes you can progress.

          But if it's a 5-man show and they make you do the same thing every day then I'd say no...

          I will say that almost every major raise I've ever gotten has been as the result of changing jobs.

          [–]qutius 0 points1 point  (1 child)

          As someone who deals with storage, how much do you hate EMC? IBM?

          [–]DarkAlman[S] 0 points1 point  (0 children)

          EMC - Evil Machine Company

          I LOATH EMC, overpriced and under performing junk

          [–]341913 0 points1 point  (1 child)

          What does the average day look like for someone who has worked in the MSP space for so long?

          [–]DarkAlman[S] 1 point2 points  (0 children)

          Come the office

          Make coffee

          Go through a ton of emails, review my tickets

          Join various meetings with customers + vendors

          Follow up on tasks with the team, take escalations as they come up

          Work on whatever projects I've got on the go

          Order varies based on how many fires I have to put out that day

          [–]DanTheFeeder 0 points1 point  (1 child)

          I actually am a bit like you, I saw in a previous comment you started out in Programming but dropped it and pursed something still in computers. I remember doing the programming course for about a year and realized this really isn't my speed.

          I have a degree in IT but kinda having trouble finding a job. Most places tell me I'm too green or when they ask for help desk I don't really have much experience in it. I did work for an MSP a few months before COVID happened and got laid off when it really hit the fan in the spring. My only experience so far is doing deployment both for that MSP and some contract work I had this past Spring.

          Are there any tips or maybe something I can pursue with what experience I have? It's been really frustrating trying to get a job personally, I've tried recruiting agencies, Indeed/Ziprecruiter, etc but I feel like overall I just lack something to land a job.

          [–]DarkAlman[S] 1 point2 points  (0 children)

          Everyone starts on the helpdesk. Just keep applying and expect that you'll have to put your time in to build experience.

          If there's something in IT that you want to specialize in or are curious about like Cloud, 365, Linux, Storage, networking, security, etc start researching, watch videos, do some lab stuff, anything to get some knowledge and experience. Even a little bit can go a long way in an interview as it shows that you could be worth pushing down that track.

          "Looking for someone with all these certs, 10 years experience, willing to pay less than starting wage for a tier 1 tech" - more common than you think.

          [–]ElZilchoTX 0 points1 point  (1 child)

          As an IT Engineer/Manager at an MSP what are the biggest “blind spots” that you face when is comes to monitoring end-to-end infrastructure/performance/security/traffic etc? What tools do you use to monitor? What does your boss care about the most? What are you personally measured on?

          [–]DarkAlman[S] 1 point2 points  (0 children)

          As an IT Engineer/Manager at an MSP what are the biggest “blind spots” that you face when is comes to monitoring end-to-end infrastructure/performance/security/traffic etc?

          It's not so much blind spots as information overload. It's easy to collect data, it's another to be able to filter it and turn it into something useful. So much of what we collect becomes noise.

          What tools do you use to monitor?

          We use a purpose made tool but I won't say which one.

          What does your boss care about the most?

          How much revenue are we generating? are the customers happy with the service?

          What are you personally measured on?

          How many contracts we sign over the course of the year, our retention rate, how much additional business (sales) have we generated, how many billable hours do I average in a week.

          [–]w4fun 0 points1 point  (0 children)

          Hi! I would like to know more regarding management. How dou you manage resources? Do you you any project management tools, do you have an app like Jira, do you need to integrate with customers CRM? How does it all connect?

          [–]NuclearAmoury 0 points1 point  (1 child)

          How do you deal with ridiculous SLAs?

          For me, I deal with them with a lot of stress.

          [–]DarkAlman[S] 0 points1 point  (0 children)

          At my old job with a lot of caffeine and overtime

          At my current job I have the power to not put us in that position in the first place.

          [–]X_CosmicProductions 0 points1 point  (0 children)

          Hey! I'm a 16 year old guy that enjoys thinking about ai and finds it fascinating. I have a bachelor in front of me in applied informatics. Or I could go thevcomouter science route. What is your opinion on that?

          [–]Dragnskull 0 points1 point  (1 child)

          what's your background in this field? what would you say is different between standard business/residential IT vs engineer level positions?

          I have an extensive background in residential and small business IT and am currently going to school for a BA in computer engineering, haven't considered "IT Engineer" because I've never actually heard of that term before, but it sounds like it may be more in line with my experience

          [–]DarkAlman[S] 0 points1 point  (0 children)

          My background was 7 years working at a small business that grew into an Enterprise due to various acquisitions. I started as a desktop tech and became a server generalist eventually becoming the senior engineer in the department.

          Today I mostly manage the team, but I also work as a consultant to design and build IT solutions.

          As a consultant I specialize in Full-stack Virtualization, Networking, Storage, Backup and Disaster Recovery, and I'm an Active Directory guru.

          The major difference between SBS (Small Business) and the Enterprise world is the kind of technologies you get to work with.

          Working at an SBS you have to be a generalist and know a bit of everything, while in the Enterprise you need to specialize more. That and most SBSs can't afford all the bells and whistles in terms of software and hardware, but it's not like they would take advantage of them either.

          At the Enterprise level you can afford to pay someone that deals with something specific full time.

          You've got Desktop Technicians, Network Engineers, Voice Engineers, Datacenter Engineers, Database Administrators, Storage Engineers, Application specialists, Mainframe, Email, Cloud, Security people. The list goes on.

          [–]psyco_llama 0 points1 point  (0 children)

          How would someone with 15-20 years IT experience re-enter the market?

          [–]TheLastRatatouille -5 points-4 points  (1 child)

          i met a guy from France on omegle about 2 days ago. he was so lovely and a real gentleman, so i gave him my instagram. he didnt have it, so he created one so we could talk. i went on there today to find that i think he has taken down his account of something of that nature, and, even though i barely know him, im actually really upset about it. is there any way id be able to contact them? i promise im not a creepy stalker or something. im 15 and lonely and looking for someone to talk to while im in an extended lockdown lmao

          [–]tPlayer1980 0 points1 point  (0 children)

          Can you recommend any books ? Not just dry manuals, but something that really made impression on you ?