Near-Real Time Vendor Data On-Prem

The company I am working for has a copy of a vendor database on-prem for reporting and analysis.  The database is over 4 TB at this time, and constantly growing due to acquisitions and continued patient care.

Currently, our process for updating the data is log shipping, a tried and true method which has been successfully updating the database for years now.  Requirements are ever-changing, and there is a desire to have data that is updated more often during the day.  This article describes the current process, and explains how Availability Groups can be used to fulfill the requests.

Click here for a pdf that shows the current process, and both new processes, as explained in detail below.

TSQL Tuesday #97

This month's T-SQL Tuesday is being hosted by Malathi (blog | twitter) - Setting learning goals for 2018

I have a feeling 2018 is going to bring a change in my work life and I would like to be prepared for that change.  Feeling confident in my skills will be a big part of how I get through that transition.  This post goes perfectly in line with some of the planning I have already started, and this gives me the chance to actually write it down (and therefore commit even more too)!

Learning Goals:

All of the learning I have done so far has been looking up this issue or researching a specific thing, etc.  I want to do some structured learning in 2018, so my goal is to be prepared at the end of Q1 2018 to take the 70-462 certification exam (Administering Microsoft SQL Server 2012 Databases).  I know that certification exams can be crammed for and don't always give a sense of a person's skills, but I am using them as they were intended, and it has helped me create and organize a learning plan.  I have more knowledge than I realize, but I am excited about tieing things together in my head for more of a well-rounded understanding.  Not knowing all the specific details and the underlying "why" behind the foundational topics that are normal for a DBA can sometimes cause me to hesitate.  This leads to me having a lack of confidence in my skills at key times.  Also, I don't always use the correct terminology even though my overall understanding is in place.

Using Triggers for auditing changes to sql jobs

Recently there was a job that was changed to stop at a step earlier than normal (presumably for testing).  When it ran that night, it didn't run all the steps.  This job has a lot of dependencies which needed to be re-run the next day and caused a bit of chaos for a few applications, data warehouse data, and reports.


In order to see who made the change, there would have to be some sort of auditing running prior to when the change happened.  Auditing of sql jobs and more had already been identified as a project for our department, but hasn't been prioritized yet.  This of course, doesn't help identify what happened for this specific time.  :)


 In the mean-time, management wants to be able to track these types of changes.  Therefore, I started researching the multiple options and found some common solutions:


TSQL Tuesday #96

This month's T-SQL Tuesday is being hosted by Ewald Cress (blog | twitter) - Folks Who Have Made a Difference

There are 3 people who have really contributed positively to my career in data.  Two are in a dba role, the other wasn't anywhere close. They all helped in a different way, which is why I am listing them all.

The first person I want to talk about is a past boss.  When I was starting out in my career, I was an administrative assistant in the pharmacy department of an HMO.  The head of the department, Mary, was a pharmacist and very friendly, yet tough.  She worked with a committee of doctors to set policy within the HMO we worked for, and it affected all of the members we served throughout Florida.

I did the normal administrative assistant type of duties, but she allowed me to stretch beyond those borders and do other stuff that was helpful as well.  One of those pet-projects that I worked on imported prescription utilization data into an Access database, manipulated it, then exported the results in both Excel reports and Word documents.  I was so proud of what was done, and should have gotten the hint that was the direction I wanted to go in (but of course I didn't at the time).  I still think about her.  She helped me in more ways than I realized and was a very good mentor (and boss) to have!

The next person who contributed to my path was a dba I used to work with, Stuart.  When I realized I should actually be in IT, I found a job as a report writer for a local company.  I worked my way into being a developer, and then started working with databases more.  I would ask Stuart all kinds of questions and he was always patient.  I think he recognized that I wanted to learn, and that I wasn't questioning the decisions behind processes, but wanting to understand for knowledge.  He helped me learn the basics of things so that I can explore more on my own.  For example, he sat down with me and helped me understand how to prioritize the list of database objects in the FROM statement and why it mattered.  That jump started many nights of studying to figure out even more.  I still talk to him from time to time and hope to be a friend for many years to come.  He also always had great stories to listen to!

I admittedly don't know how to thank either of them for their patience and understanding.  The best thing I can think of is to help others in a similar manner.  I feel all gooey inside when I think about them and hope that I can have that affect on someone else one day. :)

The third person who I wanted to talk about is Sean McCown.  I met him at a SQL Saturday in Baton Rouge a few years ago where I attended a session he presented on Powershell.  That particular SQL Saturday was my first one ever.  I hadn't even known that type of event existed until that year.  The event was fantastic and I was so excited during and after (I have to give a shout out to the BRSSUG - they are awesome people)!  The way he presented the information and answered questions was something I hadn't really experienced before.  It was very clear that he enjoys what he does, and his upbeat personality draws people in.  I remember being in awe and wanting to do that too.  He also recorded himself, as he does with most of his sessions.  That was the first time I really had thought about and started researching what it takes to brand myself.  Him and Jen also put on a weekly webshow at DBAs@Midnight, which is an incredible way to talk to them regularly. Since then I have been a speaker at a few SQL Saturday events and user group meetings - partly in thanks to him. I still think back to the initial excitement and feel so thrilled that the SQL community is filled with so many wonderful people!



Working with current and previous rows in text files

An issue came up today where the vendor was sending a text file, but the text for some of the lines was on multiple lines, rather than a single line.  After researching, I found out that I needed to compare a line of text and based on certain criteria append it to the prior line.  I couldn't just strip certain end of line markers because it was mixed.

I am using a dummy test file in this example, but here is how everything played out:

Using Notepad++ to change end of line characters (CRLF to LF)

End of Line characters include CR or LF.  Windows uses both CRLF at the end of a line, whereas Unix uses only a LF.

  • CR = Carriage Return
  • LF = Line Feed

Recently, while troubleshooting why data wont import successfully as part of an automated process, I was pulling a subset of data out of the main text file, but the end of line markers weren't correct.  I copied several lines using Notepad ++ and it automatically used CRLF markers.  The automated process expected the end of line markers to be LF to be read by the SSIS package properly.

This article will help job my memory when I run across this again... but hopefully it helps someone else too! :)

First off, within Notepad ++ to see the end of line markers, you need to indicate you want to see them.  Click on View > Show Symbol > then either Show End of Line, or Show All Characters if you want to see spaces and tabs, sometimes the second option is easier).

TSQL Tuesday #59

This month's T-SQL Tuesday is being hosted by RealSQLGuy - Tracy McKibben (blog | twitter) and he is calling out for all Heroes!

What a fabulous topic! I really had think about it though. I think I have 1 personal hero and 2 professional heroes. I should give a disclaimer here - I have a hard time picking a favorite anything, so this just follows suit I guess. :)

First, my professional heroes (not listed in any specific order):

  • Stuart - I used to work with Stuart at my previous employer. I didnt think of him in this regard at the time, but looking back I do. He was the DBA, and I was more of a programmer/database developer. Every time I had a question, he would explain why and I would walk away feeling smarter, not dumber. I have worked with people who made you feel like an idiot when you asked a question, and I never had that from Stuart. In fact, he would explain in depth if I asked. For instance, he taught me tricks on query optimization, rather than just doing it and giving it back to me with no explanation. I remember a specific moment at my current job: it was the first time I changed a 45-min cursor within our ETL process to a 5-10 min set based query. I was ecstatic! I dont think I would have thought of it in that way without working with Stuart before. In all honesty, when I think of a DBA, he is what I think about. I work with the BI side of things now, so it is different than my work then, but I wouldnt have the confidence I do now without that experience.
  • Chris - I used to work with Chris at my previous employer, too. He is the type of person that is very smart, down-to-earth and nice. He also speaks code fluently as another language (or he at least seems to). :) He is my hero because if it werent for him, I would probably not have realized that I am not a programmer. Sure - I can do it. But, watching him be able to understand and code so quickly and easily, I realized that is not my natural-born skill. For me, I have to look up syntax and really dig into a project because it didnt come as easily for me. The hero part comes from him being such a great guy about it. He also took the time to help you without making you feel stupid in any way. How many people do you know that are so good at what they do, but dont have the ego behind that? I am the type that feels heavily responsible for what I am doing, which sometimes means I keep grinding at a project or multiple until I feel like it is perfect - and it never is. When I realized I am not a programmer, it made me think of what I can do that easily - and that is what led me to working with data much more.

I have saved the best for last - my personal hero:

  • My Mother - She is kind and sweet and everyone loves her! She is what I want to be (when I grow up - haha). I have had some very challenging years and I dont know how I would have gotten through them without her. I am a positive person, but she helps me think in that way when I really dont want to. Maybe I have someone at work who doesnt like me and it shows - I take things personally and next thing you know I am not thinking happy thoughts, much more hurt. She helps me get back in the right mind-set and not to worry about it. She tells me - I need to let it slide like water slides off a duck's back. I will admit, I hate hearing her say that - but she is right and I know it. :) I also like knowing she relies on me as much as I do her - so its not all 1-sided. :) I am lucky enough to be able to say my mother is my best friend and is definitely a super-hero!

Thank you for visiting!