Wednesday, November 30, 2016

BIDATA 201A: TSQL: FUNCTIONS


--****************** SQL Programming *********************--

-- This file explains how to create and use Functions

--**********************************************************--



-- SQL Server 2000 and 2005 supports three types of user-defined functions:

      --Scalar functions

      --Inline table-valued functions

      --Multistatement table-valued functions



-- Create the simple function

CREATE FUNCTION fn_NewRegion

(@myinput nvarchar(30))

RETURNS nvarchar(30)

BEGIN

  IF @myinput IS NULL

  SET @myinput = 'Not Applicable'

  RETURN @myinput

END  





USE [AW_Project01_OLTP_DB]

GO

-- Using just a Select statement (note the Nulls on the Region column)

SELECT LastName, City, Region, Country

FROM Employees



-- Using the function to fill in the Null values

SELECT LastName, City, dbo.fn_NewRegion(Region) AS Region,

   Country

FROM Employees



-- Altering a function

ALTER FUNCTION fn_NewRegion

(@myinput nvarchar(30))

RETURNS nvarchar(30)

BEGIN

  IF @myinput IS NULL

  SET @myinput = 'International'

  RETURN @myinput

END





-- Using the function to fill in the Null values

SELECT LastName, City, dbo.fn_NewRegion(Region) AS Region,

   Country

FROM Employees



-- Dropping a function

DROP FUNCTION dbo.fn_NewRegion

BIData 201A: Assignment 08: Create the Data Warehouse script and add it to your BI solution


Create the Data Warehouse (Due by class 9)


In this milestone, you need to create the Data Warehouse script and add it to your BI solution. The steps to do this covered in the 4th module of this class. 

·       Module 4: Building the Data Warehouse (Ch4 and 5)

You will need to turn in a SQL database creation script, and backup and restore script, and a database backup file, in addition to your previous files from Milestone 01; similar to the ones shown here:


NOTE: I recommend you start your ETL script directly after creating the database so that you will have a head start on Milestone 03. You have been warned!



Build the Data Warehouse

Below is the script I ran to build it

GO
CREATE DATABASE [DW_Project01] ON PRIMARY
( NAME = N'DWPubsSales'
, FILENAME = N'C:\BIData201B\Assignment08TimPauley\DatabaseBackup\DW_Project01.mdf'
, SIZE = 10MB
, MAXSIZE = 1GB
, FILEGROWTH = 10MB )
LOG ON
( NAME = N'DWPubsSales_log' ,
FILENAME = N'C:\BIData201B\Assignment08TimPauley\DatabaseBackup\DW_Project01_log.LDF'
, SIZE = 1MB
, MAXSIZE = 1GB
, FILEGROWTH = 10MB)
GO
EXEC [DW_Project01].dbo.sp_changedbowner @loginame = N'SA'
, @map = false
GO
ALTER DATABASE [DW_Project01] SET RECOVERY BULK_LOGGED
GO


Next I created the tables



BIData 201A: Assignment 07 - Review


Milestone 1 Review

For Milestone 2 you had to:

Design and Begin Documentation (Due by class 8)


In milestone 01 you must create developer design documentation using an Excel spreadsheet and the beginning of the formal design documentation in a Word docx file. The steps to do this covered in first 3 modules of this class. 

·       Module 1:  An Overview of BI Projects and Solutions (Ch. 1 and 2)

·       Module 2: Planning and Documentation (Ch. 3)

·       Module 3: Designing the Data Warehouse (Ch. 4)

You will need to turn in a Word document, Excel spreadsheet, and a VS solution; similar to the ones shown here:




To fill in the documents you can do it manually, using the SSMS Object Explorer, or use SQL Code to help with the process.


Professor Root gave us this script to help fill in the [DataWareshouse] Excel sheet





--****************** Milestone 1 Review *********************--

-- Design and Begin Documentation (Due by class 8)

-- In milestone 01 you must create developer design documentation using an

-- Excel spreadsheet and the beginning of the formal design documentation in a

-- Word docx file. The steps to do this covered in first 3 modules of this class.

--**********************************************************--



-- To create the documentation you can use SQL's System tables to gather MetaData.

USE AW_Project01_OLTP_DB;



-- Microsoft has a number of System tables that gives you MetaData you can use for planning.

-- One of these is SysObjects.

SELECT * FROM SysObjects;

-- OR

SELECT * FROM Sys.Objects;



-- Get a list of only the "User" tables, and their Primary and Foreign Keys

SELECT

  NAME

, [Parent object] = iif(parent_obj = 0, 'NA', Object_Name(parent_obj))

--, * 

FROM SysObjects

WHERE xtype in ('u', 'pk', 'f')

ORDER BY parent_obj;



-- Microsoft has Many MetaData views, here is one just for tables

SELECT * FROM Sys.Tables;



-- Here is an example that gets the Names of the Columns

SELECT * FROM Sys.all_columns;



-- Notice that the column Data Types referance a lookup ID, system_type_id

SELECT * FROM Sys.types;



-- We can combine these three to get a custom look at the MetaData overall.

Declare @DatabaseName nVarchar(100)= 'AW_Project01_OLTP_DB'

Declare @SchemaName nVarchar(100)= '%%%' -- Defaults to all Schemas

Declare @TableName nVarchar(100)= '%%%' -- Defaults to all Tables



EXEC('USE ' + @DatabaseName)



SELECT

[Schema Name] = SCHEMA_NAME(T.[schema_id]) ,

[Table Name] = T.name  ,

[Column Name] = C.name ,

[DataType] = Ty.Name ,

[DataType Max Length ] = C.max_length ,

[DataType precision] = Case When C.precision = 0 Then 'NA' Else LTrim(Str(C.precision)) End,

[DataType scale] = Case When C.scale = 0 Then 'NA' Else LTrim(Str(C.scale)) End

FROM sys.tables AS T INNER JOIN sys.columns AS C

  ON T.OBJECT_ID = C.OBJECT_ID

INNER JOIN Sys.types AS Ty

  ON C.system_type_id = Ty.system_type_id

WHERE C.name LIKE @TableName

  AND SCHEMA_NAME(T.[schema_id]) LIKE @SchemaName

ORDER BY [Schema Name], [Table Name];





-- Because All of these tables are found only in Microsoft SQL Server, you might want to use

-- something more generic like the ANSI Information Schema Views

SELECT * FROM [INFORMATION_SCHEMA].[TABLES];

SELECT * FROM [INFORMATION_SCHEMA].[COLUMNS];



-- Of course there too can be combined to get a better overall view of the MetaData

SELECT

  [ColumnFullName] = DB_Name() + '.' + C.TABLE_SCHEMA  + '.' +  C.TABLE_NAME  + '.' + COLUMN_NAME

, DataType = Case

  When DATA_TYPE in ( 'Money', 'Decimal')

    Then IsNull(DATA_TYPE,'')

    + ' (' +  Cast(NUMERIC_PRECISION as nvarchar(50))

    +  ',' +  Cast(NUMERIC_SCALE as nvarchar(50))

    + ' )'

  When DATA_TYPE in ('bit', 'int', 'tinyint','bigint', 'datetime', 'uniqueidentifier')

    Then IsNull(DATA_TYPE,'')

  Else  IsNull(DATA_TYPE,'') + ' (' +  Cast(IsNull(CHARACTER_MAXIMUM_LENGTH,'') as nvarchar(50)) + ')'

  End

, IsNullable = IsNull(IS_NULLABLE,'')

--, ORDINAL_POSITION

--, COLUMN_DEFAULT = IsNull(COLUMN_DEFAULT,'')

FROM [INFORMATION_SCHEMA].[COLUMNS] as C

JOIN [INFORMATION_SCHEMA].[TABLES] as T

  ON C.TABLE_NAME = T.TABLE_NAME

WHERE C.TABLE_SCHEMA in ('dbo') AND C.TABLE_NAME NOT in ('sysdiagrams')

Order by C.TABLE_SCHEMA, C.TABLE_NAME, C.ORDINAL_POSITION

Go

-- NOTE: There are a number of columns that we need in our design documents

-- and if we create our SQL code just right it will type a lot of the contents for us!

-- For example I can use the result of the Previos query to start my BISolution Worksheet.



'DEMO COPYING AND PASTING THE RESULTS INTO EXCEL'

Go



-- A few years ago, Microsoft introduced an undocumented stored procedure (Sproc) that will allow you to run code using the name of each table in a database.

-- This is something that we have could do on your own by creating a SQL cursor, but it is nice to have it written for you!

-- The stored procedure is named “sp_msforeachtable” and it allows you to submit a text string for processing as shown here

sp_msforeachtable @Command1 = 'print  "?" ';

Go



-- Note that a question mark in double quotes is used as a placeholder which will be replaced with each table’s name when the Sproc is called.

-- For example, if there is a table named dbo.store in the database, this Sproc executes the following code:

print '[dbo].[stores]';

Go

-- It does this for each user table if finds in the current database, but not system tables.

-- Speaking of system tables, you can use this Sproc to get meta data about your user tables easily. Here is an example:

sp_msforeachtable @Command1 = 'Select Object_Name(object_id), * From Sys.Columns Where Object_ID = Object_ID("?")';

Go

-- You can also use Microsoft’s stored procedures, like sp_help, with each table as follows:

sp_msforeachtable @Command1 = 'sp_help [?]';

-- Notice that sp_help is called for each table in the database and that I am using square brackets instead of a double quote.

-- This make this command evaluate as an object and not just a string, which can be important sometimes, though in this case it doesn’t matter.

Go



-- We create our own Sproc and pass in the table name as an argument value.

-- This next example uses the TSQL Exec command run a string of text characters as if it were a typed-out SQL statement.

Create Proc pSelTop2

(@TableName as nVarchar(100))

AS

      Begin

             Select [TableName] = @TableName;

             Declare @SQLCode nvarchar(100)= ' Select Top 2 * from  ' + @TableName

             Exec(@SQLCode)

      End

;

Go



-- Now I can use Microsoft’s stored procedure to execute my stored procedure like this:

sp_msforeachtable @Command1 = 'exec pSelTop2 "?" ';

Go



'DEMO COPYING AND PASTING THE RESULTS INTO EXCEL'

Go





-- Each version of Microsoft SQL server comes with several new functions and Sprocs. In 2010, this included not only sp_msforeachtable,

-- but a similar stored procedure that loops through each database on the server. 

sp_msforeachdb @Command1 = 'sp_helpDB [?]';

Go



-- Keep in mind that both are considered non-supported code. Relying on these in a production database may cause trouble

-- if one of Microsoft service pack changes the code. If you are consernd about this, remember that you can always create

-- your own stored procedure using Microsoft’s code, which can be displayed using sp_HelpText.

sp_helptext [sp_msforeachtable];

Go


Monday, November 28, 2016

BIData 201A: Assignment 07: Design and Begin Documentation

Design and Begin Documentation (Due by class 8)

In milestone 01 you must create developer design documentation using an Excel spreadsheet and the beginning of the formal design documentation in a Word docx file. The steps to do this covered in first 3 modules of this class. 

·       Module 1:  An Overview of BI Projects and Solutions (Ch. 1 and 2)

·       Module 2: Planning and Documentation (Ch. 3)

·       Module 3: Designing the Data Warehouse (Ch. 4)

Watched this video on bad designs. Centipede vs Star vs Snowflake



STEP 1: Restore DB



restored successfully



Changed permission to grab database diagram forum to aid this here


OLTP Project


STEP 2: Create Excel Document
I used the instructor provided doc. Then made changes to it


Sunday, November 27, 2016

BIData 201A: : Final Description: BI Concepts Planning Implementation Project



Tim Pauley


BIData 201A, Assignment 07


Due: November 28, 2016 *Class 8


UW Extension

Certificate in Business Intelligence: Building the Data Warehouse




Building the Data Warehouse Project


In this project, you will create a simple BI solution using some of the data found in Microsoft’s AdventureWorks2012 database. You instructor has created a simplified version of this database for you, and your job will be to use this new project database to create the solution.

Project Scope


This BI solution is being created for a small department that is interested in customer sales. The tables are filled with a moderate amount of data as shown in Figure 1. The number of users of the solution will be less than ten employees. So, performance tuning is not considered an issue at this time. We will be creating the prototype of the final solution. This solution must include a SQL Server Data warehouse and an SSIS ETL package. The solution must also contain the developer documentation in the form of an Excel spreadsheet and a formal documentation in the form of a Word document.

Required:

·       (1) SQL Server Data warehouse

·       (1) SSIS ETL package

·       (1) Developer documentation (Excel spreadsheet)

·       (1) Formal documentation (Word document)

Figure 1: The number of rows in AW_Project01_OLTP_DB’s tables




OLTP Source Database


The new project database (called AW_Project01_OLTP_DB) contains the tables shown in Figure 2. Your instructor has provided a backup file to restore to this database on your computer. You will find this backup file along with a script file to perform the restoration in the project folder.

NOTE: I recommend that you place the Project folder in your C:\_BISolutions folder before you run the SQL restoration script or the script will not work. Optionally, you may also change the paths in the SQL script.

Figure2: The tables of the AW_Project01_OLTP_DB database.





Note: The data in this database has known inconsistences, so examine the data carefully and "if in doubt, leave it out!"



Figure2: The tables of the AW_Project01_OLTP_DB database.









Dimensions and Measures


From these tables, you must extract (2) measures and (5) dimensions. Keep in mind that the dimensions include a fact dimension, a many-to-many dimension, and three standard dimensions.

Timeline


The project contains 3 milestones. At each of these milestones are a portion of the project. The milestones are:

Design and Begin Documentation (Due by class 8)


In milestone 01 you must create developer design documentation using an Excel spreadsheet and the beginning of the formal design documentation in a Word docx file. The steps to do this covered in first 3 modules of this class. 

·       Module 1:  An Overview of BI Projects and Solutions (Ch. 1 and 2)

·       Module 2: Planning and Documentation (Ch. 3)

·       Module 3: Designing the Data Warehouse (Ch. 4)

You will need to turn in a Word document, Excel spreadsheet, and a VS solution; similar to the ones shown here:






Create the Data Warehouse (Due by class 9)


In this milestone, you need to create the Data Warehouse script and add it to your BI solution. The steps to do this covered in the 4th module of this class. 

·       Module 4: Building the Data Warehouse (Ch4 and 5)

You will need to turn in a SQL database creation script, and backup and restore script, and a database backup file, in addition to your previous files from Milestone 01; similar to the ones shown here:



NOTE: I recommend you start your ETL script directly after creating the database so that you will have a head start on Milestone 03. You have been warned!

Create an ETL Process and Final Documentation (Due by class 10)


In this milestone, you need to create an ETL script and an SSIS project in your BI solution. The steps to do this covered in the 5th and 6th modules of this class. 

·       Module 5: ETL processing with SQL (Ch. 6)

·       Module 6: ETL processing with SSIS (Ch. 7 and 8)

Note: You need to turn in a SQL database ETL script and SSIS project, in addition to your previous files from Milestone 01 and 02; like the ones shown here:






Project partner


Each student will have a project partner. Project partner provide assistance and peer reviews during the process.

You can choose a project partner if you want, but if you have not done so by milestone 2, one will be assigned to you.

Turning in Your Work


As each milestone is completed students are to submit their work to Canvas and on the discussion topic for each Milestone. The Student's project partner will download and provide a peer review using the zipped files posted to the discussion topic. Peer reviews should be returned within 3 to 5 days (please!)





Note: Students need only to turn in a project to qualify for a passing grade. The only purpose of the evaluation is for everyone to see the different ways that the project can be done.