Retail company - Creation, load and analysis - MSSQL

Analysis of a retail company transaction dataset by MS SQL database.

In the description Microsoft SQL Server Management Studio (SSMS) version 2022 was used to create dataset and to analyse data.

  1. Checking server and database properties

First of all, to avoid discrepancies it is better to adjust the character set (collation) of the server to match to the collation of the data. 
-- server COLLATION property
SELECT SERVERPROPERTY('collation');
GO

The reply can be for example: SQL_Latin1_General_CP1_CI_AS.

The database was named Questions and Answers, shortly QADB. 

USE QADB;
GO

SELECT DATABASEPROPERTYEX(DB_NAME(), 'collation');
SELECT DB_NAME(); 
GO
The latter SQL query can be used only when the database has been created. The best if the database collation matches with the loaded data and even better if it matches with the database server collation, leading to fewer errors in special character recognition and consequently during the data analysis/display.

Actually, to create a new database first change/define the context of the current database to the master database. The master database is a system database that contains crucial information about the SQL Server instance, such as system configuration settings, login accounts, and the metadata about all other databases within the instance. For safety reasons of a future complete restart of SQL DB and Query building processes a drop function is inserted which is followed by the real database creation:

USE master;
DROP DATABASE IF EXISTS QADB;
GO

-- creating new database (QADB)
CREATE DATABASE QADB;
GO

In the case the entire database collation has to be changed use: 
ALTER DATABASE QADB
COLLATE SQL_Latin1_General_CP1_CI_AS;
In another case, when a field of the database collation has to be changed, for example textual entries with special characters for proper display, use: 
ALTER TABLE QADB
ALTER COLUMN city nvarchar(100)
COLLATE SQL_Latin1_General_CP1_CI_AS
NOT NULL;
COLLATE SQL_Latin1_General_CP1_CI_AS;
Create user (at master db level) with specified access to the (QA) database:
CREATE LOGIN NewUser WITH PASSWORD = 'SecurePassword';

This can be set later as well, but for simple analysis it is not required as by creating the database you immediately get access to it as default user. 

Start using the created database by switching from master DB to QADB
USE QADB;
  GO
CREATE SCHEMA KDB 
AUTHORIZATION dbo;
GO
The second part orders the database server to create Master Database file (mdf) and transactions log file (ldf). There are certain restrictions and some details to follow to keep the database and the server always at their best performance; see the links. 
  1. Creating tables and fields with constraints
Create the Tables of the database with realistic and logical limitations (constraints 1, 2), such as product_price cannot be negative, product_name should not be longer than 50 characters, but can be shorter, which can be defined as variable character length: nvarchar
DROP TABLE IF EXISTS KDB.Product; -- safety drop on restart
CREATE TABLE KDB.Product (
	productID IDENTITY(1000000000000,1) NOT NULL
		CONSTRAINT PRIMARY KEY,
	product_name nvarchar(50) NOT NULL,
    	product_price int DEFAULT 0 NOT NULL
    	CHECK (product_price >= 1)
);
Constraints CHECK is a limitation that does not allow (from creation with the defined limitation) to enter such values which do not match with the declared condition. 
Create another Table to store Warehouse related information such as the quantity of available products. Being a new table it requires an identity (primary key*) field as well: stock ID:
CREATE TABLE KDB.ProductRegister (
product_stockID char(13) NOT NULL, -- IDENTITY(1,1) 
product_quantity INT NOT NULL 
	CHECK (product_quantity >= 1)
);
The above defined product_stockID in ProductRegister Table is related to the productID in 1:1 relationship, so these feilds has to be connected in a way that the product_stockID besides being an ID in its own table, requires a foreign key: the productID:
ALTER TABLE KDB.ProductRegister
ADD CONSTRAINT PK_ProductRegister_product_stockID PRIMARY KEY (product_stockID),
	CONSTRAINT FK_ProductRegister_productID FOREIGN KEY (product_stockID)
	REFERENCES KDB.Product(productID);
GO
Primary keys and Foreign keys may be named as you like but it is optimal if you define a descriptive name, including its nature "PK_" or "FK_", the name of the field in the current table and in the case of foreign key add the name of the referred primary key in the other table, as well.
A business always have clients and it is important to store the related information in the database:
CREATE TABLE KDB.Customers (
customerID INT NOT NULL IDENTITY(1,1), 
customer_name nvarchar(50) NOT NULL,
phonenumber varchar(15),
address varchar(100) -- this could be split into parts if required
);

ALTER TABLE KDB.Customers
	ADD CONSTRAINT PK_Customers_customerID PRIMARY KEY (customerID);
GO
If Warehouse, Product, and Customer information have their fields defined then only the purchases remain to be registered. There are simple and complex ways to create the required structure, here I made a medium-level complexity solution. We store the purchase (invoice/bill/transaction) main data in one table (SalesHead) and the exact items (codes) and related information are stored in a separate, but connected table Sales
DROP TABLE IF EXISTS KDB.SalesHead; -- for safety restart reasons
CREATE TABLE KDB.SalesHead (
customerID INT NOT NULL
	CONSTRAINT FK_Customers_SalesHead_customerID FOREIGN KEY (customerID)
	REFERENCES KDB.Customers (customerID),
transactionID INT IDENTITY(1,1) NOT NULL 
	CONSTRAINT PK_SalesHead_transactionID PRIMARY KEY (transactionID),
totalamount INT NOT NULL,
	CHECK (totalamount >= 1)
t_date SMALLDATETIME NOT NULL 
	);

ALTER TABLE KDB.SalesHead
	ADD CONSTRAINT UQ_SalesHead_salesCustomerID UNIQUE (customerID, transactionID) 
    totalamount;
GO
customerID, and transactionID as a combination create such a value pair which is able (as a value pair) to identify the transaction, but those were not chosen as a composite (combined) primary key* because for simple reasons it is better to create an ID for all events separately: transactionID.
The unique nature of the identity (typically a large number or a long alphanumerical value) is provided by the server, as on every new entry the identity value 'increases'. 
DROP TABLE IF EXISTS KDB.Sales;
CREATE TABLE KDB.sales (
salesID INT NOT NULL UNIQUE,
	CONSTRAINT FK_SalesHeadtransactionID_SalesID FOREIGN KEY (salesID)
	REFERENCES KDB.SalesHead(transactionID),
itemnumber INT NOT NULL IDENTITY(1,1)
		CONSTRAINT PK_SalesItemNumID PRIMARY KEY (itemnumber),
sold_product INT NOT NULL,
sold_quantity INT NOT NULL,
	CHECK (sold_quantity >= 1)
);

* Primary key: One or more fields (composite primary key) whose values uniquely identify each record in a table. A primary key cannot allow Null values and must always have a unique index. A primary key is used to relate a table to foreign keys in other tables.

Next steps...

Start of a "data collection and processing" run.

It has been decided that we need to find or collect thousands of files including 1-20 line of information crucial for manufacturing of goods at my workplace. Our firm produces wire harnesses (cables and wires tailored at given lengths, equipped with different type of connectors or plugs) that the Customers ask for. The required information which includes cable/wire type, length, end modifications such as contacts (ferrules, terminals, slide-shoes, connectors with or without housing,...) is defined by the Customer and consequently they provide these information to us in a form of a bill of material (BOM, in format of excel) file and the drawing of the final product (in format of pdf made by some computer aided drawing software abbreviated as CAD software). We have to follow these two definition files but the order of manufacturing steps how we proceed is left to us to plan and customise for each product.
So thousands of data related to over 1500 product is planned to be found and organized in a way that we may use it for loading into a new (unified) Enterprise Resource Planning (ERP) system. The data will serve as we call as production resource list (otherwise recipe), which frames basically all necessary information for the production, such as production processes along with the used raw materials and parts, the machines and tools that we apply in addition to the above mentioned BOM data.
There was a Administrative Management System used that was used almost since the foundation of the firm, ages back in time. This software was able to create BOMs for the final products, so that Warehouse could keep quantity of parts and raw materials up-to-date. Of course the materials and parts were not well named and the identification (item number) was also not carefully and precisely defined, but the administrators could deal with it because they got used to the system and naming and codes of the items in the form as they were input.
On the other hand we have faced another challenge. In spite of the fact that ISO9001 Quality Management System was already in work at this time and consequently the required (BOM and CAD) files were saved in a well defined folder structure so resources of data could be easily located (and used meanwhile production), we knew that a large part of BOMs were received from the Customers in "print screen" version, so as images. An excel sheet or a csv format is easy to extract but images require optical character recognition (OCR) softwares. So either 
  • we set up an OCR system that is able to extract data without error (or at low error level) from the images, 
  • or we extract them manually (by reading it)
  • or we ask the Customer to send those BOMs in extractable file format
and all have to do in mass as more than 200 products were involved in this question. A reasonable mixture of above mentioned options would also be acceptable taking into account the benefit of each and comparig with the time that we have to invest in the method.

This was the point where I started to think about a software based (semi)automated system which could be able to go through a list of products (item codes), find and process the data of the connected BOMs and other related files and give back an importable production resource list(s) as it is defined by the new ERP system. The requested content in the predefined data structure.

Footnote: I know by experience that all above mentioned options require time and precision. I will check all options as thouroughly as I can. I am aware of the well known fact that when the missing data will be available still we will have the long run of extract-transfer-load (ETL) process ahead us. I presume at least a year for the project taking into account the human resources that we have for the project.

Tools & skills

Python Tools&skills: Python

Tools&skills: Python modules, Matplotlib Tools&skills: Python modules, Pandas
Tools&skills: Python Tools&skills: Python, scikit learn 

Tools&skills: Python modules, Jupyter Notebook Tools&skills: Python modules, Jupyter Lab

SQL

Tools&skills: SQL, PostgreSQL Tools&Skills: SQL, PostgreSQL

Tools&skills: Microsoft Microsoft system

 Tools&Skills: Microsoft software set Tools&Skills: Microsoft software set

IDEs (Python, Java, PHP)

 Tools&Skills: coding IDEsTools&Skills: coding IDEs Tools&Skills: coding IDEs Tools&Skills: coding IDEs 

Servers

Tools&Skills: Servers Tools&Skills: Servers Tools&Skills: Servers

Data presentation

Tools&skills: Data, Matplotlib Tools&skills: Data 
Tools&skills: Data, MS Power BI Tools&skills: Data, Tableau

Text editors & Presentation

Tools&Skills: Text&Pres  Tools&Skills: Text&PresTools&Skills: Text&Pres

Cloud&Warehouse

Tools&Skills: Cloud&Warehouse, Snowflake Tools&Skills: Cloud&Warehouse, IBM Cloud

AI&ML

 Tools&Skills: AI&ML, Mindee Tools&skills: Python, scikit learn
Tools&skills: AI&ML

Other

           

Course related skills

JDS Academy - 6 week Data science course tools

Tools&skills: Courses DataCamp  

Last updated: November 2024.

Courses & certificates

2024 GeeksForGeeks - Python coding exercises/practices, continuous, GFG website

2021-2024, BSc. in Engineering Computer Sciences

2020 Google - Google Analytics For Beginners course, certificate (2020)

discontinued, Google Analytics 4 has been introduced

2020 JDS Academy - Junior Data Scientist academy, certificate (2020)

2020 Coursera - Machine learning with Python (IBM), certificate (2020)

2019 UdemyA-Z Real Life Data Science Exercises, certificate (2019)


first published 2-Feb-2020
last updated 6-July-2024


Java - Object Oriented programming - utillinks

 A collection of websites about object-oriented programming & Java

Available in english: OOP (ENG)Objektumorientált programozás (HUN)  - Wikis

Hungarian texts:

Osztályok és objektumok - Mikó Csaba (szerintem közérthető és könnyed stílusú!)

Objektum Orientált Programozás Java nyelven - Sallai András (SZIT)

Objektumok és osztályok - SZTE TTIK oktatási anyag

Objektum orientált szoftverfejlesztés - Kondorosi Károly, Szirmay-Kalos László, László Zoltán


Java in general

Java installation (Oracle website), you may select JVM, JRE or JDK

Programozás gyakorlatok, SZTE - Antal, SZTE

Oracle Java Dokumentation

Nem Java, de nagyon hasznos OO témakörben:

If someone is already motivated enough, you can also take the exam:

Preparation for Java Programmer Language Certification - Oracle, tutorial pages

Java nyelv középiskolásoknak (HUN)


UML, an additional program planning or documenting tool

Universal Modelling (or Markup) Language

UML Wiki (ENG)

In case we would like to create class, object, or operation diagrams:

UML -Sallai András (SZIT)

Rendszertervezés 4., A rendszerfejlesztés eszközei (technikák, CASE, UML) - Dr. Szepesné Stiftinger Mária (2010), NYME

UML tools, among which JDeveloper (Oracle) compatible with Netbeans developing environment (IDE) is also available here a hObjectByDesign - UMLTools, long list.


Apps/Software/Websites for learning Java

Free online Java programming interface, if we don't want to install software on our machine... it obviously has limitations.

Not only for Java, but this application is for learning AI, as well:

SoloLearn (partly free, Android, GooglePlay)

Have fun!


Java, objektum orientált programozás

 Az objektum orientált programozásról szóló weboldalak gyűjteménye (Java)

Objektumorientált programozás - Wiki

Osztályok és objektumok - Mikó Csaba (szerintem közérthető és könnyed stílusú!)

Objektum Orientált Programozás Java nyelven - Sallai András (SZIT)

Objektumok és osztályok - SZTE TTIK oktatási anyag

Objektum orientált szoftverfejlesztés - Kondorosi Károly, Szirmay-Kalos László, László Zoltán


Java általában

Java install (Oracle website), kell JVM, JRE, JDK

Programozás gyakorlatok, SZTE - Antal, SZTE

Oracle Java Dokumentáció úgy általában

Nem Java, de nagyon hasznos OO témakörben:

Objektum-orientált programozás C++ nyelven - szerzők

és ha már valaki eléggé belelendült, akkor lehet vizsgázni is:

Preparation for Java Programmer Language Certification - Oracle, tutorial pages

Java nyelv középiskolásoknak

https://regi.tankonyvtar.hu/hu/tartalom/tkt/javat-tanitok-javat/ch02.html


Kevésbé hasznos:

Java Programozási alapok  - GDF oktatási anyag, Szalai Zsolt - sajnos a videóanyag már nem működik


UML

Universal Modelling (vagy Markup) Language

UML-ről, ha osztály, objektum, vagy műveleti diagrammokat akarunk létrehozni:

UML -Sallai András (SZIT)

Rendszertervezés 4., A rendszerfejlesztés eszközei (technikák, CASE, UML) - Dr. Szepesné Stiftinger Mária (2010), NYME

UML segédeszközök, melyek közül a Netbeansbe épülő JDeveloper (Oracle) is elérhető ezen a hosszú listán keresztül.


Applikációk/szoftverek/weboldalak Java tanuláshoz

(Ingyenes) online Java programozó felület, ha nem akarunk szoftvert telepíteni a gépünkre... nyilván vannak limitációi.

Nem csak Java-hoz, applikáció tanuláshoz:

SoloLearn (részben ingyenes, Android, GooglePlay)


Végül https://java.lap.hu/ nem merem belinkelni, mert ez a régi idők gyűjtőoldala. Keress inkább Google-lel, ha kell még valami!


Snowflake universe, part #6 - Forecasting2

Forecasting with built-in ML module Further posts in  Snowflake  topic SnowFlake universe, part#1 SnowFlake, part#2 SnowPark Notebook Snow...