Managing Storage

Describe Storage Technologies

Filesystem - method of storing and organizing computer files and the data they
contain to make it easy to find and access this information. A file system also maintains
the physical location of the files so that you can find and access the files in the future

A Filesystem provides:

- a way to store data about these files — for example, their filenames,
permissions, and other attributes.

- an index — a list of the files on the drive and where they’re located on the
drive, so the operating system can see what’s on the drive in one place rather
than combing through the entire drive to find a file.

- has a Master file table(index) contains all information about the file: size,
allocation, name etc

- examples of filesystem: NTFS local filesystem for Windows, EXT3 local for
Linux, network filesystem NFS

There are 3 main types of storage technologies:

1. File based storage - a file is typically structured in a file system, which is nothing
more than a hierarchical way of organizing files so that an individual file can be
located by describing the path to that file. We know that certain attributes --
information that might describe a file and its contents, such as its owner, who can
access the file, and its size -- are conveniently stored as metadata in a file
system.

2. Block storage - a block is a chunk of data, and when appropriate blocks are
combined, it creates a file. A block has an address, and the application retrieves
a block by making a SCSI call to that address. It is a very microscopic way of
controlling storage. Unlike in the case of NAS, the application decides where to
place the data and how to organize the storage. How the blocks are combined or
accessed is left up to the application.

With block storage, files are split into evenly sized blocks of data, each with its
own address but with no additional information (metadata) to provide more
context for what that block of data is.

Under the right circumstances, granting this level of granular control to the
application allows it to extract the best performance from a given storage array.
This is the reason why block storage has been king of the hill for performance-
centric applications, mostly transactional and database-oriented

1
 3. Object based storage - is a storage architecture that manages data as objects,
as opposed to other storage architectures like file systems which manage data
as a file hierarchy and block storage which manages data as blocks within
sectors and tracks.

- abstracts file and block and stores and has a flat namespace of objects

1. objects are kept inside a single repository, and are not nested as files inside a
folder inside other folders

2. object storage keeps the blocks of data that make up a file together and adds
all of its associated metadata to that file.

3. the totality of the data, be it a document, audio or video file, image or photo,
or other unstructured data, is stored as a single object

4. Metadata is also associated with the object and provides descriptive

information about the object and the data itself. This eliminates the need for a
hierarchical structure and simplifies access by placing everything in a flat
address space (or single namespace).

Instead of organizing files in a directory hierarchy, object storage systems store files in a
flat organization of containers (called "buckets" in Amazon S3) and use unique IDs
(called "keys" in S3) to retrieve them.

2
Extra references:
https://cloudian.com/blog/object-storage-vs-file-storage/

3
Physical types of storage:

Direct-attached storage - refers to a computer storage system that is directly attached to

your server or PC instead of being attached directly to a network. Direct-attached
storage is not limited only to internal storage. It can also utilize an external disk
enclosure that contains hard disk drives, including just-a-bunch-of-disks (JBOD)
enclosures and enclosures that are part of a redundant array of independent disks
(RAID).
The important point about direct-attached storage is that it does not use a storage
network. Although a common example of direct-attached storage is the internal hard
drive in your computer, more commonly it refers to a dedicated storage array that is
attached directly to your servers.

The main interfaces and protocols that are used for direct-attached storage connections
are Serial ATA (SATA), external Serial Advanced Technology Attachment (eSATA),
serial attached SCSI (SAS), and Fibre Channel.

JBOD - is an abbreviation for “just-a-bunch-of-disks." In other words, the disks are

presented to a computer as if they were directly attached internal disks, and they are
used to increase the direct-attached storage capacity of file servers. JBODs are usually
relatively simple external enclosures, connected to one or more servers through SAS or
eSATA connections.

4
Network-attached storage - devices connect storage to a network where they are
accessed through file shares. Unlike direct-attached storage, they are not directly
attached to the computer.

Network-attached storage devices support Ethernet connection, and they typically allow
an administrator to manage disk space, set disk quotas, provide security, and utilize
snapshot technologies. Network-attached storage devices support multiple protocols.
These include network attached file systems, Common Internet File Systems (CIFS),
and Server Message Block (SMB).

Storage Area Network (SAN) - is a dedicated network that allows you to share storage.
A SAN consists of a storage device, the interconnecting network infrastructure (switches
and so on), and servers that are connected to this network. SAN devices provide
continuous and fast access to large amounts of data. The communication and data
transfer mechanism for a given deployment is commonly known as a storage fabric.
A SAN uses a separate network, and it is generally not accessible by other devices
through the local area network.

A SAN does not provide file abstraction, only block-level operations. The most common SAN
protocols used are iSCSI, Fiber Channel, and Fiber Channel over Ethernet (FCOE).

5
 HBA (Host Bus Adapter) - is most often used to refer to a Fibre Channel interface card
and iSCSI HBA for iSCSI SAN (offload TCP/IP and iSCSI processing from a server's CPU)

Understanding Disks Structure in Windows

MBR vs GPT partitioning scheme

Master Boot Record (MBR) - is legacy partitioning scheme that is used by most
operating systems:

1. the MBR itself resides entirely on the first sector (512 bytes) of a hard disk.

2. Of that space, the first 440 bytes of the MBR are devoted to the boot loader.

6
 3. the BIOS reads this code and executes it when the computer boots.

4. followed by data space used to store information about the four Primary
partitions

5. permits us to address up to 2TB (related to how each partition is described

using 32 bit disk pointers)

On an MBR disk, there can only be:

6. primary partitions -4
Or
7. primary partitions -3 plus 1 extended partition

Extended Partition is just a defined area where logical drives reside. In order to create
more than 4 partitions, extended partition is introduced. Logical Drive is the hard disk
partition created in extended partition.

GUID Partition Table (GPT) - is a new standard for the layout of the partition table:

8. part of the Extensible Firmware Interface (EFI) standard proposed by Intel as

a replacement for the PC BIOS

9. uses the 64 bit disk pointers, which allow for a maximum disk partition size of
9.4 Zeta bytes, or 9.4 billion Tera Bytes.

10. from Windows Server 2008 onwards, only x64 versions ** of the Microsoft OS
support GPT disks as the boot disk in EFI (Extensible Firmware Interface)
boot mode

11. GPT disks are not supported as the boot disk in BIOS Mode when booting to
Windows OSes.

** The difference between 32 bit OS and 64 bit OS:

12. The 32 bit system can point to or address only 4 GB memory
13. the 64 bit system can point to or address a larger number of locations in
physical memory

7
8
You can convert a disk between MBR and GPT as long as the disk is completely empty.
Unfortunately, once you've created any partitions or volumes on the disk, you can no
longer convert between the two types.

Adding new local disks in Windows

Tools for disk/partitions management: GUI Disk Management (diskmgmt.msc or in

Computer Management console) and diskpart tool from cmd.

9
To list partitions from a disk:

To check details about a partition :

Create partition and format it with NTFS:

10
1. you need to physically install or add a new virtual disk if machine is virtual
2. Windows automatically** recognizes the new hardware and makes it
available(not ready for use yet, you just can see it as an offline disk):
o If the disk is a basic disk that is already partitioned and formatted, you
can use it without initializing, but it will initially appear "offline" in Disk
Management.
o If it's a brand-new disk that has never been partitioned or formatted,
you need to initialize it first:

** if the new disk is not visible in Disk Mgmt. do a Rescan disks:

11
3. If disk is new, initialize the disk and choose MBR or GPT partition table as
necessary:

4. Create partitions/volumes, choosing the size of it, the mount point drive letter,
the filesystem (most cases is NTFS, the other ones are obsolete and not
used in enterprise environments anymore) and Quick format:

12
5. Once the formatting finished the new partition should be available in Windows
Explorer to upload data on it.

13
SAN disks (iSCSI storage) in Windows

iSCSI is an IP based storage networking standard for linking storage devices with
servers.

By carrying SCSI commands over TCP/IP networks, iSCSI can facilitate data transfers
over local area networks (LANs), wide area networks (WANs), or the Internet. Through
iSCSI, the space on storage server will be regarded as local disks by client's operation
system. In reality, all data transferred to the disk are actually transferred over network to
the storage server.

- the only interconnectivity required between servers and the disks on storage
subsystems is an IP network (this allows many servers to share the same storage
subsystem)
- is block based storage and the storage vendors offer SAN with iSCSI technologies
(EMC VNXe, Netapp iSCSI SAN etc)
- there are NAS systems with iSCSI support as well
- starting with Win 2008 R2 *** iSCSI server support was introduced (you can create
an iSCSI server/target with Win 2008 R2 )

*** Win 2008 R2 is not natively installed the iSCSI target software, needs to be
downloaded from Microsoft and installed

iSCSI target server - is the server where iSCSI target service is running.

Microsoft Internet iSCSI Initiator - enables you to connect a host computer that is
running Windows Server® 2008 R2 or higher to an external iSCSI-based storage array
through an Ethernet network adapter (any kind of iSCSI storage not only Microsoft).
iSCSI initiator service runs on the client and used to make a connection to the iSCSI
Target by logging on to a Target server.

iSCSI virtual disks - are created on iSCSI target server and associated to the iSCSI
target. iSCSI virtual disk represents an iSCSI LUN, which are connect to the clients
using iSCSI initiator.

IQN: It is a unique identifier of the Target or Initiator. The Target IQN is shown when it is
created on the Server. The initiator IQN can be found by typing a simple “iscsicli” cmd in
the command window.

Logical unit numbers (usually referred to as LUNs) allow a SAN to break its storage
down into manageable pieces, which are then assigned to one or more servers in the
14
SAN. It is a logical reference that can comprise a disk, a section of a disk, a whole disk
array, or a section of a disk array.

Install iSCSI server/target on Windows 2012 (not used in enterprise environments)

1. Add iSCSI target role:

15
2. Add virtual disk to iSCSI server and specify the size of the virtual disk that
you want:

3. Create a New iSCSI Target, give it a name and description and select the
iSCSI initiators *** (clients) that will have access to this disk (this will allow the
Target to be discovered by defined list of initiators):

*** Initiator has to be started on the client side and have an iqn

16
4. Optional CHAP authentication can be configured for this disk

5. On the client side, open iSCSI initiator, provide the iSCSI Target server IP or
host name to the initiator, and the initiator will be able to do a discovery of the
iSCSI Target:

17
18
 6. Once the connection is established, the iSCSI virtual disk will be presented to
the initiator as a disk. By default, this disk will be offline.

7. The rest of the steps are the same as for a local disk

- network adapters should be dedicated to network communication iSCSI and should

be GigEthernet or 10 Gig adapters

- for iSCSI disks in Windows, you cannot use teamed network adapters, because they
are not supported with iSCSI (because you have multipath solution based on
Microsoft Multipath I/O (MPIO) which Windows Server includes one or more DSMs
as part of the operating system and also your hardware vendor will usually supply
an MPIO device-specific module (DSM) for your hardware)

- Jumbo frames of 9000 bytes are recommended to be activated end to end on the
iSCSI network

Extra references
https://blogs.technet.microsoft.com/filecab/2012/05/21/introduction-of-iscsi-target-in-
windows-server-2012/

Resizing partitions/volumes

Extend existing volume in Windows

You can add space to a volume without having to back up or reboot, if the volume is a
simple volume and there is free space at the end of the volume (including the boot
volume)!!!

1. on the volume choose Extend Volume:

19
 2. choose the amount to extend with:

3. refresh in Win Explorer to see the new size of the volume

Shrink a volume that has extra free space(local disks only)

1. select the volume to shrink

2. enter the size to shrink with

20
Check disk tool: chkdsk
- scans the volumes for filesystem errors and physical errors like bad sectors
- depending on the size of your volume it can take from 5 min to couple of
hours
- can be started from GUI or cmd:

A simple chkdsk Drive_letter: will only check the error but won’t do any corrections. If
chkdsk /f Drive_letter is used the volume will be dismounted (if is the OS partition will
21
request you to schedule at next reboot as the OS partition cannot be dismounted with
disrupting the OS) and errors corrected.

There is an error event logged in System event log when a volume needs a chkdsk,
otherwise the filesystem may become corrupted:

Memory dump

A memory dump - consists of the recorded state of the working memory of an

Operating System (applications can create dumps also when they crash) at a specific
time, generally when the program has crashed or otherwise terminated abnormally.

- a system crash (bugcheck) is usually caused by kernel-mode code and not

user-mode processes
- memory dump files can be analyzed by WinDbg tool
- there is two types:
o Kernel Memory Dump – stop code, lists all kernel running processes,
loaded drivers and threads at that time
o Small memory dump – less details compared with the above full dump

- Page file has to be set up on the boot volume and has to be the size of the
physical memory installed on the server, otherwise the Kernel Dump file
might not be generated when a crash happens

22
 Page file
Windows memory management is based on Virtual Memory, where each process has its
own private virtual address space. Windows will move the least used memory pages to a
hidden file called the page file, when approaching a low memory condition.

The Page file is a special file used by Windows to store modified pages, and the process
of moving pages from RAM to the Page file is called "Paging".

Page files have two primary roles:

- Physical extensions of RAM that store modified data
- Record in information about the state of the system in case of a system
crashes

Extra references
Understanding Crash Dump Files:
https://blogs.technet.microsoft.com/askperf/2008/01/08/understanding-crash-dump-files/

How to crash dump a Windows VM:

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&ext
ernalId=1001624
23
File sharing/permissions, map/unmap a share

- NTSF is the preferred filesystem on Windows servers

- In NTFS when a file/folder is created an Access Control List (ACL) is created

that includes information that controls which users and groups can access the
file or folder, as well as what type of access is granted to particular users and
groups

There are two types of permissions used in NTFS:

1. Explicit permissions: Permissions granted directly to a file or folder.

2. Inherited permissions: Permissions that are granted to a folder (parent object

or container) that flow into child objects (subfolders or files inside the parent
folder).

24
Looking at Effective NTFS Permissions of a user/group for a file/folder:

25
The owner of a file/folder controls how permissions are set on the object and to whom
permissions are granted. If for some reason you have been denied access to a file or
folder, you need to reset the permissions by taking ownership of the file or folder and
modifying the permissions.

Encrypting File System (EFS) is a core file encryption technology used to store
encrypted files on NTFS file system volumes. Encrypted files cannot be used unless a
user has access to the keys required to decrypt the information.

After a file has been encrypted, you do not have to manually decrypt that file before you
can use it. Rather, once you encrypt a file or folder, you can work with that file or folder
just as you would with any other file or folder.

EFS can be used to encrypt individual folders or files and BitLocker can be used to
encrypt entire volumes.

Sharing files/folders – for files to be access over network they need to be Shared and
assigned Share permissions along with NTFS permissions.

CIFS and SMB are Windows file sharing protocols (CIFS being the latest version of
SMB).

NFS is traditionally a Unix file sharing protocol but now Windows Server supports it
natively.

SMB/CIFS uses Windows-style access control lists, whereas NFS uses Unix-style file
permissions (User ID owner, Group ID owner, and read/write/execute permissions).

26
Do not grant Everyone Share permissions as it can turn into a vulnerability, but use
specific users/groups to grant Share permissions!!!

How you list all shared folders active on a server/client:

27
How to map a shared folder from a remote server/client, once Share and NTFS
permissions have been granted:

Mapp a network drive from cmd:

List the mapped drive in My Computer:

28
To list mapped network drives from cmd:

An administrative share is a shared folder typically used for administrative purposes

and usually hidden. To make any shared folder or drive hidden, the share name must
have a $ at the end of it.

You have to have Administrator rights on the remote server to be able to access the $
shares:

Copying/moving/deleting files/folders from cmd:

Copy files: copy /robocopy

29
Robocopy – the newest tool available by default from Win Vista upwards

- /e parameter tells robocopy to include all subdirectories. This includes empty

folders.
- /z parameter give you the ability to restart the process in case the connection is
lost to the destination
- robocopy will automatically copy hidden and system files. It will create new
directories if they don't exist at the target location.
- gives you a detailed statistic and possibility to create a log file
- /mir it can mirror a folder and its content and will preserve the permissions

30
 Delete files/folders

rmdir – removes directory

del – delete files

Volume Shadow Copies - allows to restore previous states of the entire volume, you can’t
restore previous states of single files and/or folders.

Select the volume and click Enable:

31
32
Microsoft suggests to use a dedicated drive to store Volume Shadow Copies in case of high-
load.

A first snapshot will be generated. Default VSS settings work as following:

- Volume Shadow Copies will be stored in the same volume

- Volume Shadow Copies will take a maximum amount of 10% of the local disk space
- The system reserves a minimum of 300MB of disk space for the shadow copies
- The system schedules two shadow copies per day (7.00 AM and 12.00 PM)

To modify these settings click Settings:

33
To restore a previous snapshot just select it and click Revert:

34
Once Shadow Copies enabled for a drive, contents of folders and subfolders can be restored as
well to a previous version:

35
Enable Quota on a Volume

- Quotas are a feature of the NTFS disk file system

- Quotas allow Windows administrators to monitor and control the use of disk space by
users
- are configured on a per-volume basis
- Quotas apply to all users of the volume unless exceptions are configured for individual
users. For example, a user in the video editing department would probably need more
disk space than someone in the accounts department.
- Members of the Administrators group can never be denied access to disk space (the
most an administrator will ever receive is a warning)

36
Enable Quota on file shares:

- is part of the File Server role

- can set quotas for specific folders on the volume, we can use templates to ensure
consistent application of quotas
- can set quotas which are automatically applied to all sub-folders of a folder.

There are two kinds of quota available:

- hard quotas which set a limit and enforce it
- soft quotas which set a limit but only report on it.

Soft quotas are useful for monitoring disk space use. Quotas are commonly applied using
quota templates, which are a mechanism for easily applying the same quota settings to one or
more folders.

37
Extra references:
Configure folders quota and templates
https://www.red-gate.com/simple-talk/sysadmin/exchange/implementing-windows-server-2008-
file-system-quotas/

38