Добавлена папка source в CristalDiskMark

This commit is contained in:
2026-05-29 13:04:54 +07:00
commit bdc2295ee4
240 changed files with 94035 additions and 0 deletions
@@ -0,0 +1,47 @@
## VM Fleet ##
These are the historical release comments for VM Fleet 1.0.
VM Fleet 0.9 10/2017 (minor)
* watch-cpu: now provides total normalized cpu utility (accounting for turbo/speedstep)
* sweep-cputarget: now provides average CSV FS read/write latency in the csv
VM Fleet 0.8 6/2017
* get-cluspc: add SMB Client/Server and SMB Direct (not defaulted in Storage group yet)
* test-clusterhealth: flush output pipeline for Debug-StorageSubsystem output
* watch-cluster: restart immediately if all child jobs are no longer running
* watch-cpu: new, visualizer for CPU core utilization distributions
VM Fleet 0.7 3/2017
* create/destroy-vmfleet & update-csv: don't rely on the csv name containing the friendlyname of the vd
* create-vmfleet: err if basevhd inaccessible
* create-vmfleet: simplify call-throughs using $using: syntax
* create-vmfleet: change vhdx layout to match scvmm behavior of seperate directory per VM (important for ReFS MRV)
* create-vmfleet: use A1 VM size by default (1VCPU 1.75GiB RAM)
* start-vmfleet: try starting "failed" vms, usually works
* set-vmfleet: add support for -SizeSpec <Azure Size Specs> for A/D/D2v1 & v2 size specification, for ease of reconfig
* stop-vmfleet: pass in full namelist to allow best-case internal parallelization of shutdown
* sweep-cputarget: use %Processor Performance to rescale utilization and account for Turbo effects
* test-clusterhealth: support cleaning out dumps/triage material to simplify ongoing monitoring (assume they're already gathered/etc.)
* test-clusterhealth: additional triage output for storport unresponsive device events
* test-clusterhealth: additional triage comments on SMB client connectivity events
* test-clusterhealth: new test for Mellanox CX3/CX4 error counters that diagnose fabric issues (bad cable/transceiver/roce specifics/etc.)
* get-log: new triage log gatherer for all hv/clustering/smb event channels
* get-cluspc: new cross-cluster performance counter gatherer
* remove run-<>.ps1 scripts that were replaced with run-demo-<>.ps1
* check-outlier: EXPERIMENTAL way to ferret out outlier devices in the cluster, using average sampled latency
VM Fleet 0.6 7/18/2016
* CPU Target Sweep: a sweep script using StorageQoS and a linear CPU/IOPS model to build an empirical sweep of IOPS as a function of CPU, initially for the three classic small IOPS mixes (100r, 90:10 and 70:30 4K). Includes an analysis script which provides the linear model for each off of the results.
* Update sweep mechanics which allow generalized specification of DISKSPD sweep parameters and host performance counter capture.
* install-vmfleet to automate placement after CSV/VD structure is in place (add path, create dirs, copyin, pause)
* add non-linearity detection to analyze-cputarget
* get-linfit is now a utility script (produces objects describing fits)
* all flag files (pause/go/done) pushed down to control\flag directory
* demo scripting works again and autofills vm/node counts
* watch-cluster handles downed/recovered nodes gracefully
* update-csv now handles node names which are logical prefixes of another (node1, node10)
@@ -0,0 +1,79 @@
param(
[string] $csvfile = $(throw "please provide the path to a cputarget sweep result file"),
[switch] $zerointercept = $false,
[int] $sigfigs = 5
)
function get-sigfigs(
[decimal]$value,
[int]$sigfigs
)
{
$log = [math]::Ceiling([math]::log10([math]::abs($value)))
$decimalpt = $sigfigs - $log
# if all sigfigs are above the decimal point, round off to
# appropriate power of 10
if ($decimalpt -lt 0) {
$pow = [math]::Abs($decimalpt)
$decimalpt = 0
$value = [math]::Round($value/[math]::Pow(10,$pow))*[math]::pow(10,$pow)
}
"{0:F$($decimalpt)}" -f $value
}
write-host -ForegroundColor Green CPU Target Sweep Report`n
write-host -ForegroundColor Green The following equations and coefficients are the linear
write-host -ForegroundColor Green fit to the measured results at the given write ratios.
write-host -ForegroundColor Cyan ("-"*20)
write-host -ForegroundColor Yellow NOTE: take care that these formula are only used to reason about
write-host -ForegroundColor Yellow " " the region where these values are in a linear relationship.
write-host -ForegroundColor Yellow " In particular, at high AVCPU the system may be saturated."
write-host -ForegroundColor Yellow "Use R^2 (coefficient of determination) as a quality check for the fit."
write-host -ForegroundColor Yellow "Values close to 100% mean that the data is indeed linear. If R2 is"
write-host -ForegroundColor Yellow "significantly less than 100%, a closer look at system behavior may"
write-host -ForegroundColor Yellow "be required."
if ($zerointercept) {
write-host -ForegroundColor Red NOTE: forcing to a "(AVCPU=0,IOPS=0)" intercept may introduce error
} else {
write-host -ForegroundColor Red "NOTE: with a non-zero constant coefficient, care should be used at`nlow AVCPU that the result is meaningful"
}
# do the check fit of QoS to IOPS
# this will let us check for non-CPU limited saturation (poor r2 is a giveaway)
# we can have an excellent CPU->IOPS fit but not actually have been able to stress in much CPU
# and have lots of repeated measurements as we tried to step up QoS
$h = @{}
get-linfit -csvfile $csvfile -xcol QOS -ycol IOPS -idxcol WriteRatio -zerointercept:$zerointercept |% {
$h[$_.Key] = $_
}
# now fit IOPS to Average CPU
get-linfit -csvfile $csvfile -xcol AVCPU -ycol IOPS -idxcol WriteRatio -zerointercept:$zerointercept | sort -Property Key |% {
write-host -ForegroundColor Cyan ("-"*20)
write-host $_.Key
if ($zerointercept) {
write-host ("{0} = {1}({2})" -f $_.Y,$(get-sigfigs $_.B $sigfigs),$_.X)
} else {
if ($_.A -ge 0) {
$sign = "+"
} else {
$sign = ""
}
write-host ("{0} = {1}({2}){4}{3}" -f $_.Y,$(get-sigfigs $_.B $sigfigs),$_.X,$(get-sigfigs $_.A $sigfigs),$sign)
}
write-host ("N = {1}`nR^2 goodness of fit {0:P2}" -f $_.R2,$_.N)
if ($h[$_.Key].R2 -le 0.5) {
write-host -ForegroundColor Yellow "WARNING: for $($_.Key) it does not appear that IOPS moved in"
write-host -ForegroundColor Yellow "`trelation to attempts to raise the QoS limit. Check if the"
write-host -ForegroundColor Yellow "`tsystem is storage limited for this mix."
}
}
@@ -0,0 +1,201 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param (
[int] $interval = 10
)
class RunningStat {
# This is an implementation of an online (add one value at a time) method to calculate
# up to the fourth central moment (mean, variance, skewness and kurtosis) of a series
# of decimal values.
#
# Implementation is due to https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
# which is in turn due http://people.xiph.org/~tterribe/notes/homs.html by Timothy Terriberry,
# also cited by Pebay: http://prod.sandia.gov/techlib/access-control.cgi/2008/086212.pdf
#
# Note: skewness appears to match Excel calculations. Kurtosis does NOT at this time, and
# should be used with caution.
[decimal] $n;
[decimal] $M1;
[decimal] $M2;
[decimal] $M3;
[decimal] $M4;
[decimal] $Min;
[decimal] $Max;
RunningStat()
{
$this.M1 = 0
$this.M2 = 0
$this.M3 = 0
$this.M4 = 0
$this.Min = 0
$this.Max = 0
}
[void] Clear()
{
$this.M1 = 0
$this.M2 = 0
$this.M3 = 0
$this.M4 = 0
$this.Min = 0
$this.Max = 0
}
[void] Add([decimal] $v)
{
$n1 = $this.n
$this.n += 1
$delta = $v - $this.M1
$deltan = $delta / $this.n
$deltan2 = $deltan * $deltan
$term1 = $delta * $deltan * $n1
$this.M1 += $deltan
$this.M4 += $term1 * $deltan2 * (($this.n * $this.n) - (3 * $this.n) + 3) + (6 * $deltan2 * $this.M2) - (4 * $deltan * $this.M3)
$this.M3 += $term1 * $deltan * ($this.n - 2) - (3 * $deltan * $this.M2)
$this.M2 += $term1
if ($n1 -eq 0 -or $v -gt $this.Max) { $this.Max = $v }
if ($n1 -eq 0 -or $v -lt $this.Min) { $this.Min = $v }
}
[decimal] Min()
{
return $this.Min
}
[decimal] Max()
{
return $this.Max
}
[decimal] Mean()
{
return $this.M1
}
[decimal] Variance()
{
return $this.M2/($this.n - 1)
}
[decimal] StdDev()
{
return [math]::Sqrt($this.Variance())
}
[decimal] Skew()
{
return [math]::Sqrt($this.n) * $this.M3 / [math]::Pow($this.M2, 1.5)
}
[decimal] Kurtosis()
{
return (($this.n * $this.M4) / ($this.M2 * $this.M2)) - 3
}
}
# populate a hash by sbl device number
$dev = gwmi -Namespace root\wmi ClusPortDeviceInformation
$devhash = @{}
$dev |% { $devhash[[int]$_.devicenumber] = $_ }
function get-outliers(
[string[]] $paths
)
{
# filters and labels for the sigma buckets
$sigflt = @(@(">5", '$sig -gt 5'),
@("4-5", '$sig -gt 4 -and $sig -le 5'),
@("3-4", '$sig -gt 3 -and $sig -le 4')) |% {
new-object -TypeName psobject -Property @{
Label = $_[0];
Test= $_[1];
}
}
$ctrs = (get-counter -ComputerName (get-clusternode |? State -eq Up) -SampleInterval $interval -Counter $($paths |% { "\Cluster Disk Counters(*)\$_" })).countersamples |? { $_.instancename -ne "_total" }
$stat = [RunningStat]::new()
foreach ($path in $paths) {
# select out the specific subset of the counters (read, write, etc.)
$ctr = $ctrs |? { $_.Path -like "*$path" }
$stat.Clear()
$ctr.CookedValue |% { $stat.Add($_) }
$stddev = $stat.StdDev()
$mean = $stat.Mean()
# hash of flagged devices
$flagged = @{}
write-host -ForegroundColor Green ("-"*20)
write-host -ForegroundColor green Sample: $path
write-host Number of measured devices: $ctr.count
write-host Average of $("$path : {0:F3}ms" -f ($mean*1000))
write-host Standard Deviation of $("{0:F3}ms" -f ($stddev*1000))
# enumerate from highest to lowest deviation of merit
foreach ($sigma in $sigflt) {
# get new outliers at this deviation
$outlier = $ctr | sort -property InstanceName |? {
if ($stddev -eq 0) { $sig = 0 } else { $sig = (($_.CookedValue -$mean)/$stddev) }
-not $flagged[$_.InstanceName] -and (iex $sigma.Test)
}
if ($outlier) {
write-host -fore Yellow $sigma.Label sigma : $outlier.count total
$outlier |% {
# remember we flagged this device already
$flagged[$_.InstanceName] = 1
$thisdev = $devhash[[int]$_.InstanceName]
# parse source node
# \\foo\... -> 0 1 2=foo 3 ...
$sourcenode = ($_.Path -split '\\')[2]
write-host -ForegroundColor Red $sourcenode SSB Device: $_.InstanceName`n`tAverage of $path $("{0:F3}ms ({1:F1} sigma)" -f ($_.CookedValue * 1000),(($_.CookedValue -$mean)/$stddev))
write-host -ForegroundColor Red "`tConnected Node:" $thisdev.ConnectedNode
write-host -ForegroundColor Red "`tConnected Node Local PhysicalDisk Number:" $thisdev.ConnectedNodeDeviceNumber
write-host -ForegroundColor Red "`tModel | SerialNumber: $($thisdev.ProductId)` | $($thisdev.SerialNumber)"
}
}
}
}
}
get-outliers -paths "Remote: Read Latency","Local: Read Latency","Remote: Write Latency","Local: Write Latency"
@@ -0,0 +1,90 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
# script to check the state/health of pause in the fleet
#
# isactive - whether there is an active pause (independent of health)
# !isactive - whether there is an active pause and all VMs have responded to it
#
# true/false return
param(
[switch] $isactive = $false
)
$pause = "C:\ClusterStorage\collect\control\flag\pause"
$pauseepoch = gc $pause -ErrorAction SilentlyContinue
if ($pauseepoch -eq $null) {
write-host -fore red Pause not in force
return $false
}
# if only performing the pause active check, done
if ($isactive) {
return $true
}
# accumulate hash of pause flags mapped to current/stale state
$h = @{}
dir $pause-* |% {
$thispause = gc $_ -ErrorAction SilentlyContinue
if ($thispause -eq $pauseepoch) {
$pausetype = 'Current'
} else {
$pausetype = 'Stale'
}
if ($_.name -match 'pause-(.+)\+(vm.+)') {
# 1 is CSV, 2 is VM name
$h[$matches[2]] = $pausetype
} else {
write-host -fore red ERROR: malformed pause $_.name present
}
}
# now correlate to online vms and see if we agree all online are paused.
# note that if we shutdown some vms and then check pause the current flags
# will be higher than online, so we need to verify individually to not
# spoof ourselves.
$vms = get-clustergroup |? GroupType -eq VirtualMachine |? Name -like 'vm-*' |? State -eq Online
$pausedvms = $vms |? { $h[$_.Name] -eq 'Current' }
if ($pausedvms.Count -eq $vms.Count) {
write-host -fore green OK: All $vms.count VMs paused
} else {
write-host -fore red WARNING: of "$($vms.Count)," still waiting on ($vms.Count - $pausedvms.Count) to acknowledge pause
compare-object $vms $pausedvms -Property Name -PassThru | sort -Property Name | ft -AutoSize Name,OwnerNode | Out-Host
return $false
}
return $true
@@ -0,0 +1,126 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param ($group = '*')
$g = Get-ClusterGroup |? GroupType -eq VirtualMachine |? Name -like "vm-$group-*"
###########################################################
write-host -fore green State Pivot
$g | group -Property State -NoElement | sort -Property Name | ft -autosize
###########################################################
write-host -fore green Host Pivot
$g | group -Property OwnerNode,State -NoElement | sort -Property Name | ft -autosize
###########################################################
write-host -fore green Group Pivot
$g |% {
if ($_.Name -match "^vm-([^-]+)-") {
$_ | add-member -NotePropertyName Group -NotePropertyValue $matches[1] -PassThru
}
} | group -Property Group,State -NoElement | sort -Property Name | ft -AutoSize
###########################################################
write-host -fore green IOPS Pivot
# build 5 orders of log steps
$logstep = 10,20,50
$log = 1
$logs = 1..5 |% {
$logstep |% { $_ * $log }
$log *= 10
}
# build log step names; 0 is the > range catchall
$lognames = @{}
foreach ($step in $logs) {
if ($step -eq $logs[0]) {
$lognames[$step] = "< $step"
} else {
$lognames[$step] = "$pstep - $($step - 1)"
}
$pstep = $step
}
$lognames[0] = "> $($logs[-1])"
# now bucket up VMs by flow rates
$qosbuckets = @{}
$qosbuckets[0] = 0
$logs |% {
$qosbuckets[$_] = 0
}
Get-StorageQoSFlow |% {
$found = $false
foreach ($step in $logs) {
if ($_.InitiatorIops -lt $step) {
$qosbuckets[$step] += 1;
$found = $true
break
}
}
# if not bucketed, it is greater than range
if (-not $found) {
$qosbuckets[0] += 1
}
}
# find min/max buckets with nonzero counts, by $logs index
# this lets us present a continuous range, with interleaved zeroes
$bmax = -1
$bmin = -1
foreach ($i in 0..($logs.Count - 1)) {
if ($qosbuckets[$logs[$i]]) {
if ($bmin -lt 0) {
$bmin = $i
}
$bmax = $i
}
}
# raise max if we have > range
if ($qosbuckets[0]) {
$bmax = $logs.Count - 1
}
$range = @($logs[$bmin..$bmax])
# add > range if needed, at end
if ($qosbuckets[0]) {
$range += 0
}
$(foreach ($i in $range) {
New-Object -TypeName psobject -Property @{
Count = $qosbuckets[$i];
IOPS = $lognames[$i]
}
}) | ft Count,IOPS
@@ -0,0 +1,38 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
$pause = "C:\ClusterStorage\collect\control\flag\pause"
if (gi $pause -ErrorAction SilentlyContinue) {
write-host -fore green Clearing pause from $([string](gi $pause).LastWriteTime)
do {
del $pause -Force -ErrorAction SilentlyContinue
} while (-not $?)
# del $pause-* -ErrorAction SilentlyContinue
} else {
write-host -fore yellow Pause not set
}
@@ -0,0 +1,366 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
[string]$BaseVHD = $(throw "please specify a base vhd"),
[int]$VMs = $(throw "please specify a number of vms per node csv"),
[string[]]$Groups = @(),
[string]$AdminPass = $(throw 'need admin password for autologin'),
[string]$Admin = 'administrator',
[string]$ConnectPass = $(throw 'need password for loopback host connection'),
[string]$ConnectUser = $(throw 'need username for loopback host connection'),
[validateset('CreateVMSwitch','CopyVHD','CreateVM','CreateVMGroup','AssertComplete')][string]$StopAfter,
[validateset('Force','Auto','None')][string]$Specialize = 'Auto',
[switch]$FixedVHD = $true,
[string[]]$Nodes = @()
)
function Stop-After($step)
{
if ($stopafter -eq $step) {
write-host -ForegroundColor Green Stop after $step
return $true
}
return $false
}
##################
# validate existence of basevhd
if (!(test-path -path $BaseVHD)) {
throw "Base VHD $BaseVHD not found"
}
if (Get-ClusterNode |? State -ne Up) {
throw "not all cluster nodes are up; please address before creating vmfleet"
}
# if no nodes specified, use the entire cluster
if ($nodes.count -eq 0) {
$nodes = Get-ClusterNode
}
# convert to fixed vhd(x) if needed
if ((get-vhd $basevhd).VhdType -ne 'Fixed' -and $fixedvhd) {
# push dynamic vhd to tmppath and place converted at original
# note that converting a dynamic will leave a sparse hole on refs
# this is OK, since the copy will not copy the hole
$f = gi $basevhd
$tmpname = "tmp-$($f.Name)"
$tmppath = join-path $f.DirectoryName $tmpname
del -Force $tmppath -ErrorAction SilentlyContinue
ren $f.FullName $tmpname
write-host -ForegroundColor Yellow "convert $($f.FullName) to fixed via $tmppath"
convert-vhd -Path $tmppath -DestinationPath $f.FullName -VHDType Fixed
if (-not $?) {
ren $tmppath $f.Name
throw "ERROR: could not convert $($f.fullname) to fixed vhdx"
}
del $tmppath
}
# Create the fleet vmswitches with a fixed IP at the base of the APIPA range
icm $nodes {
if (-not (Get-VMSwitch -Name Internal -ErrorAction SilentlyContinue)) {
New-VMSwitch -name Internal -SwitchType Internal
Get-NetAdapter |? DriverDescription -eq 'Hyper-V Virtual Ethernet Adapter' |? Name -eq 'vEthernet (Internal)' | New-NetIPAddress -PrefixLength 16 -IPAddress '169.254.1.1'
}
} | ft -AutoSize
#### STOPAFTER
if (Stop-After "CreateVMSwitch") {
return
}
# create $vms vms per each csv named as <nodename><group prefix>
# vm name is vm-<group prefix><$group>-<hostname>-<number>
icm $nodes -ArgumentList $stopafter,(Get-Command Stop-After) {
param( [string]$stopafter,
$fn )
set-item -Path function:\$($fn.name) -Value $fn.definition
# workaround evaluation bug and make $stopafter evaluate in the session
$null = $stopafter
function apply-specialization( $path )
{
# all steps here can fail immediately without cleanup
# error accumulator
$ok = $true
# create run directory
del -Recurse -Force z:\run -ErrorAction SilentlyContinue
mkdir z:\run
$ok = $ok -band $?
if (-not $ok) {
Write-Error "failed run directory creation for $vhdpath"
return $ok
}
# autologon
$null = reg load 'HKLM\tmp' z:\windows\system32\config\software
$ok = $ok -band $?
$null = reg add 'HKLM\tmp\Microsoft\Windows NT\CurrentVersion\WinLogon' /f /v DefaultUserName /t REG_SZ /d $using:admin
$ok = $ok -band $?
$null = reg add 'HKLM\tmp\Microsoft\Windows NT\CurrentVersion\WinLogon' /f /v DefaultPassword /t REG_SZ /d $using:adminpass
$ok = $ok -band $?
$null = reg add 'HKLM\tmp\Microsoft\Windows NT\CurrentVersion\WinLogon' /f /v AutoAdminLogon /t REG_DWORD /d 1
$ok = $ok -band $?
$null = reg add 'HKLM\tmp\Microsoft\Windows NT\CurrentVersion\WinLogon' /f /v Shell /t REG_SZ /d 'powershell.exe -noexit -command c:\users\administrator\launch.ps1'
$ok = $ok -band $?
$null = [gc]::Collect()
$ok = $ok -band $?
$null = reg unload 'HKLM\tmp'
$ok = $ok -band $?
if (-not $ok) {
Write-Error "failed autologon injection for $vhdpath"
return $ok
}
# scripts
copy -Force C:\ClusterStorage\collect\control\master.ps1 z:\run\master.ps1
$ok = $ok -band $?
if (-not $ok) {
Write-Error "failed injection of specd master.ps1 for $vhdpath"
return $ok
}
del -Force z:\users\administrator\launch.ps1 -ErrorAction SilentlyContinue
gc C:\ClusterStorage\collect\control\launch-template.ps1 |% { $_ -replace '__CONNECTUSER__',$using:connectuser -replace '__CONNECTPASS__',$using:connectpass } > z:\users\administrator\launch.ps1
$ok = $ok -band $?
if (-not $ok) {
Write-Error "failed injection of launch.ps1 for $vhdpath"
return $err
}
echo $vmspec > z:\vmspec.txt
$ok = $ok -band $?
if (-not $ok) {
Write-Error "failed injection of vmspec for $vhdpath"
return $ok
}
# load files
$f = 'z:\run\testfile1.dat'
if (-not (gi $f -ErrorAction SilentlyContinue)) {
fsutil file createnew $f (10GB)
$ok = $ok -band $?
fsutil file setvaliddata $f (10GB)
$ok = $ok -band $?
}
if (-not $ok) {
Write-Error "failed creation of initial load file for $vhdpath"
return $ok
}
return $ok
}
function specialize-vhd( $vhdpath )
{
$vhd = (gi $vhdpath)
$vmspec = $vhd.Directory.Name,$vhd.BaseName -join '+'
# mount vhd and its largest partition
$o = Mount-VHD $vhd -NoDriveLetter -Passthru
if ($o -eq $null) {
Write-Error "failed mount for $vhdpath"
return $false
}
$p = Get-Disk -number $o.DiskNumber | Get-Partition | sort -Property size -Descending | select -first 1
$p | Add-PartitionAccessPath -AccessPath Z:
$ok = apply-specialization Z:
Remove-PartitionAccessPath -AccessPath Z: -InputObject $p
Dismount-VHD -DiskNumber $o.DiskNumber
return $ok
}
$csvs = Get-ClusterSharedVolume
# handle restore cases by mapping the csv to the friendly name of the volume
# don't rely on the csv name to contain this data
$vh = @{}
Get-Volume |? FileSystem -eq CSVFS |% { $vh[$_.Path] = $_ }
$csvs |% {
$v = $vh[$_.SharedVolumeInfo.Partition.Name]
if ($v -ne $null) {
$_ | Add-Member -NotePropertyName VDName -NotePropertyValue $v.FileSystemLabel
}
}
foreach ($csv in $csvs) {
if ($($using:groups).Length -eq 0) {
$groups = @( 'base' )
} else {
$groups = $using:groups
}
# identify the CSvs for which this node should create its VMs
# the trailing characters (if any) are the group prefix
if ($csv.VDName -match "^$env:COMPUTERNAME(?:-.+){0,1}") {
foreach ($group in $groups) {
if ($csv.VDName -match "^$env:COMPUTERNAME-([^-]+)$") {
$g = $group+$matches[1]
} else {
$g = $group
}
foreach ($vm in 1..$using:vms) {
$stop = $false
$newvm = $false
$name = "vm-$g-$env:COMPUTERNAME-$vm"
$path = Join-Path $csv.SharedVolumeInfo.FriendlyVolumeName $name
# place vhdx in subdirectory, per scvmm layout defaults
# $vhd = $path+".vhdx"
$vhd = join-path $path "$name.vhdx"
# if the vm cluster group exists, we are already deployed
if (-not (Get-ClusterGroup -Name $name -ErrorAction SilentlyContinue)) {
if (-not $stop) {
$stop = Stop-After "AssertComplete"
}
if ($stop) {
Write-Host -ForegroundColor Red "vm $name not deployed"
} else {
Write-Host -ForegroundColor Yellow "create vm $name @ metadata path $path with vhd $vhd"
# create vm if not already done
# note that when restarting interrupted creation, the vm could have moved elsewhere
# under cluster control.
$o = Get-ClusterGroup |? GroupType -eq VirtualMachine |? Name -eq $name
if (-not $o) {
# force re-specialization
$newvm = $true
# if the cluster group doesn't exist, we're on the canonical node to create the vm
# if the vm exists, tear it down and refresh
$o = get-vm -Name $name -ErrorAction SilentlyContinue
if ($o) {
# interrupted between vm creation and role creation; redo it
write-host "REMOVING vm $name for re-creation"
if ($o.State -ne 'Off') {
Stop-VM -Name $name -Force -Confirm:$false
}
Remove-VM -Name $name -Force -Confirm:$false
} else {
# scrub and re-create the vm metadata path and vhd
rmdir -ErrorAction SilentlyContinue -Recurse $path
$null = mkdir -ErrorAction SilentlyContinue $path
cp $using:basevhd $vhd
}
#### STOPAFTER
if (-not $stop) {
$stop = Stop-After "CopyVHD"
}
if (-not $stop) {
$o = New-VM -VHDPath $vhd -Generation 2 -SwitchName Internal -Path $path -Name $name
# create A1 VM. use set-vmfleet to alter fleet sizing post-creation.
$o | Set-VM -ProcessorCount 1 -MemoryStartupBytes 1.75GB -StaticMemory
# do not monitor the internal switch connection; this allows live migration
$o | Get-VMNetworkAdapter| Set-VMNetworkAdapter -NotMonitoredInCluster $true
}
}
#### STOPAFTER
if (-not $stop) {
$stop = Stop-After "CreateVM"
}
if (-not $stop) {
# create clustered vm role and assign default owner node
$o | Add-ClusterVirtualMachineRole
Set-ClusterOwnerNode -Group $o.VMName -Owners $env:COMPUTERNAME
}
}
} else {
Write-Host -ForegroundColor Green "vm $name already deployed"
}
#### STOPAFTER
if (-not $stop) {
$stop = Stop-After "CreateVMGroup"
}
if (-not $stop -or ($using:specialize -eq 'Force')) {
# specialize as needed
# auto only specializes new vms; force always; none skips it
if (($using:specialize -eq 'Auto' -and $newvm) -or ($using:specialize -eq 'Force')) {
write-host -fore yellow specialize $vhd
if (-not (specialize-vhd $vhd)) {
write-host -fore red "Failed specialize of $vhd, halting."
}
} else {
write-host -fore green skip specialize $vhd
}
}
}
}
}
}
}
@@ -0,0 +1,72 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param([int] $interval = 60)
$cnodes = (Get-ClusterNode).Count
$cvms = (Get-ClusterGroup |? GroupType -eq 'VirtualMachine').Count/$cnodes
# loop through the set of run scripts
$f = gi C:\clusterstorage\collect\control\run-demo-*.ps1
# qos policies to loop
$policy = 'SilverVM','GoldVM','PlatinumVM',$null,$null
while ($true) {
$null = & update-csv.ps1
foreach ($run in $f) {
cls
# touch the current run file to trigger VM updates
# see the master script
(gi $run).LastWriteTime = (get-date)
# pull the show-me line of the run script (comment, trimmed)
# format is a single line, with # standing in for newline for multi-line output
# substitute in the configuration's node and vms/node count
# TBD: autodetect vd configuration
$comm = (gc $run | sls '^#-#').Line
write-host -fore green $($comm.trimstart('#- ') -replace '__CVMS__',$cvms -replace '__CNODES__',$cnodes -replace '#',"`n")
# apply random QoS policy
$p = $policy[(0..($policy.count - 1) | Get-Random)]
$null = & set-storageqos.ps1 -policyname $p 2>&1
write-host -fore yellow `nActive QoS Policy
if ($p) {
Get-StorageQosPolicy -Name $p | ft -AutoSize
} else {
write-host NONE
}
sleep $interval
}
}
@@ -0,0 +1,102 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
[string[]] $vms = $null
)
if ($vms) {
write-host -ForegroundColor Yellow "Destroying VM Fleet content for: $vms"
} else {
write-host -ForegroundColor Yellow "Destroying VM Fleet"
}
# stop and remove clustered vm roles
write-host -ForegroundColor Green "Removing VM ClusterGroups"
Get-ClusterGroup |? GroupType -eq VirtualMachine |? {
if ($vms) {
$_.Name -in $vms
} else {
$_.Name -like 'vm-*'
}
} |% {
write-host "Removing ClusterGroup for $($_.Name)"
$_ | Stop-ClusterGroup
$_ | Remove-ClusterGroup -RemoveResources -Force
}
# remove all vms
write-host -ForegroundColor Green "Removing VMs"
icm (Get-ClusterNode) {
Get-VM |? {
if ($using:vms) {
$_.Name -in $using:vms
} else {
$_.Name -like 'vm-*'
}
} |% {
write-host "Removing VM for $($_.Name) @ $($env:COMPUTERNAME)"
$_ | Remove-VM -Confirm:$false -Force
}
# do not remove the internal switch if teardown is partial
if ($using:vms -eq $null) {
write-host -ForegroundColor Green "Removing Internal VMSwitch"
Get-VMSwitch -SwitchType Internal | Remove-VMSwitch -Confirm:$False -Force
}
}
# handle restore cases by mapping the csv to the friendly name of the volume
# don't rely on the csv name to contain this data
$csv = Get-ClusterSharedVolume
$vh = @{}
Get-Volume |? FileSystem -eq CSVFS |% { $vh[$_.Path] = $_ }
$csv |% {
$v = $vh[$_.SharedVolumeInfo.Partition.Name]
if ($v -ne $null) {
$_ | Add-Member -NotePropertyName VDName -NotePropertyValue $v.FileSystemLabel
}
}
# now delete content from csvs corresponding to the cluster nodes
write-host -ForegroundColor Green "Removing CSV content for VMs"
Get-ClusterNode |% {
$csv |? VDName -match "$($_.Name)(-.+)?"
} |% {
dir -Directory $_.sharedvolumeinfo.friendlyvolumename |? {
if ($vms) {
$_.Name -in $vms
} else {
$_.Name -like 'vm-*'
}
} |% {
write-host "Removing CSV content for $($_.BaseName) @ $($_.FullName)"
del -Recurse -Force $_.FullName
}
}
@@ -0,0 +1,281 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
##
## netsh advfirewall firewall set rule group="Performance Logs and Alerts" new enable=yes
##
[CmdletBinding( DefaultParameterSetName = "BySeconds" )]
param(
[Parameter(
ParameterSetName = 'BySeconds',
ValueFromPipeline = $false,
ValueFromPipelineByPropertyName = $false,
Mandatory = $false)]
[ValidateRange(1,[int]::MaxValue)]
[int] $Seconds = 1,
[Parameter(
ParameterSetName = 'ByStart',
ValueFromPipeline = $false,
ValueFromPipelineByPropertyName = $false,
Mandatory = $false)]
[switch] $Start,
[Parameter(
ParameterSetName = 'ByStop',
ValueFromPipeline = $false,
ValueFromPipelineByPropertyName = $false,
Mandatory = $false)]
[switch] $Stop = $false,
<# --------- common #>
[Parameter(
ParameterSetName = 'BySeconds',
Mandatory = $false)]
[Parameter(
ParameterSetName = 'ByStart',
Mandatory = $false)]
[Parameter(
ParameterSetName = 'ByStop',
Mandatory = $false)]
[string] $Cluster = ".",
<# --------- start-time parameters #>
[Parameter(
ParameterSetName = 'BySeconds',
Mandatory = $false)]
[Parameter(
ParameterSetName = 'ByStart',
Mandatory = $false)]
[ValidateSet('PhysicalDisk','CPU','SMB','SMBD','SSB','SSB Cache','CSVFS','Storage','Spaces')]
[string[]] $Set = '*',
[Parameter(
ParameterSetName = 'BySeconds',
Mandatory = $false)]
[Parameter(
ParameterSetName = 'ByStart',
Mandatory = $false)]
[string] $AddSpec,
[Parameter(
ParameterSetName = 'BySeconds',
Mandatory = $false)]
[Parameter(
ParameterSetName = 'ByStart',
Mandatory = $false)]
[int] $SampleInterval = 1,
<# --------- stop-time parameters #>
[Parameter(
ParameterSetName = 'BySeconds',
Mandatory = $false)]
[Parameter(
ParameterSetName = 'ByStop',
Mandatory = $false)]
[switch] $Force = $false,
[Parameter(
ParameterSetName = 'BySeconds',
Mandatory = $true)]
[Parameter(
ParameterSetName = 'ByStop',
Mandatory = $true)]
[string] $Destination
)
if ($psCmdlet.ParameterSetName -ne 'ByStart' -and
(gi -ErrorAction SilentlyContinue $Destination)) {
if (-not $force) {
Write-Error "$Destination already exists, please delete or use -Force to overwrite"
return
} else {
del -ErrorAction SilentlyContinue $Destination
}
}
$sets = @{
'PhysicalDisk' = '\PhysicalDisk(*)\*','+getclusport';
'CSVFS' = '\Cluster CSVFS(*)\*','\Cluster CSV Volume Cache(*)\*','\Cluster CSV Volume Manager(*)\*','\Cluster CSVFS Block Cache(*)\*','\Cluster CSVFS Direct IO(*)\*','\Cluster CSVFS Redirected IO(*)\*','+getcsv';
'SSB' = '\Cluster Disk Counters(*)\*','+getclusport';
'SMB' = '\SMB Client Shares(*)\*','\SMB Server Shares(*)\*';
'SMBD' = '\SMB Direct Connection(*)\*';
'SSB Cache' = '\Cluster Storage Hybrid Disks(*)\*','\Cluster Storage Cache Stores(*)\*';
'Spaces' = '\Storage Spaces Write Cache(*)\*','\Storage Spaces Tier(*)\*','\Storage Spaces Virtual Disk(*)\*'
'ReFS' = '\ReFS(*)\*';
'CPU' = '\Hyper-V Hypervisor Logical Processor(*)\*','\Processor Information(*)\*';
'Storage' = 'PhysicalDisk','CSVFS','SSB','SSB Cache','ReFS','Spaces';
}
$special = @{
"getclusport" = $false;
"getcsv" = $false;
}
$cleanup = @()
$pc = @()
if ($Set.count -eq 1 -and $Set[0] -eq '*') {
# list of all keys
$pc = $sets.keys |% { $_ }
} else {
$pc = $Set
}
# repeat expansion (sets in sets, possibly containing special collection rules)
do {
$expansion = $false
$pc = $pc |% {
$n = $_
switch ($n[0]) {
# performance counter - passthru
'\' { $n }
# special collection (sets variable -> $true)
'+' {
if ($special.ContainsKey($n.Substring(1))) {
$special[$n.Substring(1)] = $true
} else {
throw "unrecognized special gather command $($n.Substring(1))"
}
}
# set expansion
default {
$sets[$n]
$expansion = $true
}
}
}
} while ($expansion)
# uniq the counters
$pc = ($pc | group -NoElement).Name
# --
function start-logman(
[string] $name,
[string[]] $counters,
[int] $sampleinterval
)
{
$computer = $env:COMPUTERNAME
$f = "c:\perfctr-$name-$computer.blg"
$null = logman create counter "perfctr-$name" -o $f -f bin -si $sampleinterval --v -c $counters
$null = logman start "perfctr-$name"
write-host "performance counters on: $computer"
}
function stop-logman(
[string] $name
)
{
$computer = $env:COMPUTERNAME
$f = "c:\perfctr-$name-$computer.blg"
$null = logman stop "perfctr-$name"
$null = logman delete "perfctr-$name"
write-host "performance counters off: $computer"
echo $("\\$computer\$f" -replace ':','$')
}
if (-not $Stop) {
icm (get-clusternode -cluster $Cluster |? State -eq Up) -ArgumentList (get-command start-logman) {
param($fn)
set-item -path function:\$($fn.name) -value $fn.definition
start-logman $using:AddSpec $using:pc -sampleinterval $using:SampleInterval
}
if ($Start) {
write-host -ForegroundColor Yellow INFO: captures started - reinvoke with `-stop to complete capture
return
}
sleep $Seconds
} else {
write-host -ForegroundColor Yellow INFO: completing previously started capture
}
# now capture all counter files
$f = @()
$f += icm (get-clusternode -cluster $Cluster |? State -eq Up) -ArgumentList (get-command stop-logman) {
param($fn)
set-item -path function:\$($fn.name) -value $fn.definition
stop-logman $using:AddSpec $using:Destination
}
# add all counter blg to the cleanup step
$cleanup += $f
#--
# specials
#--
# make capture directory, and add to cleanup list
# note that all specials are generated into this directory,
# and will be automatically cleaned up when it is deleted
$t = New-TemporaryFile
del $t
$null = md $t
$cleanup += $t
if ($special['getclusport']) {
get-clusternode -cluster $Cluster |? State -eq Up |% {
$exp = "$t\clusport-$_.xml"
gwmi -ComputerName $_ -Namespace root\wmi ClusPortDeviceInformation | export-clixml -Path $exp
$f += $exp
}
}
if ($special['getcsv']) {
$exp = "$t\csv.xml"
Get-ClusterSharedVolume -cluster $Cluster | export-clixml -Path $exp
$f += $exp
}
compress-archive -DestinationPath $Destination -Path $f
del -Force -Recurse $cleanup
write-host "performance counters: $Destination"
@@ -0,0 +1,143 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
$csvfile,
[string]$delimeter = "`t",
[string]$xcol = $(throw "please specify column containing x values"),
[string]$ycol = $(throw "please specify column containing y values"),
[string[]]$idxcol,
[switch]$zerointercept = $false
)
# given the input data, produce the linear fit coefficients for
#
# zerointercept = true: y = bx
# !zerointercept = true: y = a + bx
#
# specify the column containing the x and y measurements
# a single index column can be used to differentiate rows which should be
# used for the fit (i.e., "test1", "test2", ...).
#
# method: ordinary least squares
function do-linfit(
[string] $xcol,
[string] $ycol,
[string] $key
)
{
BEGIN
{
$sumxy = [decimal]0
$sumx2 = [decimal]0
$sumx = [decimal]0
$sumy = [decimal]0
$n = 0
$pipe = @()
}
PROCESS
{
$x = [decimal]$_.$xcol
$y = [decimal]$_.$ycol
$sumxy += $x*$y
$sumx2 += $x*$x
$sumx += $x
$sumy += $y
$n += 1
# accumulate pipeline for second pass
$pipe += $_
}
END
{
if ($n -eq 0) {
Write-Error "ERROR: no measurements matched"
return
}
# perform requested fit
$a = 0
$b = 0
if ($zerointercept) {
$a = 0
$b = ($sumxy/$sumx2)
} else {
$b = ($sumxy - (($sumx*$sumy)/$n)) / ($sumx2 - (($sumx*$sumx)/$n))
$a = ($sumy - $b*$sumx)/$n
}
# calculate r2 (coefficient of determination) with respect to the fit
$meany = $sumy/$n
# total sum of squares
$sstot = [decimal]0
$pipe |% {
$v = [decimal]$_.$ycol - $meany
$sstot += $v*$v
}
# residual sum of squares
$ssres = [decimal]0
$pipe |% {
$v = [decimal]$_.$ycol - ($a + $b*[decimal]$_.$xcol)
$ssres += $v*$v
}
$r2 = 1 - ($ssres/$sstot)
new-object -TypeName psobject -Property @{ 'A' = $a; 'B' = $b; 'R2' = $r2; 'N' = $n; 'Key' = $key; 'X' = $xcol; 'Y' = $ycol }
}
}
# process the results into a hash keyed by the index columns
$h = @{}
import-csv -Path $csvfile -Delimiter $delimeter |% {
$key = ""
foreach ($col in $idxcol) {
$key += "$col $($_.$col)"
}
$h[$key] += ,$_
}
$h.Keys |% {
$h[$_] | do-linfit -xcol $xcol -ycol $ycol -key $_
}
@@ -0,0 +1,80 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
[string] $zip = $(throw "please provide zip filename for compressed logs"),
[ValidateSet("*","HyperV","FailoverClustering","SMB")][string[]] $set = "*",
[int] $timespan = 0
)
# convert minutes to milliseconds
if ($timespan -gt 0) {
$timespan *= 60*1000
}
$logs = icm (get-clusternode) {
$map = @{
"HyperV" = "Microsoft-Windows-Hyper-V";
"FailoverClustering" = "Microsoft-Windows-FailoverClustering"
"SMB" = "Microsoft-Windows-SMB"
}
if ($using:set -eq "*") {
$lset = $map.Keys |% { $_ }
} else {
$lset = $using:set
}
$q = @"
/q:"*[System[TimeCreated[timediff(@SystemTime) <= __TIME__]]]"
"@
if ($using:timespan -gt 0) {
$timefilt = $q -replace '__TIME__',$using:timespan
} else {
$timefilt = $null
}
$lset |% {
# get logs and normalize to legal filenames
$prov = wevtutil el | sls $map[$_]
$provf = $prov -replace '/','-'
foreach ($i in 0..($prov.Count - 1)) {
$localpath = "$($provf[$i])--$env:computername.evtx"
del -Force $localpath -ErrorAction SilentlyContinue
wevtutil epl $prov[$i] "c:\$localpath" $timefilt
write-output "\\$env:computername\c$\$localpath"
}
}
}
compress-archive -Path $logs -DestinationPath $zip
del -force $logs
@@ -0,0 +1,74 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
$source = $(throw "Must specify the source directory for the vmfleet scripts")
)
$col = get-clustersharedvolume |? Name -match '(collect)'
if ($col -eq $null) {
write-error "The collect CSV is not present"
return
}
$fqsrc = gi $source
if ($fqsrc -eq $null) {
write-error "The source directory for the vmfleet scripts is not accessible"
}
# update the csv mounts to their normalized names and install scripts and directory structure
# remove internet-download blocks on ps1 scripting, if present
& $fqsrc\update-csv.ps1 -renamecsvmounts
$control = 'C:\ClusterStorage\collect\control'
cp -r $fqsrc $control
dir $control\*.ps1 |% { Unblock-File $_ }
mkdir $control\result
mkdir $control\flag
mkdir $control\tools
# put the control directory onto the path
if (-not ([System.Environment]::GetEnvironmentVariable("path") -split ';' |? { $_ -eq $control })) {
$newpath = [System.Environment]::GetEnvironmentVariable("path") + ";$control"
[System.Environment]::SetEnvironmentVariable("path",
$newpath,
[System.EnvironmentVariableTarget]::Process)
[System.Environment]::SetEnvironmentVariable("path",
$newpath,
[System.EnvironmentVariableTarget]::User)
}
# disable the csv balancer so that motion is under fleet control
(get-cluster).CsvBalancer = 0
# finally set fleet pause and touch the base run file so that we ensure
# the fleet will start with it.
set-pause
(gi $control\run.ps1).IsReadOnly = $false
(gi $control\run.ps1).LastWriteTime = (Get-Date)
@@ -0,0 +1,34 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
$script = 'c:\run\master.ps1'
while ($true) {
Write-Host -fore Green Launching $script `@ $(Get-Date)
& $script -connectuser __CONNECTUSER__ -connectpass __CONNECTPASS__
sleep -Seconds 1
}
@@ -0,0 +1,225 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
#
# VM Master Script
#
# This script must be executed on autologon by the VM. It establishes a mapping a location containing the Run Script
# and then repeatedly polls/executes the latest one it finds. This allows the runner to inject new run scripts on the
# fly and/or touch pre-existing ones to bump them to the top of the list.
#
# In the initial form, it assumes the VM is connected by an internal switch and it just polls all the node names to
# find the one which will work. This could be improved by using specialization to inject the specific path to a
# configuration file.
#
param(
[string] $connectuser,
[string] $connectpass
)
$null = net use l: /d
if ($(Get-SmbMapping) -eq $null) {
New-SmbMapping -LocalPath l: -RemotePath \\169.254.1.1\c$\clusterstorage\collect\control -UserName $connectuser -Password $connectpass -ErrorAction SilentlyContinue
}
# update tooling
cp l:\tools\* c:\run -Force
$run = 'c:\run\run.ps1'
$master = 'c:\run\master.ps1'
$mypause = "l:\flag\pause-$(gc c:\vmspec.txt)"
$mydone = "l:\flag\done-$(gc c:\vmspec.txt)"
$done = $false
$donetouched = $false
$pause = $false
$tick = 0
function get-newfile(
$sourcepat,
$dest,
$clean = $false,
$silent = $false
)
{
$sf = dir $sourcepat | sort -Property LastWriteTime -Descending | select -first 1
$df = gi $dest -ErrorAction SilentlyContinue
# no source?
if ($sf -eq $null) {
write-host -fore green NO source present
# clean if needed
if ($clean) {
rm $dest
}
# no new file is an update condition
$true
} elseif ($df -eq $null -or $sf.lastwritetime -gt $df.lastwritetime) {
write-host -fore green NEWER source $sf.fullname '=>' $dest
# update with newer source file and indicate update condition
cp $sf.fullname $dest -Force
$true
} else {
if (-not $silent) {
write-host -fore green NO newer source $sf.lastwritetime $sf.fullname '**' $df.LastWriteTime $df.fullname
}
# already have latest, indicate no new
$false
}
}
function get-flagfile(
[string] $flag,
[switch] $gc = $false
)
{
if ($gc) {
gc "l:\flag\$flag" -ErrorAction SilentlyContinue
} else {
gi "l:\flag\$flag" -ErrorAction SilentlyContinue
}
}
while ($true) {
# update master control?
# assume runners have a simple loop to re-execute on exit
if (get-newfile l:\master*.ps1 $master -silent:$true) {
break
}
# check and acknowledge pause - only drop flag once
$pauseepoch = get-flagfile pause -gc:$true
if ($pauseepoch -ne $null) {
# pause clears done
$done = $false
# drop into pause flagfile if needed
if ($pause -eq $false) {
write-host -fore red PAUSE IN FORCE
$pause = $true
echo $pauseepoch > $mypause
} else {
if ($tick % 10 -eq 0) {
write-host -fore red -NoNewline '.'
}
}
} elseif ($done) {
# drop epoch into done flagfile if needed
if (-not $donetouched) {
echo $goepoch > $mydone
$donetouched = $true
}
# if go flag is now different, release for the next go around
if ($goepoch -ne (get-flagfile go -gc:$true)) {
$done = $false
write-host -fore cyan `nReleasing from Done
} else {
if ($tick % 10 -eq 0) {
write-host -fore yellow -NoNewline '.'
}
}
} else {
# clear pause & donetouched flags
$pause = $false
$donetouched = $false
# update run script?
$null = get-newfile l:\run*.ps1 $run -clean:$true
$runf = gi $run -ErrorAction SilentlyContinue
if ($runf -eq $null) {
write-host -fore yellow no control file found
sleep 30
continue
}
# update go epoch - a change to this (if present) will be what
# allows us to proceed past a done flag
$goepoch = get-flagfile go -gc:$true
if ($goepoch -ne $null) {
write-host -fore cyan Go Epoch acknowledged at: $goepoch
}
# acknowledge script
write-host -fore cyan Run Script $runf.lastwritetime "`n$("*"*20)"
gc $runf
write-host -fore cyan ("*"*20)
# launch and monitor pause and new run file
$j = start-job -arg $run { param($run) & $run }
while (($jf = wait-job $j -Timeout 1) -eq $null) {
$halt = $null
# check pause or new run file: if so, stop and loop
if (get-flagfile pause) {
$halt = "pause set"
}
if (get-newfile l:\run*.ps1 $run -clean:$true -silent:$true) {
$halt = "new run file"
}
if ($halt -ne $null) {
write-host -fore yellow STOPPING CURRENT "(reason: $halt)"
$j | stop-job
$j | remove-job
break
}
}
# job finished?
if ($jf -ne $null) {
$result = $jf | receive-job
if ($result -eq "done") {
write-host -fore yellow DONE CURRENT
$done = $true
}
$jf | remove-job
}
write-host -fore cyan ("*"*20)
}
# force gc to teardown potentially conflicting handles and enforce min pause
[system.gc]::Collect()
sleep 1
$tick += 1
}
@@ -0,0 +1,39 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
#-# System Config: __CNODES__ Systems x __CVMS__ VMs/System#Storage Config: 3-way S2D Mirror#Current Workload: 100% 4K Random Read
[string](get-date)
$b = 4
$t = 8
$o = 20
$w = 0
C:\run\diskspd.exe -n -h `-t$t `-o$o `-b$($b)k `-r$($b)k `-w$w -W10 -d60 -C10 -D -L (dir C:\run\testfile?.dat)
[string](get-date)
@@ -0,0 +1,38 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
#-# System Config: __CNODES__ Systems x __CVMS__ VMs/System#Storage Config: 3-way S2D Mirror#Current Workload: 70:30 4K Random Read/Write
[string](get-date)
$b = 4
$t = 8
$o = 20
$w = 30
C:\run\diskspd.exe -n -h `-t$t `-o$o `-b$($b)k `-r$($b)k `-w$w -W10 -d60 -C10 -D -L (dir C:\run\testfile?.dat)
[string](get-date)
@@ -0,0 +1,38 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
#-# System Config: __CNODES__ Systems x __CVMS__ VMs/System#Storage Config: 3-way S2D Mirror#Current Workload: 90:10 4K Random Read/Write
[string](get-date)
$b = 4
$t = 8
$o = 20
$w = 10
C:\run\diskspd.exe -n -h `-t$t `-o$o `-b$($b)k `-r$($b)k `-w$w -W10 -d60 -C10 -D -L (dir C:\run\testfile?.dat)
[string](get-date)
@@ -0,0 +1,78 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
[string](get-date)
# buffer size/alighment, threads/target, outstanding/thread, write%
$b = __b__; $t = __t__; $o = __o__; $w = __w__
# optional - specify rate limit in iops, translated to bytes/ms for DISKSPD
$iops = __iops__
if ($iops -ne $null) { $g = $iops * $b * 1KB / 1000 }
# io pattern, (r)andom or (s)equential (si as needed for multithread)
$p = '__p__'
# durations of test, cooldown, warmup
$d = __d__; $cool = __Cool__; $warm = __Warm__
# sweep template always captures
$addspec = '__AddSpec__'
$gspec = $null
if ($g -ne $null) { $gspec = "g$($g)" }
$result = "result-b$($b)t$($t)o$($o)w$($w)p$($p)$($gspec)-$($addspec)-$(gc c:\vmspec.txt).xml"
$dresult = "l:\result"
$lresultf = join-path "c:\run" $result
$dresultf = join-path $dresult $result
### prior to this is template
if (-not (gi $dresultf -ErrorAction SilentlyContinue)) {
$res = 'xml'
$gspec = $null
if ($g -ne $null) { $gspec = "-g$($g)" }
C:\run\diskspd.exe -Z20M -z -h `-t$t `-o$o $gspec `-b$($b)k `-$($p)$($b)k `-w$w `-W$warm `-C$cool `-d$($d) -D -L `-R$res (dir C:\run\testfile?.dat) > $lresultf
# export result and indicate done flag to master
# use unbuffered copy to force IO downstream
xcopy /j $lresultf $dresult
del $lresultf
Write-Output "done"
} else {
write-host -fore green already done $dresultf
# indicate done flag to master
# this should only occur if controller does not change variation
Write-Output "done"
}
[system.gc]::Collect()
[string](get-date)
@@ -0,0 +1,94 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
[string](get-date)
# buffer size/alighment, threads/target, outstanding/thread, write%
$b = 4; $t = 1; $o = 8; $w = 10
# optional - specify rate limit in iops, translated to bytes/ms for DISKSPD
$iops = 500
if ($iops -ne $null) { $g = $iops * $b * 1KB / 1000 }
# io pattern, (r)andom or (s)equential (si as needed for multithread)
$p = 'r'
# durations of test, cooldown, warmup
$d = 30*60; $cool = 30; $warm = 60
$addspec = 'base'
$gspec = $null
if ($g -ne $null) { $gspec = "g$($g)" }
$result = "result-b$($b)t$($t)o$($o)w$($w)p$($p)$($gspec)-$($addspec)-$(gc c:\vmspec.txt).xml"
$dresult = "l:\result"
$lresultf = join-path "c:\run" $result
$dresultf = join-path $dresult $result
# cap -> true to capture xml results, otherwise human text
$cap = $false
### prior to this is template
if (-not (gi $dresultf -ErrorAction SilentlyContinue)) {
if ($cap) {
$res = 'xml'
} else {
$res = 'text'
}
$gspec = $null
if ($g -ne $null) { $gspec = "-g$($g)" }
$o = C:\run\diskspd.exe -Z20M -z -h `-t$t `-o$o $gspec `-b$($b)k `-$($p)$($b)k `-w$w `-W$warm `-C$cool `-d$($d) -D -L `-R$res (dir C:\run\testfile?.dat)
if ($cap) {
# export result and indicate done flag to master
# use unbuffered copy to force IO downstream
$o | Out-File $lresultf -Encoding ascii -Width 9999
xcopy /j $lresultf $dresult
del $lresultf
Write-Output "done"
} else {
#emit to human watcher
$o | Out-Host
}
} else {
write-host -fore green already done $dresultf
# indicate done flag to master
# this should only occur if controller does not change variation
Write-Output "done"
}
[system.gc]::Collect()
[string](get-date)
@@ -0,0 +1,36 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
$pause = "C:\ClusterStorage\collect\control\flag\pause"
$p = gi $pause -ErrorAction SilentlyContinue
if ($p) {
write-host -fore green Pause already set $p.CreationTime
} else {
echo (get-random) > $pause
write-host -fore red Pause set `@ (get-date)
}
@@ -0,0 +1,54 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
# apply given QoS policy (by name) to all VMs on specified nodes
param (
$policyname = $null,
[object[]] $node = $(get-clusternode |? State -eq Up)
)
if ($policyname -ne $null) {
# QoS policy must exist, else error out
$qosp = get-storageqospolicy -name $policyname
if ($qosp -eq $null) {
# cmdlet error sufficient
return
}
$id = $qosp.PolicyId
} else {
# clears QoS policy
$id = $null
}
icm $node {
# note: set-vhdqos should be replaced with set-vmharddiskdrive
get-vm |% { get-vmharddiskdrive $_ |% { Set-VMHardDiskDrive -QoSPolicyID $using:id -VMHardDiskDrive $_ }}
}
@@ -0,0 +1,157 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param( [Parameter(ParameterSetName = 'FullSpec', Mandatory = $true)]
[int] $ProcessorCount,
[Parameter(ParameterSetName = 'FullSpec', Mandatory = $true)]
[int64] $MemoryStartupBytes,
[Parameter(ParameterSetName = 'FullSpec')]
[int64] $MemoryMaximumBytes = 0,
[Parameter(ParameterSetName = 'FullSpec')]
[int64] $MemoryMinimumBytes = 0,
[Parameter(ParameterSetName = 'FullSpec')]
[switch]$DynamicMemory = $true,
[Parameter(ParameterSetName = 'SizeSpec', Mandatory = $true)]
[ValidateSet('A0','A1','A1v2','A2','A2mv2','A2v2','A3','A4','A4mv2','A4v2','A5','A6','A7','A8mv2','A8v2','D1','D11','D11v2','D12','D12v2','D13','D13v2','D14','D14v2','D1
5v2','D1v2','D2','D2v2','D3','D3v2','D4','D4v2','D5v2','DS11','DS11v2','DS12','DS12v2','DS13','DS13v2','DS14','DS14v2','DS15v2')]
[string]$Compute = 'A1'
)
# to regenerate validateset
# ($vmsize.keys | sort |% { "'$_'" }) -join ','
# c = compute cores
# m = memory
# a = alias for another (ex: d -> ds, d -> dv2)
# note that alias is only chased once
$vmsize = @{
# general purpose table
'A0' = @{ 'c' = 1; 'm' = 0.75GB; };
'A1' = @{ 'c' = 1; 'm' = 1.75GB; };
'A2' = @{ 'c' = 2; 'm' = 3.5GB; };
'A3' = @{ 'c' = 4; 'm' = 7GB; };
'A4' = @{ 'c' = 8; 'm' = 14GB; };
'A5' = @{ 'c' = 2; 'm' = 14GB };
'A6' = @{ 'c' = 4; 'm' = 28GB };
'A7' = @{ 'c' = 8; 'm' = 56GB };
'A1v2' = @{ 'c' = 1; 'm' = 2GB; };
'A2v2' = @{ 'c' = 2; 'm' = 4GB; };
'A4v2' = @{ 'c' = 4; 'm' = 8GB; };
'A8v2' = @{ 'c' = 8; 'm' = 16GB; };
'A2mv2' = @{ 'c' = 2; 'm' = 16GB; }
'A4mv2' = @{ 'c' = 4; 'm' = 32GB; };
'A8mv2' = @{ 'c' = 8; 'm' = 64GB; };
'D1' = @{ 'c' = 1; 'm' = 3.5GB };
'D2' = @{ 'c' = 2; 'm' = 7GB };
'D3' = @{ 'c' = 4; 'm' = 14GB };
'D4' = @{ 'c' = 8; 'm' = 28GB };
'D1v2' = @{ 'a' = 'D1' }
'D2v2' = @{ 'a' = 'D2' }
'D3v2' = @{ 'a' = 'D3' }
'D4v2' = @{ 'a' = 'D4' }
'D5v2' = @{ 'c' = 16; 'm' = 56GB };
# memory optimized table (just the d's)
'D11' = @{ 'c' = 2; 'm' = 14GB };
'D12' = @{ 'c' = 4; 'm' = 28GB };
'D13' = @{ 'c' = 8; 'm' = 56GB };
'D14' = @{ 'c' = 16; 'm' = 112GB };
'DS11' = @{ 'a' = 'D11' };
'DS12' = @{ 'a' = 'D12' };
'DS13' = @{ 'a' = 'D13' };
'DS14' = @{ 'a' = 'D14' };
'D11v2' = @{ 'a' = 'D11' };
'D12v2' = @{ 'a' = 'D12' };
'D13v2' = @{ 'a' = 'D13' };
'D14v2' = @{ 'a' = 'D14' };
'D15v2' = @{ 'c' = 20; 'm' = 140GB };
'DS11v2' = @{ 'a' = 'D11v2' };
'DS12v2' = @{ 'a' = 'D12v2' };
'DS13v2' = @{ 'a' = 'D13v2' };
'DS14v2' = @{ 'a' = 'D14v2' };
'DS15v2' = @{ 'a' = 'D15v2' };
}
$g = Get-ClusterGroup |? GroupType -eq VirtualMachine | group -Property OwnerNode -NoElement
icm $g.Name {
# import vmsize hash (cannot $using[$using])
$vmsize = $using:vmsize
Get-ClusterGroup |? GroupType -eq VirtualMachine |? OwnerNode -eq $env:COMPUTERNAME |% {
$g = $_
switch ($using:PSCmdlet.ParameterSetName) {
'FullSpec' {
if ($using:MemoryMaximumBytes -eq 0) {
$MemoryMaximumBytes = $MemoryStartupBytes
} else {
$MemoryMaximumBytes = $using:MemoryMaximumBytes
}
if ($using:MemoryMinimumBytes -eq 0) {
$MemoryMinimumBytes = $MemoryStartupBytes
} else {
$MemoryMinimumBytes = $using:MemoryMinimumBytes
}
$memswitch = '-StaticMemory'
$dynamicMemArg = ""
if ($using:DynamicMemory) {
$memswitch = '-DynamicMemory'
$dynamicMemArg += "-MemoryMinimumBytes $MemoryMinimumBytes -MemoryMaximumBytes $MemoryMaximumBytes"
}
if ($g.State -ne 'Offline') {
write-host -ForegroundColor Yellow Cannot alter VM sizing on running VMs "($($_.Name))"
} else {
iex "Set-VM -ComputerName $($g.OwnerNode) -Name $($g.Name) -ProcessorCount $using:ProcessorCount -MemoryStartupBytes $using:MemoryStartupBytes $dynamicMemArg $memswitch"
}
}
'SizeSpec' {
$a = $vmsize[$using:compute].a
if ($a -eq $null) {
$a = $using:compute
}
Set-VM -ComputerName $($g.OwnerNode) -Name $($g.Name) -ProcessorCount $vmsize[$a].c -MemoryStartupBytes $vmsize[$a].m -StaticMemory
}
}
}
}
@@ -0,0 +1,506 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
[string] $addspec = "base",
[string] $runtemplate = "c:\clusterstorage\collect\control\run-sweeptemplate.ps1",
[string] $runfile = "c:\clusterstorage\collect\control\run-sweep.ps1",
[string[]] $labeltemplate = @('b','t','o','w','p','-$addspec'),
[Parameter(Mandatory =$true)]
[int[]] $b,
[Parameter(Mandatory =$true)]
[int[]] $t,
[Parameter(Mandatory =$true)]
[int[]] $o,
[Parameter(Mandatory =$true)]
[int[]] $w,
[int[]] $iops = $null,
[ValidateSet('r','s','si')]
[string[]] $p = 'r',
[ValidateRange(1,[int]::MaxValue)]
[int] $d = 60,
[ValidateRange(0,[int]::MaxValue)]
[int] $warm = 60,
[ValidateRange(0,[int]::MaxValue)]
[int] $cool = 60,
[string[]] $pc = $null,
[switch] $midcheck = $false
)
#############
# a single named variable with a set of values
class variable {
[int] $_ordinal
[object[]] $_list
[string] $_label
variable([string] $label, [object[]] $list)
{
$this._list = $list
$this._ordinal = 0
$this._label = $label
}
# current value of the variable ("red"/"blue"/"green")
[object] value()
{
if ($this._list.count -gt 0) {
return $this._list[$this._ordinal]
} else {
return $null
}
}
# label/name of the variable ("color")
[object] label()
{
return $this._label
}
# increment to the next member, return whether a logical carry
# has occured (overflow)
[bool] increment()
{
# empty list passes through
if ($this._list.Count -eq 0) {
return $true
}
# non-empty list, increment
$this._ordinal += 1
if ($this._ordinal -eq $this._list.Count) {
$this._ordinal = 0
return $true
}
return $false
}
# back to initial state
[void] reset()
{
$this._ordinal = 0
}
}
# a set of variables which allows enumeration of their combinations
# this behaves as a numbering system indexing the respective variables
# order is not specified
class variableset {
[hashtable] $_set = @{}
variableset([variable[]] $list)
{
$list |% { $this._set[$_.label()] = $_ }
$this._set.Values |% { $_.reset() }
}
# increment the enumeration
# returns true if the enumeration is complete
[bool] increment()
{
$carry = $false
foreach ($v in $this._set.Values) {
# if the variable returns the carry flag, increment
# the next, and so forth
$carry = $v.increment()
if (-not $carry) { break }
}
# done if the most significant carried over
return $carry
}
# enumerator of all variables
[object] getset()
{
return $this._set.Values
}
# return value of specific variable
[object] get([string]$label)
{
return $this._set[$label].value()
}
# return a label representing the current value of the set, following the input label template
# add a leading '-' to get a seperator
# use a leading '$' to eliminate repetition of label (just produce value)
[string] label([string[]] $template)
{
return $($template |% {
$str = $_
$pfx = ''
$done = $false
$norep = $false
do {
switch ($str[0])
{
'-' {
$pfx = '-'
$str = $str.TrimStart('-')
}
'$' {
$norep = $true
$str = $str.TrimStart('$')
}
default {
$done = $true
}
}
} while (-not $done)
$lookstr = $str
if ($norep) {
$str = ''
}
# only produce labels for non-null values
if ($this._set[$lookstr].value() -ne $null) {
"$pfx$str" + $this._set[$lookstr].value()
}
}) -join $null
}
}
#############
function start-logman(
[string] $computer,
[string] $name,
[string[]] $counters
)
{
$f = "c:\perfctr-$name-$computer.blg"
$null = logman create counter "perfctr-$name" -o $f -f bin -si 1 --v -c $counters -s $computer
$null = logman start "perfctr-$name" -s $computer
write-host "performance counters on: $computer"
}
function stop-logman(
[string] $computer,
[string] $name,
[string] $path
)
{
$f = "c:\perfctr-$name-$computer.blg"
$null = logman stop "perfctr-$name" -s $computer
$null = logman delete "perfctr-$name" -s $computer
xcopy /j $f $path
del -force $f
write-host "performance counters off: $computer"
}
function new-runfile(
[variableset] $vs
)
{
# apply current subsitutions to produce a new run file
gc $runtemplate |% {
$line = $_
foreach ($v in $vs.getset()) {
# non-null goes in as is, null goes in as evaluatable $null
if ($v.value() -ne $null) {
$vsub = $v.value()
} else {
$vsub = '$null'
}
$line = $line -replace "__$($v.label())__",$vsub
}
$line
} | out-file $runfile -Encoding ascii -Width 9999
}
function show-run(
[variableset] $vs
)
{
# show current substitions (minus the underscore bracketing)
write-host -fore green RUN SPEC `@ (get-date)
foreach ($v in $vs.getset()) {
if ($v.value() -ne $null) {
$vsub = $v.value()
} else {
$vsub = '$null'
}
write-host -fore green "`t$($v.label()) = $($vsub)"
}
}
function get-runduration(
[variableset] $vs
)
{
$vs.get('d') + $vs.get('Warm') + $vs.get('Cool')
}
function get-doneflags(
[switch] $assertnone = $false
)
{
$assert = $false
$tries = 0
do {
if ($tries -gt 0) {
sleep 1
}
# increment attempts
$tries += 1
# capture start of the iteration
$t0 = (get-date)
# count number of done flags which agree with completion of the current go epoch
$good = 0
dir $done |% {
$thisdone = gc $_ -ErrorAction SilentlyContinue
if ($thisdone -eq $goepoch) {
# if asserting that none should be complete, this would be an error!
if ($assertnone) {
write-host -fore red ERROR: $_.basename is already done
$assert = $true
}
$good += 1
}
}
# color status message per condition
if ($assert -or $good -ne $vms) {
$color = 'yellow'
} else {
$color = 'green'
}
$t1 = (get-date)
$tdur = $t1 - $t0
write-host -fore $color done check iteration $tries with "$good/$vms" `@ $t1 "($('{0:F2}' -f $tdur.totalseconds) seconds)"
# loop if not asserting, not all vms are done, and still have timeout to use
} while ((-not $assertnone) -and $good -ne $vms -and $tries -lt $timeout)
# return assertion status?
if ($assertnone) {
return (-not $assert)
}
# return incomplete run failure
if ($good -ne $vms) {
write-host -fore red ABORT: only received "$good/$vms" completions in $timeout attempts `@ (get-date)
return $false
}
# all worked!
return $true
}
function do-run(
[variableset] $vs
)
{
# apply specified run parameters. note null is ignored.
show-run $vs
write-host -fore yellow Generating new runfile `@ (get-date)
new-runfile $vs
# if we do not have a pause to clear, need the sleep here since go
# will release the fleet.
# smb fileinfo cache +5 seconds (so that fleet will see updated timestamp)
if (-not ($checkpause -and (check-pause -isactive:$true))) {
$script:checkpause = $false
sleep 15
}
write-host START Go Epoch: $goepoch `@ (get-date)
echo $goepoch > $go
# release any active pause on first loop
if ($checkpause) {
write-host CLEAR PAUSE `@ (get-date)
# same sleep need, pause will release the fleet
$script:checkpause = $false
sleep 15
# capture time zero prior to clear - can take time
$t0 = get-date
clear-pause
} else {
# capture time zero
$t0 = get-date
}
# start performance counter capture
if ($pc -ne $null) {
$curpclabel = $vs.label($labeltemplate)
icm (get-clusternode) -ArgumentList (get-command start-logman) {
param($fn)
set-item -path function:\$($fn.name) -value $fn.definition
start-logman $env:COMPUTERNAME $using:curpclabel $using:pc
}
}
######
# sleep half, check for false done if possible (clear can take time/short runs), continue
$sleep = get-runduration $vs
$t1 = get-date
$td = $t1 - $t0
if ($midcheck) {
$remainingsleep = $sleep/2 - $td.TotalSeconds
if ($remainingsleep -gt 0) {
write-host SLEEP TO MID-RUN "($('{0:F2}' -f $remainingsleep) seconds)" `@ (get-date)
sleep $remainingsleep
}
if ($td.TotalSeconds -lt ($sleep - 5)) {
write-host MID-RUN CHECK Go Epoch: $goepoch `@ (get-date)
# check for early completions, assert none are done yet
if (-not (get-doneflags -assertnone:$true)) {
return $false
}
write-host -fore green MID-RUN CHECK PASS Go Epoch: $goepoch `@ (get-date)
}
# capture time and sleep for the remaining interval
$t1 = get-date
$td = $t1 - $t0
$remainingsleep = $sleep - $td.TotalSeconds
if ($remainingsleep -gt 0) {
write-host SLEEP TO END "($('{0:F2}' -f $remainingsleep) seconds)" `@ (get-date)
sleep $remainingsleep
}
} else {
sleep ($sleep - $td.TotalSeconds)
}
######
# stop performance counter capture
if ($pc -ne $null) {
icm (get-clusternode) -ArgumentList (get-command stop-logman) {
param($fn)
set-item -path function:\$($fn.name) -value $fn.definition
stop-logman $env:COMPUTERNAME $using:curpclabel "C:\ClusterStorage\collect\control\result"
}
}
if (-not (get-doneflags)) {
return $false
}
# advance go epoch
$script:goepoch += 1
return $true
}
#############
$vms = (get-clustergroup |? GroupType -eq VirtualMachine |? Name -like "vm-*" |? State -ne Offline).count
# spec location of control files
$go = "c:\clusterstorage\collect\control\flag\go"
$done = "c:\clusterstorage\collect\control\flag\done-*"
$timeout = 120
$checkpause = $true
# ensure we start a new go epoch
$goepoch = 0
$gocontent = gc $go -ErrorAction SilentlyContinue
if ($gocontent -eq '0') {
$goepoch = 1
}
# construct the variable list describing the sweep
############################
############################
## Modify from here down
############################
############################
# add any additional sweep parameters here to match those specified on the command line
# ensure your sweep template script contains substitutable elements for each
#
# __somename__
#
# bracketed by two underscore characters. Consider adding your new parameters to
# the label template so that result files are well-named and distinguishable.
$v = @()
$v += [variable]::new('b',$b)
$v += [variable]::new('t',$t)
$v += [variable]::new('o',$o)
$v += [variable]::new('w',$w)
$v += [variable]::new('p',$p)
$v += [variable]::new('iops',$iops)
$v += [variable]::new('d',$d)
$v += [variable]::new('Warm',$warm)
$v += [variable]::new('Cool',$cool)
$v += [variable]::new('AddSpec',$addspec)
$sweep = [variableset]::new($v)
do {
write-host -ForegroundColor Cyan '---'
$r = do-run $sweep
if (-not $r) {
return
}
} while (-not $sweep.increment())
@@ -0,0 +1,53 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
[string[]] $group = "*",
[int] $number = 0
)
icm (get-clusternode |? State -eq Up) {
$n = $using:number
# failed is an unclean offline tbd root causes (can usually be recovered)
$using:group |% {
# sorted list of vms by vm number
$vms = @(Get-ClusterGroup |? OwnerNode -eq $env:COMPUTERNAME |? GroupType -eq VirtualMachine |? Name -like "vm-$_-*" | sort -Property @{ Expression = { $null = $_.Name -match '-(\d+)$'; [int] $matches[1] }})
# start limited number, if specified, else all
$(if ($n -gt 0 -and $vms.Count -gt $n) {
$vms[0..($n - 1)]
} else {
$vms
}) |? {
$_.State -eq 'Offline' -or $_.State -eq 'Failed'
} | Start-ClusterGroup
}
} | ft -AutoSize
@@ -0,0 +1,53 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
[string[]] $group = @('*'),
[ValidateSet('Save','Shutdown','TurnOff')][string] $method = 'Shutdown'
)
icm (get-clusternode |? State -eq Up) -arg $group,$method {
param([string[]] $group, $method)
$group |% {
$g = Get-ClusterGroup |? GroupType -eq VirtualMachine |? OwnerNode -eq $env:COMPUTERNAME |? Name -like "vm-$_-*" |? State -ne 'Offline'
if ($g) {
# stop-clustergroup currently defaults to vm save
# use remoted stop-vm for the shutdown case
if ($method -eq 'Save') {
$g | Stop-ClusterGroup
} elseif ($method -eq 'TurnOff') {
Stop-VM -ComputerName $env:COMPUTERNAME -Name $g.Name -Force -TurnOff
} else {
Stop-VM -ComputerName $env:COMPUTERNAME -Name $g.Name -Force
}
}
}
} | ft -AutoSize
@@ -0,0 +1,212 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
$outfile = "result-cputarget.tsv",
$cputargets = $(throw "please specify a set of cpu percentage targets"),
$addspec = ""
)
$result = "C:\ClusterStorage\collect\control\result"
# count the number of vms in the configuration
$vms = (Get-ClusterResource |? ResourceType -eq 'Virtual Machine' | measure).Count
# target cpu utilization and qos to +/- given percentage
$cputargetwindow = 5
$qoswindow = 5
# clean result file and set column headers
del -Force $outfile -ErrorAction SilentlyContinue
'WriteRatio','QOS','AVCPU','IOPS','AVRLat','AVWLat' -join "`t" > $outfile
# make qos policy and reset
Get-StorageQosPolicy -Name SweepQos -ErrorAction SilentlyContinue | Remove-StorageQosPolicy -Confirm:$false
New-StorageQosPolicy -Name SweepQoS -MinimumIops $qos -MaximumIops $qos -PolicyType Dedicated
set-storageqos -policyname SweepQoS
function is-within(
$value,
$target,
$percentage
)
{
($value -ge ($target - ($target*($percentage/100))) -and
$value -le ($target + ($target*($percentage/100))))
}
function get-pc(
[string] $blg,
[int] $center,
[string] $ctr
)
{
# get central n samples of a performance counter's sample
$pc = Import-Counter -Path $blg -Counter "\\*\$ctr"
$t0 = ($pc.length - $center)/2
$t1 = $t0 + $center - 1
($pc[$t0 .. $t1].CounterSamples.CookedValue | measure -Average).Average
}
$pc = @('\Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time',
'\Processor Information(_Total)\% Processor Performance',
'\Cluster CSVFS(_Total)\reads/sec',
'\Cluster CSVFS(_Total)\avg. sec/read',
'\Cluster CSVFS(_Total)\writes/sec',
'\Cluster CSVFS(_Total)\avg. sec/write')
# limit the number of attempts per sweep (mix) to 4 per targeted cpu util
$sweeplimit = ($cputargets.count * 4)
foreach ($w in 0,10,30) {
# track measured qos points, starting at given value
$h = @{}
$qosinitial = $qos = 400
foreach ($cputarget in $cputargets) {
# move the qos window using previous run information
if ($cputarget -ne $cputargets[0]) {
$nextqos = [int](($cputarget*$iops/$avcpu)/$vms)
$qos = $nextqos
}
write-host -ForegroundColor Cyan Starting outer loop with CPU target $cputarget and initial QoS $qos
do {
# failsafes
if ($h[$qos]) { write-host -ForegroundColor Red already measured $qos; break }
if ($h.Keys.Count -ge $sweeplimit) { write-host -ForegroundColor Red $sweeplimit tries giving up; break }
Set-StorageQosPolicy -Name SweepQoS -MaximumIops $qos
write-host -fore Cyan Starting loop with QoS target $qos
$curaddspec = "$($addspec)w$($w)qos$qos"
start-sweep.ps1 -addspec $curaddspec -b 4 -o 32 -t 1 -w $w -p r -d 60 -warm 15 -cool 15 -pc $pc
# HACKHACK bounce collect
Get-ClusterSharedVolume |? { $_.SharedVolumeInfo.FriendlyVolumeName -match 'collect' } | Move-ClusterSharedVolume
sleep 1
# get average IOPS at DISKSPD
$iops = $(dir $result\*.xml |% {
$x = [xml](gc $_)
($x.Results.TimeSpan.Iops.Bucket | measure -Property Total -Average).Average
} | measure -Sum).Sum
# get average cpu utilization for central 60 seconds of each node
# get average of all nodes
$avcpu = $(dir $result\*.blg |% {
$center = 60
$trt = get-pc $_ $center '\Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time'
$ppc = get-pc $_ $center '\Processor Information(_Total)\% Processor Performance'
$trt*$ppc/100
} | measure -Average).Average
# get average latency for central 60 seconds, all nodes
# note we must aggregate the product of av latency and iops per node to get total
# latency, and then divide by total iops to get whole-cluster average.
$csvrtotal = 0
$csvwtotal = 0
($avrlat,$avwlat) = $(dir $result\*.blg |% {
$csvrlat = get-pc $_ $center '\Cluster CSVFS(_Total)\avg. sec/read'
$csvwlat = get-pc $_ $center '\Cluster CSVFS(_Total)\avg. sec/write'
$csvr = get-pc $_ $center '\Cluster CSVFS(_Total)\reads/sec'
$csvw = get-pc $_ $center '\Cluster CSVFS(_Total)\writes/sec'
$csvrtotal += $csvr
$csvwtotal += $csvw
[pscustomobject] @{ 'avrtime' = $csvrlat*$csvr; 'avwtime' = $csvwlat*$csvw }
} | measure -Sum -Property avrtime,avwtime).Sum
$avrlat /= $csvrtotal
$avwlat /= $csvwtotal
# capture results
$w,$qos,$avcpu,$iops,$avrlat,$avwlat -join "`t" >> $outfile
# archive results
compress-archive -Path $(dir $result\* -Exclude *.zip) -DestinationPath $result\$curaddspec.zip
dir $result\* -Exclude *.zip | del
write-host Archived results to $result\$curaddspec.zip
# stop within targetwindow% of cpu (+/- % of target)
if (is-within $avcpu $cputarget $cputargetwindow) {
write-host -ForegroundColor Green "Stopping in target window at $('{0:N2}' -f $avcpu) with QoS $qos"
break
}
# assume cpu and qos have a linear relationship - extrapolate to target
# could do a direct linear fit of measurements so far
$nextqos = [int]($cputarget*$qos/$avcpu)
# stop if next qos target is within qoswindow% of any previous measurement
$inwindow = $false
foreach ($previous in $h.keys) {
if (is-within $nextqos $previous $qoswindow) {
$inwindow = $true
break
}
}
if ($inwindow) {
write-host -ForegroundColor Yellow "Stopping in window of prior measurement at $('{0:N2}' -f $avcpu) with QoS $qos"
break
}
# stop if next qos target is less than initial
if ($nextqos -lt $qosinitial) {
write-host -ForegroundColor Red "Stopping with underflow targeting $nextqos less than initial $qosinitial"
break
}
write-host -ForegroundColor Cyan "Loop acheived $('{0:N2}' -f $avcpu) @ QoS $qos v. target $cputarget - next loop targeting QoS $nextqos"
# record this datapoint as measured, move along to the next
$h[$qos] = 1
$qos = $nextqos
} while ($true)
}
}
set-storageqos -policyname $null
@@ -0,0 +1,637 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
[switch] $CleanOperationalIssues = $false
)
function new-namedblock(
[string] $name,
[scriptblock] $block,
[switch] $nullpass = $true,
[int] $mustbe = -1
)
{
new-object psobject -Property @{
'Name' = $name;
'Block' = $block;
'NullPass' = $nullpass;
'MustBe' = $mustbe
}
}
function write-blocktitle(
[string[]] $s
)
{
write-host -fore cyan ('*'*20) $s
}
function display-jobs()
{
BEGIN { $j = @() }
PROCESS { $j += $_ }
END {
# consume job results
$null = $j | wait-job
$j | sort -Property Name |% {
write-blocktitle $_.Name,('({0:F1}s)' -f ($_.PsEndTime - $_.PsBeginTime).TotalSeconds)
$_ | receive-job
$_ | remove-job
}
}
}
# script block containers for helper functions
$evfns = {
function get-fltevents(
[decimal] $timedeltams,
[string] $provider,
[string[]] $evid,
[scriptblock] $flt = { $true },
[string] $source = $null
)
{
# simple query vs. a provider for a single event within some timedelta of current (in ms)
# optional addition of a source provider name filter (for system log filtering)
$qstr = @"
<QueryList>
<Query Id="0" Path="_PROV_">
<Select Path="_PROV_">*[System[_SOURCE_(_EVENTS_) and TimeCreated[timediff(@SystemTime) &lt;= _MS_]]]</Select>
</Query>
</QueryList>
"@
$srcstr = @"
Provider[@Name='_SOURCE_'] and
"@
$events = ($evid |% {
"EventID=$_"
}) -join " or "
$query = $qstr -replace '_MS_',$timedeltams -replace '_PROV_',$provider -replace '_EVENTS_',$events
if ($source) {
$query = $query -replace '_SOURCE_',($srcstr -replace '_SOURCE_',$source)
} else {
$query = $query -replace '_SOURCE_',''
}
Get-WinEvent -FilterXml $query -ErrorAction SilentlyContinue | Where-Object -FilterScript $flt
}
}
$fns = {
function do-clustersymmetry(
[object] $gather,
[object[]] $filters,
[switch] $saygather = $false
)
{
# deserialization note: the scriptblocks on the inputs are downconverted to strings
# so as a result we must reinstantiate
# assert all nodes agree on object counts from the provided gather element
$data = icm (get-clusternode |? State -eq up) ([scriptblock]::create($gather.block))
if ($saygather) {
write-host -ForegroundColor yellow ('*'*15) $gather.name
}
foreach ($f in $filters) {
write-host -fore yellow ('*'*10) $f.name
$r = $data | where-object -FilterScript ([scriptblock]::create($f.block))
# group results by node
$nodeg = $r | group -property pscomputername -NoElement
# regular symmetery
# if grouping by count (per node) yields more than one element, we know some nodes have different counts
# i.e., not all are 60: perhaps one is 58, etc.
# this is always failure.
if ($nodeg -ne $null -and ($nodeg | group -Property Count | measure).count -ne 1) {
write-host -ForegroundColor Red Fail
$nodeg | sort -Property Name,Count | ft -autosize
} else {
if ($nodeg -ne $null) {
# if no enforced count or count is correct, pass
if ($f.mustbe -lt 0 -or $f.mustbe -eq $nodeg[0].Count) {
write-host -ForegroundColor Green Pass with $nodeg[0].Count per node
} else {
# enforced count not correct
write-host -ForegroundColor Red Fail - required count of $f.mustbe not consistent on each node
$nodeg | sort -Property Name,Count | ft -autosize
}
} elseif ($f.nullpass) {
write-host -ForegroundColor Green Pass with none on any node
} else {
write-host -ForegroundColor Red Fail with none on any node
}
}
}
}
}
# Detect RDMA its type (by manufacturer) so that, if needed, we can assert QoS/Cos for RoCE
$netadapters = Get-NetAdapterRdma |? Enabled | Get-NetAdapter
$rdma = $false
$roce = $false
$rocematch = 'Mellanox'
if ($netadapters) {
write-host -fore green Detected RDMA adapters: will require RDMA
$rdma = $true
# will need a tweak as additional non-Mellanox RoCE arrive (and/or if IB)
$qroce = $netadapters |? DriverProvider -match $rocematch
if ($qroce) {
$drvdesc = $qroce[0].DriverDescription
write-host -fore green Detected $qroce[0].DriverProvider RDMA adapters: will require RoCE configuration
write-host -fore green Adapter Description: $drvdesc
$roce = $true
}
}
### Basic Health Checks: serialized
$j = @()
$j += start-job -Name 'Basic Health Checks' {
# nodes up
$cn = Get-ClusterNode
if ($cn.count -eq $($cn |? State -eq Up).count) {
write-host -ForegroundColor Green All cluster nodes Up
} else {
write-host -ForegroundColor Red Following cluster nodes not Up:
$cn |? State -ne Up
}
# node uptime
$o = icm ($cn |? State -eq Up) {
$w = gwmi win32_operatingsystem
$w.ConvertToDateTime($w.localdatetime) - $w.ConvertToDateTime($w.lastbootuptime)
}
$reboots = $o |? TotalHours -lt 1
if ($reboots.length -and $reboots.length -ne $o.length) {
write-host -ForegroundColor Yellow WARNING: $reboots.length nodes have rebooted in the last hour. Ensure that
write-host -ForegroundColor Yellow `t no unexpected events are occuring in the cluster.
}
write-host -ForegroundColor Green Cluster node uptime:
$o | sort PsComputerName | ft PsComputerName,@{ Label="Uptime"; Expression={"{0}d:{1:00}h:{2:00}m.{3:00}s" -f $_.Days,$_.Hours,$_.Minutes,$_.Seconds}}
# subsystem check
$ss = Get-StorageSubSystem |? Model -eq 'Clustered Windows Storage'
if (($ss | measure).count -ne 1) {
write-host -ForegroundColor Red Expected single clustered storage subsystem, found:
$ss | ft -autosize
return
}
$ssuh = $ss |? HealthStatus -ne Healthy
if ($ssuh) {
write-host -ForegroundColor Red WARNING: clustered storage subsystem is not healthy
$ssuh | ft -AutoSize
write-host -ForegroundColor Red Output of Debug-StorageSubSystem follows
$ssuh | Debug-StorageSubSystem | fl
} else {
write-host -ForegroundColor Green Clustered storage subsystem Healthy
}
# pool health
$p = $ss | Get-StoragePool |? IsPrimordial -ne $true |? HealthStatus -ne Healthy
if ($p -eq $null) {
write-host -ForegroundColor Green All pools Healthy
} else {
write-host -ForegroundColor Red Following pools not Healthy:
$p | ft -autosize
}
}
# vd health
$j += start-job -name 'Virtual Disk Health' {
$ss = Get-StorageSubSystem |? Model -eq 'Clustered Windows Storage'
$vd = $ss | Get-VirtualDisk |? HealthStatus -ne Healthy
if ($vd -eq $null) {
write-host -fore green All operational virtual disks Healthy
} else {
write-host -fore red Following virtual disks not Healthy:
$vd | ft -autosize
}
}
# disk state
$j += start-job -name 'Physical Disk Health' {
$pd = Get-StorageSubSystem |? Model -eq 'Clustered Windows Storage' | Get-PhysicalDisk
$nonauto = $pd |? Usage -notmatch 'Journal|Auto-Select'
if ($nonauto) {
write-host -fore yellow WARNING: there are physical disks which are not auto-select/journal for usage.
write-host -fore yellow `t It is possible that while storage resilience has been restored from
write-host -fore yellow `t a failure, it is no longer evenly distributed between cluster nodes.
write-host -fore yellow `t Consider recovering before doing performance/operational work.
$nonauto | ft -autosize
} else {
write-host -fore green All physical disks are in normal auto-select or journal state
}
}
# consolidated op issues - should logically split?
$j += start-job -name 'Operational Issues and Storage Jobs' -ArgumentList $CleanOperationalIssues {
param( $CleanOperationalIssues )
$ev = icm (get-clusternode) {
get-winevent -LogName Microsoft-Windows-Storage-Storport/Operational |? Id -eq 502 |% {
$ex = [xml]$_.ToXml()
$guid = ($ex.Event.EventData.Data |? Name -eq ClassDeviceGuid).'#text'
$_ | Add-Member -NotePropertyName DeviceGuid -NotePropertyValue $guid -PassThru
}
}
if ($ev) {
write-host -fore yellow WARNING: unresponsive device events have been logged by storport.
write-host -fore yellow `tThese may correspond to retired devices, and should be investigated.
$ev | ft -autosize PsComputerName,TimeCreated,Id,DeviceGuid,Message
write-host -fore yellow Corresponding devices by Device GUID:
$d = Get-StorageSubSystem Cluster* | Get-StoragePool |? IsPrimordial -eq $false | Get-PhysicalDisk
$ev |% {
$d |? ObjectId -match "PD:$($_.DeviceGuid)"
} | ft -AutoSize
}
# look for livekernelreport and/or bugcheck dumps
$dmps = icm (get-clusternode |? State -eq Up) {
$obj = @()
$obj += dir $($env:windir + "\livekernelreports")
$obj += dir $($env:windir + "\minidump") -ErrorAction SilentlyContinue
$obj += dir $($env:windir + "\memory.dmp") -ErrorAction SilentlyContinue
if ($using:CleanOperationalIssues -and $obj.count -gt 0) {
$obj | del -Force -Recurse -ErrorAction SilentlyContinue
}
$obj
} | sort -property PsParentPath,LastWriteTime,PsComputerName
if ($dmps) {
if ($CleanOperationalIssues) {
write-host -fore red NOTE: the following failure reports were forcibly removed
} else {
write-host -fore yellow WARNING: there are failure reports that may require triage
}
$dmps | ft -AutoSize
}
# Storage Jobs
$sj = get-storagejob
if ($sj |? JobState -ne Completed) {
write-host -ForegroundColor red WARNING: there are active storage jobs running. Investigate the root cause before continuing.
$sj | ft -autosize
} else {
write-host -fore green No storage rebuild or regeneration jobs are active
}
}
# SMB Connectivity Error Check
$fltblk = {
param( $ev, $warncol, $warn )
$r = icm (get-clusternode |? State -eq Up) -ArgumentList @((get-command get-fltevents), $ev) {
param($fn, $ev)
set-item -path function:\$($fn.name) -value $fn.definition
$flttcp = {
# report tcp (type 1) connectivity events
$x = [xml] $_.ToXml()
[int]($x.Event.EventData.Data |? Name -eq 'ConnectionType').'#text' -eq 1
}
$fltrdma = {
# report rdma (type 2) connectivity events
$x = [xml] $_.ToXml()
[int]($x.Event.EventData.Data |? Name -eq 'ConnectionType').'#text' -eq 2
}
$last5 = (1000*60*5)
$lasthour = (1000*60*60)
$lastday = (1000*60*60*24)
new-object psobject -Property @{
'RDMA Last5Min' = (get-fltevents -flt $fltrdma -timedeltams $last5 -provider "Microsoft-Windows-SmbClient/Connectivity" -evid $ev).count;
'RDMA LastHour' = (get-fltevents -flt $fltrdma -timedeltams $lasthour -provider "Microsoft-Windows-SmbClient/Connectivity" -evid $ev).count;
'RDMA LastDay' = (get-fltevents -flt $fltrdma -timedeltams $lastday -provider "Microsoft-Windows-SmbClient/Connectivity" -evid $ev).count;
'TCP Last5Min' = (get-fltevents -flt $flttcp -timedeltams $last5 -provider "Microsoft-Windows-SmbClient/Connectivity" -evid $ev).count;
'TCP LastHour' = (get-fltevents -flt $flttcp -timedeltams $lasthour -provider "Microsoft-Windows-SmbClient/Connectivity" -evid $ev).count;
'TCP LastDay' = (get-fltevents -flt $flttcp -timedeltams $lastday -provider "Microsoft-Windows-SmbClient/Connectivity" -evid $ev).count;
}
}
$hdr = (($r[0] | gm -MemberType NoteProperty |? Definition -like 'int*').Name | sort)
$rdmafail = ($r |% { $row = $_; $hdr |? {$_ -like 'RDMA*' } |% { $row.$_ }} | measure -sum).sum -ne 0
if ($rdmafail) {
write-host -ForegroundColor $warncol $warn
}
$r | sort -Property PsComputerName | ft -Property (@('PsComputerName') + $hdr)
}
$w = @"
WARNING: the SMB Client is receiving RDMA disconnects. This is an error whose root"
`t cause may be PFC/CoS misconfiguration (RoCE) on hosts or switches, physical"
`t issues (ex: bad cable), switch or NIC firmware issues, and will lead to severely"
`t degraded performance. Additional triage is included in other tests."
"@
$j += start-job -name 'SMB Connectivity Error Check - Disconnect Failures' -ArgumentList 30804,([ConsoleColor]'Red'),$w -InitializationScript $evfns $fltblk
$w = @"
WARNING: the SMB Client is receiving RDMA connect errors. This is an error whose root
`t cause may be actual lack of connectivity or fundamental problems with the RDMA
`t network fabric. Please inspect especially if in the Last5 bucket.
"@
$j += start-job -name 'SMB Connectivity Error Check - Connect Failures' -ArgumentList 30803,([ConsoleColor]'Yellow'),$w -InitializationScript $evfns $fltblk
if ($roce) {
$j += start-job -name 'RoCE: Mellanox Disable Check' -InitializationScript $evfns {
$r = icm (get-clusternode |? State -eq Up) -ArgumentList (get-command get-fltevents) {
param($fn)
set-item -path function:\$($fn.name) -value $fn.definition
$r = get-fltevents -timedeltams (1000*60*60*24*30) -provider 'System' -source 'mlx4eth63' -evid 35
}
if ($r) {
write-host -ForegroundColor red WARNING: Mellanox indicates that RDMA has been disabled due to mis/non-configuration
write-host -ForegroundColor red `t of Priority Flow Control at some point in the past 30 days. Unless this has been recently
write-host -ForegroundColor red `t "corrected," RDMA may be down.
write-host -ForegroundColor red Most recent event "(System log)" follows
$r[0] | fl
} else {
write-host -ForegroundColor Green Pass
}
}
$j += start-job -name 'RoCE: Mellanox Error Check' {
$r = $null
$pc = $null
switch ($using:drvdesc) {
"Mellanox ConnectX-3 Pro Ethernet Adapter" {
$pc = @{
'\Mellanox Adapter Diagnostic Counters(_Total)\Responder Out-of-order Sequence Received' = 'Out Of Order';
'\Mellanox Adapter Traffic Counters(_Total)\Packets Received Bad CRC Error' = 'Rec BadCRC';
'\Mellanox Adapter Traffic Counters(_Total)\Packets Received Frame Length Error' = 'Rec FrmLenErr';
'\Mellanox Adapter Traffic Counters(_Total)\Packets Received Symbol Error' = 'Rec SymlErr';
'\Mellanox Adapter Traffic Counters(_Total)\Packets Received Discarded' = 'Rec Discard';
'\Mellanox Adapter Traffic Counters(_Total)\Packets Outbound Discarded' = 'Outbnd Discard'
'\Mellanox Adapter Traffic Counters(_Total)\Packets Outbound Errors' = 'Outbnd Err';
}
}
"Mellanox ConnectX-4 Adapter" {
$pc = @{
'\Mellanox WinOF-2 Diagnostics(_Total)\Responder out of order sequence received' = 'Out Of Order';
'\Mellanox WinOF-2 Port Traffic(_Total)\Packets Received Bad CRC Error' = 'Rec BadCRC';
'\Mellanox WinOF-2 Port Traffic(_Total)\Packets Received Frame Length Error' = 'Rec FrmLenErr';
'\Mellanox WinOF-2 Port Traffic(_Total)\Packets Received Symbol Error' = 'Rec SymlErr';
'\Mellanox WinOF-2 Port Traffic(_Total)\Packets Received Discarded' = 'Rec Discard';
'\Mellanox WinOF-2 Port Traffic(_Total)\Packets Outbound Discarded' = 'Outbnd Discard'
'\Mellanox WinOF-2 Port Traffic(_Total)\Packets Outbound Errors' = 'Outbnd Err';
}
}
default {
write-host -ForegroundColor Red "Unknown adapter type: $($using:drvdesc)"
}
}
# no counters, no results
if ($pc -ne $null) {
$r = icm (get-clusternode |? State -eq Up) -ArgumentList $pc {
param($pc)
$c = get-counter ($pc.Keys |% { $_ }) -ErrorAction SilentlyContinue
if ($c) {
$o = new-object psobject -Property @{ 'Errors' = $false }
$c.CounterSamples | sort -Property Path |% {
if ($_.path -match '\\\\[^\\]+(\\.*$)') {
$o | Add-Member -NotePropertyName $pc[$matches[1]] -NotePropertyValue $_.CookedValue
if ($_.CookedValue -ne 0) { $o.Errors = $true }
}
}
$o
}
}
}
if ($r.length -ne (get-clusternode |? State -eq Up).length) {
write-host -ForegroundColor Yellow WARNING: retransmit statistics not available from all nodes. Ensure driver updates applied.
write-host -ForegroundColor Yellow `t $r.length nodes responded out of $((get-clusternode |? Up).length)
}
if ($r |? Errors) {
write-host -ForegroundColor Red "WARNING: Any non-zero error counters may indicate physical or switch/NIC"
write-host -ForegroundColor Red "`t issues, likely leading to packet drops and retransmits, which will degrade"
write-host -ForegroundColor Red "`t performance. At high enough rates they can lead to SMB connection drops."
} else {
write-host -ForegroundColor Green "Pass - no errors detected"
}
$hdr = @( 'PsComputerName' )
$hdr += $($pc.Values |% { $_ })
$r | sort -property PsComputerName | ft -Property $hdr
}
}
## Begin Symmetry Checks
$totalf = new-namedblock 'Total' { $true }
$totalf_nonull = new-namedblock 'Total' { $true } -nullpass:$false
###
$t = new-namedblock 'Clusport Device Symmetry Check' { gwmi -namespace root\wmi ClusportDeviceInformation }
$f = @($totalf)
$f += ,(new-namedblock 'Disk Type' { $_.DeviceType -eq 0} -nullpass:$false)
# temporarily remove hybrid check - the attribute is not synchronously updated
# and so may be (harmlessly) inaccurate for a period of time in early configuration
#$f += ,(new-namedblock 'Hybrid Media' { $_.DeviceAttribute -band 0x4})
$f += ,(new-namedblock 'Solid/Non-Rotational Media' { $_.DeviceAttribute -band 0x8})
$f += ,(new-namedblock 'Enclosure Type' { $_.DeviceType -eq 1} -nullpass:$false)
$f += ,(new-namedblock 'Virtual' { $_.DeviceAttribute -band 0x1} -nullpass:$true)
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
###
$t = new-namedblock 'Physical Disk View Symmetry Check' { Get-StorageSubSystem |? Model -eq 'Clustered Windows Storage' | Get-PhysicalDisk }
$f = @($totalf)
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
###
$t = new-namedblock 'Enclosure View Symmetry Check' { Get-StorageSubSystem |? Model -eq 'Clustered Windows Storage' | Get-StorageEnclosure }
$f = @($totalf)
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
###
$t = new-namedblock 'SMB SBL Multichannel Symmetry Check' { Get-SmbMultichannelConnection -SmbInstance SBL }
$f = @($totalf)
$f += ,(new-namedblock 'RDMA Capable' { $_.ClientRdmaCapable -and $_.ServerRdmaCapable } -nullpass:$(-not $rdma))
$f += ,(new-namedblock 'Selected & Non-Failed' { $_.Selected -and -not $_.Failed } -nullpass:$(-not $rdma))
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
###
$t = new-namedblock 'SMB CSV Multichannel Symmetry Check' { Get-SmbMultichannelConnection -SmbInstance CSV }
$f = @($totalf)
$f += ,(new-namedblock 'RDMA Capable' { $_.ClientRdmaCapable -and $_.ServerRdmaCapable } -nullpass:$(-not $rdma))
$f += ,(new-namedblock 'Selected & Non-Failed' { $_.Selected -and -not $_.Failed } -nullpass:$(-not $rdma))
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
###
if ($rdma) {
# rdma gives us an easy way of identifying a set of adapters to do this test with.
# it would be good to extend this more generally
$t = @()
$t += new-namedblock 'RDMA Adapter IP Check' { Get-NetAdapterRdma |? Enabled | Get-NetAdapter |? HardwareInterface | Get-NetIPAddress -ErrorAction SilentlyContinue |? AddressState -eq 'Preferred' }
$t += new-namedblock 'RDMA Adapter (Virtual) IP Check' { Get-NetAdapterRdma |? Enabled | Get-NetAdapter |? { -not $_.HardwareInterface } | Get-NetIPAddress -ErrorAction SilentlyContinue |? AddressState -eq 'Preferred' }
$t += new-namedblock 'RDMA Adapter (Physical) IP Check' { Get-NetAdapterRdma |? Enabled | Get-NetAdapter |? HardwareInterface | Get-NetIPAddress -ErrorAction SilentlyContinue |? AddressState -eq 'Preferred' }
$f = @($totalf)
$j += start-job -InitializationScript $fns -Name $t[0].name {
$using:t |% { do-clustersymmetry $_ $using:f -saygather:$true }
}
}
###
$t = new-namedblock 'RDMA Adapters Symmetry Check' { Get-NetAdapterRdma |? Enabled | Get-NetAdapter }
$f = @($totalf)
$f += ,(new-namedblock 'Operational' { $_.Speed -gt 0 } -nullpass:$(-not $rdma))
$f += ,(new-namedblock 'Up' { $_.ifOperStatus -eq 'Up' } -nullpass:$(-not $rdma))
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
###
if ($roce) {
# assert SMB Direct policy defined
$t = new-namedblock 'RoCE/QoS Configuration for SMB Direct' { Get-NetQosPolicy }
$f = @($totalf_nonull)
$f += ,(new-namedblock 'SMB Direct' { $_.NetDirectPort -eq 445 -and $_.PriorityValue -ne 0 } -nullpass:$false)
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
# this is strictly insufficient, should ensure the enabled one is the same as that defined for SMB Direct
$t = new-namedblock 'RoCE/CoS Definitions' { Get-NetQosFlowControl }
$f = @($totalf_nonull)
$f += ,(new-namedblock 'Enabled' { $_.Enabled } -nullpass:$false)
$f += ,(new-namedblock 'Disabled' { -not $_.Enabled } -nullpass:$false)
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
<#
#
$t = new-namedblock 'RoCE/CoS Applied to RoCE RNICs' { Get-NetAdapterRdma | get-netadapter -Physical -ErrorAction SilentlyContinue |? DriverProvider -match $using:rocematch }
$f = @($totalf_nonull)
$f += ,(new-namedblock 'Operational FlowControl Specs' { $_.OperationalFlowControl } -nullpass:$false)
$f += ,(new-namedblock 'Operational Port/Protocol Classification Specs' { $_.OperationalClassifications } -nullpass:$false)
$f += ,(new-namedblock 'Operational CoS Traffic Classes' { $_.OperationalTrafficClasses } -nullpass:$false)
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
#>
}
###
if (Get-StorageFileServer |? SupportsContinuouslyAvailableFileShare) {
$t = new-namedblock 'SMB Server CA FS Scope Net Interface Symmetry Check' {
$fs = Get-StorageFileServer |? SupportsContinuouslyAvailableFileShare
if ($fs) {
Get-SmbServerNetworkInterface |? ScopeName -eq $fs.FriendlyName
} else {
$null
}
}
$f = @()
$f = ,(new-namedblock 'Rdma Capable' { $_.RdmaCapable } -nullpass:$(-not $rdma))
$j += start-job -InitializationScript $fns -Name $t.name { do-clustersymmetry $using:t $using:f }
}
# consume job results
$j | display-jobs
@@ -0,0 +1,132 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
[switch]$disableintegrity = $false,
[switch]$renamecsvmounts = $false,
[switch]$movecsv = $true,
[switch]$movevm = $true,
[switch]$shiftcsv = $false
)
$csv = get-clustersharedvolume
# handle restore cases by mapping the csv to the friendly name of the volume
# don't rely on the csv name to contain this data
$vh = @{}
Get-Volume |? FileSystem -eq CSVFS |% { $vh[$_.Path] = $_ }
$csv |% {
$v = $vh[$_.SharedVolumeInfo.Partition.Name]
if ($v -ne $null) {
$_ | Add-Member -NotePropertyName VDName -NotePropertyValue $v.FileSystemLabel
}
}
if ($disableintegrity) {
$csv |% {
dir -r $_.SharedVolumeInfo.FriendlyVolumeName | Set-FileIntegrity -Enable:$false -ErrorAction SilentlyContinue
}
}
if ($renamecsvmounts) {
$csv |% {
if ($_.SharedVolumeInfo.FriendlyVolumeName -match 'Volume\d+$') {
ren $_.SharedVolumeInfo.FriendlyVolumeName $_.VDName
}
}
}
function move-csv(
$rehome = $true,
$shift = $false
)
{
if ($shift) {
write-host -fore Yellow Shifting CSV owners
# rotation order (n0 -> n1, n1 -> n2, ... nn -> n0)
$nodes = (Get-ClusterNode |? State -eq Up | sort -Property Name).Name
$nh = @{}
foreach ($i in 1..($nodes.Length-1)) {
$nh[$nodes[$i-1]] = $nodes[$i]
}
$nh[$nodes[$nodes.Length-1]] = $nodes[0]
Get-ClusterNode |% {
$node = $_.Name
$csv |? VDName -match "$node(?:-.+){0,1}" |% {
$_ | Move-ClusterSharedVolume $nh[$_.OwnerNode.Name]
}
}
} elseif ($rehome) {
# write-host -fore Yellow Re-homing CSVs
# move all csvs named by node names back to their named node
get-clusternode |? State -eq Up |% {
$node = $_.Name
$csv |? VDName -match "$node(?:-.+){0,1}" |? OwnerNode -ne $node |% { $_ | Move-ClusterSharedVolume $node }
}
}
}
if ($shiftcsv) {
# shift rotates all csvs node ownership by one node, in lexical order of
# current node owner name. this is useful for forcing out-of-position ops.
move-csv -shift:$true
} elseif ($movecsv) {
# move puts all csvs back on their home node
move-csv -rehome:$true
}
if ($movevm) {
icm (Get-ClusterNode |? State -eq Up) {
Get-ClusterGroup |? GroupType -eq VirtualMachine |% {
if ($_.Name -like "vm-*-$env:COMPUTERNAME-*") {
if ($env:COMPUTERNAME -ne $_.OwnerNode) {
write-host -ForegroundColor yellow moving $_.name $_.OwnerNode '->' $env:COMPUTERNAME
# the default move type is live, but live does not degenerately handle offline vms yet
if ($_.State -eq 'Offline') {
Move-ClusterVirtualMachineRole -Name $_.Name -Node $env:COMPUTERNAME -MigrationType Quick
} else {
Move-ClusterVirtualMachineRole -Name $_.Name -Node $env:COMPUTERNAME
}
} else {
# write-host -ForegroundColor green $_.name is on $_.OwnerNode
}
}
}
}
}
@@ -0,0 +1,36 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
while ($true) {
$empty = dir C:\ClusterStorage\collect\control\result |? Length -eq 0
if ($empty) { continue }
sleep 20
break
}
write-host -fore green Done
@@ -0,0 +1,531 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
$Cluster = ".",
$SampleInterval = 2,
[ValidateSet("CSV FS","SSB Cache","SBL","SBL Local","SBL Remote","SBL*","S2D BW","Hyper-V LCPU","SMB SRV","SMB Transport","*")]
[string[]] $Sets = "CSV FS",
$Log = $null
)
if ($null -ne $log) {
del -Force $log -ErrorAction SilentlyContinue
}
function write-log(
[string[]] $str
)
{
if ($null -ne $log) {
$str |% {
"$(get-date) $_" | Out-File -Append -FilePath $log -Width 9999 -Encoding ascii
}
}
}
# display name
# ctr name
# display order
# format string
# scalar multiplier
class CounterColumn {
[string] $displayname
[string] $setname
[string[]] $ctrname
[int] $width
[string] $fmt
[decimal] $multiplier
[ValidateSet("Average","AverageAggregate","Sum")][string] $aggregate
[boolean] $divider
CounterColumn(
[string] $displayname,
[string] $setname,
[string[]] $ctrname,
[int] $width,
[string] $fmt,
[decimal] $multiplier,
[string] $aggregate,
[boolean] $divider
)
{
$this.displayname = $displayname
$this.setname = $setname
$this.ctrname = $ctrname
$this.width = $width
$this.fmt = $fmt
$this.multiplier = $multiplier
$this.aggregate = $aggregate
$this.divider = $divider
if ($this.width -lt $this.displayname.length + 1) {
$this.width = $this.displayname.length + 1
}
}
}
class CounterColumnSet {
[CounterColumn[]] $columns
[string[]] $counters
[string] $topfmt
[string] $linfmt
[string] $name
[string] $totalline
[string[]] $nodelines
CounterColumnSet($name)
{
$this.columns = $null
$this.name = $name
}
[void] Add([CounterColumn] $c)
{
$this.columns += $c
}
[void] Seal()
{
# assemble the top-line fmt and per-line fmt strings
# top-line is just strings/width
# per-line adds the (likely numeric) format specifier
$n = 1
$this.topfmt = $this.linfmt = "{0,-16}"
foreach ($col in $this.columns) {
$str = ''
if ($col.divider) {
$str += '| '
}
$str += "{$n,-$($col.width)"
$this.topfmt += "$str}"
$this.linfmt += "$($str):$($col.fmt)}"
$n += 1
}
# assemble the list of counter instances that will be needed
# note that some may be internally aggregated (i.e., total = read + write)
# in cases where a counterset does not provide an explicit total
$this.counters = ($this.columns |% {
$s = $_.setname
$_.ctrname |% { "\$($s)(_Total)\$($_)" }
} | group -NoElement).Name
}
[void] DisplayPre(
[hashtable] $samples,
[hashtable] $psamples
)
{
# aggregate each column across all live sampled nodes
$this.totalline = $this.linfmt -f $(
"Total"
foreach ($col in $this.columns) {
# average aggreate doesn't work across nodes if the base is not consistent
# for instance: cannot average latency safely, but can average cpu utilization
if ($col.aggregate -ne 'Average' -or $col.aggregate -eq 'AverageAggregate') {
$(foreach ($node in $psamples.keys) {
get-samples $psamples $node $col
}) | get-aggregate $col
} else {
$null
}
}
)
# individual nodes
# flags downed/non-responsive nodes by noting which are not
# present in the processed samples
$this.nodelines = foreach ($node in $samples.keys | sort) {
if ($psamples.ContainsKey($node)) {
$this.linfmt -f $(
$node
foreach ($col in $this.columns) {
$s = get-samples $psamples $node $col
if ($null -ne $s) {
$a = $s | get-aggregate $col
$a
} else {
"-"
}
}
)
} else {
$this.topfmt -f $(
$node
0..($this.columns.count - 1) |% { "-" }
)
}
}
}
[void] Display()
{
write-host ($this.topfmt -f (,$this.name + $this.columns.displayname))
write-host -fore green $this.totalline
$this.nodelines |% { write-host $_ }
}
}
function get-aggregate(
[CounterColumn] $col
)
{
BEGIN {
$n = 0
$v = 0
}
PROCESS {
$n += 1
$v += $_
}
END {
if ($n -gt 0) {
switch -wildcard ($col.aggregate) {
'Sum' {
#write-host $col.displayname $col.multipler $v
$col.multiplier * $v
}
'Average*' {
#write-host $col.displayname $col.multiplier $v $n
$col.multiplier * $v / $n
}
}
} else {
$null
}
}
}
# get samples out of per-node hash of ctr hashes
function get-samples(
[hashtable] $h,
[string] $node,
[CounterColumn] $col
)
{
foreach ($i in $col.ctrname) {
$k = "$($col.setname)+$($i)"
if ($h.ContainsKey($node)) {
if ($h[$node].ContainsKey($k)) {
$h[$node][$k]
} else {
write-log "missing $node[$k] : $($h[$node].Keys.Count) total keys"
}
} else {
write-log "missing $node"
}
}
}
$allctrs = @()
###
$c = [CounterColumnSet]::new("CSV FS")
$c.Add([CounterColumn]::new("IOPS", "Cluster CSVFS", @("Reads/sec","Writes/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Reads", "Cluster CSVFS", @("Reads/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Writes", "Cluster CSVFS", @("Writes/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("BW (MB/s)", "Cluster CSVFS", @("Read Bytes/sec","Write Bytes/sec"), 13, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("Read", "Cluster CSVFS", @("Read Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Write", "Cluster CSVFS", @("Write Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Read Lat (ms)", "Cluster CSVFS", @("Avg. sec/Read"), 15, '0.000', 1000, 'Average', $true))
$c.Add([CounterColumn]::new("Write", "Cluster CSVFS", @("Avg. sec/Write"), 8, '0.000', 1000, 'Average', $false))
$c.Add([CounterColumn]::new("Read QAvg", "Cluster CSVFS", @("Avg. Read Queue Length"), 11, '0.000', 1, 'Average', $true))
$c.Add([CounterColumn]::new("Write", "Cluster CSVFS", @("Avg. Write Queue Length"), 8, '0.000', 1, 'Average', $false))
$c.Seal()
$allctrs += $c
###
$c = [CounterColumnSet]::new("SSB Cache")
$c.Add([CounterColumn]::new("Hit/Sec", "Cluster Storage Hybrid Disks", @("Cache Hit Reads/Sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Miss/Sec", "Cluster Storage Hybrid Disks", @("Cache Miss Reads/Sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Remap/Sec" ,"Cluster Storage Cache Stores", @("Page ReMap/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Cache (MB/s)", "Cluster Storage Hybrid Disks", @("Cache Populate Bytes/sec","Cache Write Bytes/sec"), 13, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("RdPop", "Cluster Storage Hybrid Disks", @("Cache Populate Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("WrPop", "Cluster Storage Hybrid Disks", @("Cache Write Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Destage (MB/s)", "Cluster Storage Cache Stores", @("Destage Bytes/sec"), 15, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("Update", "Cluster Storage Cache Stores", @("Update Bytes/sec"), 7, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Total (Pgs)", "Cluster Storage Cache Stores", @("Cache Pages"), 11, '0.00E+0', 1, 'Sum', $true))
$c.Add([CounterColumn]::new("Standby", "Cluster Storage Cache Stores", @("Cache Pages StandBy"), 9, '0.00E+0', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("L0", "Cluster Storage Cache Stores", @("Cache Pages StandBy L0"), 9, '0.00E+0', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("L1", "Cluster Storage Cache Stores", @("Cache Pages StandBy L1"), 9, '0.00E+0', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("L2", "Cluster Storage Cache Stores", @("Cache Pages StandBy L2"), 9, '0.00E+0', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Dirty", "Cluster Storage Cache Stores", @("Cache Pages Dirty"), 9, '0.00E+0', 1, 'Sum', $false))
$c.Seal()
$allctrs += $c
###
foreach ($subset in '','Local','Remote') {
$name = 'SBL'
$prefix = ''
if ($subset.Length) {
$name += " $subset"
$prefix = "$($subset): "
}
$c = [CounterColumnSet]::new($name)
$c.Add([CounterColumn]::new("IOPS", "Cluster Disk Counters", @(($prefix + "Read/sec"),($prefix + "Writes/sec")), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Reads", "Cluster Disk Counters", @($prefix + "Read/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Writes", "Cluster Disk Counters", @($prefix + "Writes/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("BW (MB/s)", "Cluster Disk Counters", @(($prefix + "Read - Bytes/sec"),($prefix + "Write - Bytes/sec")), 13, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("Read", "Cluster Disk Counters", @($prefix + "Read - Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Write", "Cluster Disk Counters", @($prefix + "Write - Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Read Lat (ms)", "Cluster Disk Counters", @($prefix + "Read Latency"), 15, '0.000', 1000, 'Average', $true))
$c.Add([CounterColumn]::new("Write", "Cluster Disk Counters", @($prefix + "Write Latency"), 8, '0.000', 1000, 'Average', $false))
$c.Add([CounterColumn]::new("Read QAvg", "Cluster Disk Counters", @($prefix + "Read Avg. Queue Length"), 11, '0.000', 1, 'Average', $true))
$c.Add([CounterColumn]::new("Write", "Cluster Disk Counters", @($prefix + "Write Avg. Queue Length"), 8, '0.000', 1, 'Average', $false))
$c.Seal()
$allctrs += $c
}
###
$c = [CounterColumnSet]::new("SMB SRV")
$c.Add([CounterColumn]::new("IOPS", "SMB Server Shares", @("Data Requests/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Reads", "SMB Server Shares", @("Read Requests/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Writes", "SMB Server Shares", @("Write Requests/sec"), 12, '#,#', 1, 'Sum', $false))
$c.Add([CounterColumn]::new("Data BW (MB/s)", "SMB Server Shares", @("Data Bytes/sec"), 13, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("Read", "SMB Server Shares", @("Read Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Write", "SMB Server Shares", @("Write Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Total BW (MB/s)", "SMB Server Shares", @("Transferred Bytes/sec"), 13, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("Rcv", "SMB Server Shares", @("Received Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Snd", "SMB Server Shares", @("Sent Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Seal()
$allctrs += $c
##
$c = [CounterColumnSet]::new("S2D BW")
$c.Add([CounterColumn]::new("CSV (MB/s)", "Cluster CSVFS", @("Read Bytes/sec","Write Bytes/sec"), 10, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Read", "Cluster CSVFS", @("Read Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Write", "Cluster CSVFS", @("Write Bytes/sec"), 8 ,'#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("SBL (MB/s)", "Cluster Disk Counters", @("Read - Bytes/sec","Write - Bytes/sec"), 10, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("Read", "Cluster Disk Counters", @("Read - Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Write", "Cluster Disk Counters", @("Write - Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Cache (MB/s)", "Cluster Storage Hybrid Disks", @("Cache Hit Read Bytes/sec","Cache Write Bytes/sec"), 12, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("Read", "Cluster Storage Hybrid Disks", @("Cache Hit Read Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Write", "Cluster Storage Hybrid Disks", @("Cache Write Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Disk (MB/s)", "Cluster Storage Hybrid Disks", @("Disk Read Bytes/sec","Disk Write Bytes/sec"), 11, '#,#', 0.000001, 'Sum', $true))
$c.Add([CounterColumn]::new("Read", "Cluster Storage Hybrid Disks", @("Disk Read Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Add([CounterColumn]::new("Write", "Cluster Storage Hybrid Disks", @("Disk Write Bytes/sec"), 8, '#,#', 0.000001, 'Sum', $false))
$c.Seal()
$allctrs += $c
##
$c = [CounterColumnSet]::new("Hyper-V LCPU")
$c.Add([CounterColumn]::new("Logical Total%", "Hyper-V Hypervisor Logical Processor", @("% Total Run Time"), 8, "0.00", 1, 'AverageAggregate', $false))
$c.Add([CounterColumn]::new("Guest%", "Hyper-V Hypervisor Logical Processor", @("% Guest Run Time"), 8, "0.00", 1, 'AverageAggregate', $false))
$c.Add([CounterColumn]::new("Hypervisor%", "Hyper-V Hypervisor Logical Processor", @("% Hypervisor Run Time"), 13, "0.00", 1, 'AverageAggregate', $false))
$c.Add([CounterColumn]::new("Root Total%", "Hyper-V Hypervisor Root Virtual Processor", @("% Total Run Time"), 12, "0.00", 1, 'AverageAggregate', $true))
$c.Add([CounterColumn]::new("Guest%", "Hyper-V Hypervisor Root Virtual Processor", @("% Guest Run Time"), 8, "0.00", 1, 'AverageAggregate', $false))
$c.Add([CounterColumn]::new("Hypervisor%", "Hyper-V Hypervisor Root Virtual Processor", @("% Hypervisor Run Time"), 12, "0.00", 1, 'AverageAggregate', $false))
$c.Add([CounterColumn]::new("Remote%", "Hyper-V Hypervisor Root Virtual Processor", @("% Remote Run Time"), 7, "0.00", 1, 'AverageAggregate', $false))
$c.Seal()
$allctrs += $c
##
$c = [CounterColumnSet]::new("SMB Transport")
$c.add([CounterColumn]::new("Read IOPS", "SMB Client Shares", @("Read Requests/sec"), 11, "#,#", 1, 'Sum', $false))
$c.add([CounterColumn]::new("Write", "SMB Client Shares", @("Write Requests/sec"), 8, "#,#", 1, 'Sum', $false))
$c.add([CounterColumn]::new("RDMA Read", "SMB Client Shares", @("Read Requests transmitted via SMB Direct/sec"), 11, "#,#", 1, 'Sum', $true))
$c.add([CounterColumn]::new("Write", "SMB Client Shares", @("Write Requests transmitted via SMB Direct/sec"), 8, "#,#", 1, 'Sum', $false))
$c.Seal()
$allctrs += $c
##
if ($Sets.Count -eq 1 -and $Sets[0] -eq '*') {
$ctrs = $allctrs
} else {
$ctrs = $Sets |% {
$s = $_
$allctrs |? { $_.name -like $s } # allows the SBL* wildcard
}
}
function start-sample(
[CounterColumnSet[]] $ctrs,
[int] $SampleInterval
)
{
# clear any previous job instance
Get-Job -Name watch-cluster -ErrorAction SilentlyContinue | Stop-Job
Get-Job -Name watch-cluster -ErrorAction SilentlyContinue | Remove-Job
# flatten list of counters and uniquify for the total counter set
# some display counter sets may repeat specific values (which is fine)
$counters = ($ctrs.counters |% { $_ |% { $_ }} | group -NoElement).Name
icm -AsJob -JobName watch-cluster (Get-ClusterNode -Cluster $Cluster) {
# extract countersamples, the object does not survive transfer between powershell sessions
# extract as a list, not as the individual counters
get-counter -Continuous -SampleInterval $using:SampleInterval $using:ctrs.counters |% {
,$_.countersamples
}
}
}
# start the first sample job and allow frame draw the first time through
$j = start-sample $ctrs $SampleInterval
$downtime = $null
$skipone = $false
$loops = 0
$restart = $false
# hash of most recent samples/node
$samples = @{}
Get-ClusterNode -Cluster $Cluster |% { $samples[$_.Name] = $null }
while ($true) {
if (-not $restart) {
Start-Sleep -Seconds $SampleInterval
# sleep again if needed to prime the sample pipeline;
# there are no samples if we just restarted the sampling jobs
if ($skipone) {
$skipone = $false
continue
}
# receive updates into the per-node hash
foreach ($child in $j.ChildJobs) {
$samples[$child.Location] = $child | receive-job -ErrorAction SilentlyContinue
}
# null out downed nodes and remember first time we saw one drop out
$down = 0
$j.ChildJobs |? State -ne Running |% {
$samples[$_.Location] = $null
$down += 1
}
if ($down -and $null -eq $downtime) {
$downtime = get-date
}
# if everything is down, we will attempt restart
if ($down -eq $j.ChildJobs.Count) {
$restart = $true
break
}
}
# if explicit restart is required, or it has been 30 seconds with a downed node, restart the jobs to retry
if ($restart -or ($null -ne $downtime -and ((get-date)-$downtime).totalseconds -gt 30)) {
$j | stop-job
$j | remove-job
$j = start-sample $ctrs $SampleInterval
# force gc to clear out accumulated job state quickly
[system.gc]::Collect()
$downtime = $null
$skipone = $true
$restart = $false
continue
}
# now process samples into per-node hashes of set/ctr containing lists of the
# cooked values acrosss the (possible) multiple instances
$psamples = @{}
foreach ($node in $samples.keys) {
if ($samples[$node]) {
$nsamples = @{}
# flatten samples - if we are lagging, we'll have a list
# of consecutive (increasing by timestamp) samples
# we could try to be more efficient by dumping all but the
# final sample, but later ...
$samples[$node] |% { $_ } |% {
($setinst,$ctr) = $($_.path -split '\\')[3..4]
$set = ($setinst -split '\(')[0]
$k = "$set+$ctr"
$nsamples[$k] = $_.cookedvalue
}
$psamples[$node] = $nsamples
}
}
# post-process the samples into the counterset, then clear and dump
$ctrs.DisplayPre($samples, $psamples)
Clear-Host
$drawsep = $false
$ctrs |% {
if ($drawsep) {
write-host -fore Green $('-'*20)
}
$drawsep = $true
$_.Display()
}
# restart the jobs every so many loops to prevent resource growth
$loops += 1
if ($loops -gt 100) {
$loops = 0
$restart = $true
}
}
@@ -0,0 +1,210 @@
<#
DISKSPD - VM Fleet
Copyright(c) Microsoft Corporation
All rights reserved.
MIT License
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
#>
param(
$ComputerName = $env:COMPUTERNAME,
$SampleInterval = 2
)
function div-to-width(
[int] $div
)
{
# 0 - 100 scale
# ex: 4 -> 100/4 = 25 buckets + 1 more for == 100
1+100/$div
}
function center-pad(
[string] $s,
[int] $width
)
{
if ($width -le $s.length) {
$s
} else {
(' ' * (($width - $s.length)/2) + $s)
}
}
function get-legend(
[int] $width,
[int] $div
)
{
# now produce the scale, a digit at a time in vertical orientation
# at each multiple of 10% which aligns with a measurement bucket.
# the width is the width of the measured values
#
# 0 5 1
# 0 0
# 0
$lines = @()
$lines += ,('-' * $width)
foreach ($dig in 0..2) {
$o = foreach ($pos in 0..($width - 1)) {
$val = $pos * $div
if ($val % 10 -eq 0) {
switch ($dig) {
0 { if ($val -eq 100) { 1 } else { $val / 10 }}
1 { if ($val -ne 0) { 0 } else { ' ' }}
2 { if ($val -eq 100) { 0 } else { ' ' }}
}
} else { ' ' }
}
$lines += ,($o -join '')
}
# trailing comments (horizontal scale name)
'Percent CPU Utilization' |% {
$lines += ,(center-pad $_ $width)
}
$lines
}
# these are the valid divisions, in units of percentage width.
# they must evenly divide 100% and either 10% or 20% for scale markings.
# determine which is the best fit based on window width.
$div = 0
foreach ($i in 1,2,4,5) {
if ((div-to-width $i) -le [console]::WindowWidth) {
$div = $i
break
}
}
# if nothing fit, widen to 4% divisions
if ($div -eq 0) {
$div = 4
}
$width = div-to-width $div
# get the constant legend; use the remaining height for the vertical cpu core bars.
# note total height includes variable label line at bottom (instance + aggregagte)
$legend = get-legend $width $div
$clip = [console]::WindowHeight - $legend.Count - 1
# insist on a clip no lower than 10
if ($clip -lt 10) {
$clip = 10
}
# set window and buffer size simultaneously so we don't have extra scrollbars
cls
[console]::SetWindowSize($width,$clip + $legend.Count + 1)
[console]::BufferWidth = [console]::WindowWidth
[console]::BufferHeight = [console]::WindowHeight
# scale divisions at x%
# this should evenly divide 100%
$m = [array]::CreateInstance([int],$width)
# which processor counterset should we use?
# pi is only the root partition if hv is active
# hvlp is the host physical processors when hv is active
# via ctrs, hv is active iff hvlp is present and has multiple instances
$cset = get-counter -ComputerName $ComputerName -ListSet 'Hyper-V Hypervisor Logical Processor' -ErrorAction SilentlyContinue
if ($cs -ne $null -and $cs.CounterSetType -eq [Diagnostics.PerformanceCounterCategoryType]::MultiInstance) {
$cpuset = '\Hyper-V Hypervisor Logical Processor(*)\% Total Run Time'
} else {
$cpuset = '\Processor Information(*)\% Processor Time'
}
# processor performance counter (turbo/speedstep)
$ppset = '\Processor Information(_Total)\% Processor Performance'
while ($true) {
# reset measurements & the lines to output
$lines = @()
foreach ($i in 0..($m.length - 1)) {
$m[$i] = 0
}
# avoid remoting for the local case
if ($ComputerName -eq $env:COMPUTERNAME) {
$samp = (get-counter -SampleInterval $SampleInterval -Counter $cpuset,$ppset).Countersamples
} else {
$samp = (get-counter -SampleInterval $SampleInterval -Counter $cpuset,$ppset -ComputerName $ComputerName).Countersamples
}
# get all specific instances and count them into appropriate measurement bucket
$samp |% {
if ($_.Path -like "*$ppset") { # scaling factor for total utility
$pperf = $_.CookedValue/100
} elseif ($_.InstanceName -notlike '*_Total') { # per cpu: ignore total and per-numa total
$m[[math]::Floor($_.CookedValue/$div)] += 1
} elseif ($_.InstanceName -eq '_Total') { # get total
$total = $_.CookedValue
}
}
# work down the veritical altitude of each strip, starting at vclip
$altitude = $clip
do {
$lines += ,($($m |% {
# if we are potentially clipped, handle
if ($altitude -eq $clip) {
# clipped?
# unclipped but at clip?
# nothing, less than altitude
if ($_ -gt $altitude) { 'x' }
elseif ($_ -eq $altitude) { '*' }
else { ' ' }
} else {
# normal
# >=, output
# <, nothing
if ($_ -ge $altitude) { '*' }
else { ' ' }
}
}) -join '')
} while (--$altitude)
cls
write-host -NoNewline ($lines + $legend -join "`n")
write-host -NoNewLine ("`n" + (center-pad ("{2} Total: {0:0.0}% Normalized: {1:0.0}%" -f $total,($total*$pperf),$ComputerName) $width))
# move the cursor to indicate average utilization
# column number is zero based, width is 1-based
[console]::SetCursorPosition([math]::Floor(($width - 1)*$total/100),[console]::CursorTop-$legend.Count)
}