Test Methodology
The YES Certification test kit is set up to accommodate diverse types of hardware specializations (e.g., Network Server). These specializations are designated by test projects within the YES TestConsole. Each test project contains a unique set of tests chosen specifically for each project. These tests are then grouped into test suites that are designed to make certification testing flow more smoothly.
Test projects can be specialized further by tailoring the scope of the test suites to better suit the product type. For example, a Workstation project can be further specialized for a laptop or tablet by answering Yes to “Is this a laptop?” This is all handled within the TestConsole and does not require a custom test plan to be created ahead of time.
Although each project is designed for a specific product workload, some tests may be used across different projects where applicable.
Network Server
Manual Test Suite
The Manual Test Suite contains tests that require user interaction to either setup or to run each test. These tests must be run individually one after the other.
Video Test Objective: Verify that the display supports screen saver and other functionality on the system and monitor combination.
Optical Verify/Write Test Objective: Verifies that the optical devices function correctly with the OS. YES Certification requires that the most capable functionality of each optical device be tested. The most capable function is always write capability.
Hibernate Test (optional) Objective: Verify that the OS can correctly put the machine into and recover from hibernation mode.
Sleep Test (optional) Objective: Verify that the OS can correctly put the machine into and recover from sleep mode.
Memory Persistence Test Objective: The Memory Persistent Test will test the ability of Persistent Memory modules (NVDIMM) to store and retain information in a non-volatile method. The server will be powered off and rebooted during the test.
Kdump Test Objective: Verify that the OS and hardware can correctly perform a kernel crash dump (kdump)locally, and/or via the network.
GPU Compute Test Overview: This test is designed to test Accelerator/GPU functionality of video adapters and embedded GPU chipsets. At this time only NVIDIA adapters and GPUs (graphics processing units) are tested. All other GPU technologies will result in a NOT APPLICABLE test result.
Automated Test Suite
The automated tests will run without any user interaction. These tests can be run individually or queued up as the entire suite. When launched as a suite, they will still run individually, in sequence.
CPU Frequency Test Objective: Verify that the OS can change and monitor CPUs that support multiple frequencies.
CPU Hot Plug Test Objective: Verify that the OS can change CPUs.
Memory Hot Plug Test (optional) Objective: Verify that the OS can change memory with this server.
Watchdog Timer Test Objective: Verify that the watchdog timer works correctly with the OS.
Firmware BIOS Test (optional) Objective: Verify that the BIOS/firmware interfaces properly with the OS.
Verify NIC Test Objective: Verify that the NIC configuration is correct including LAN speeds and IP addresses. This test must pass before starting the stress tests.
Verify Time Sync Setup Test Objective: Verify that the time is correctly synchronized between the SUT and the TC. This test must pass before starting the stress tests.
Stress Test Suite
The stress tests are run without any user interaction, but unlike the automated tests, they are run together as a group to test the system under high load conditions. All stress tests must start and run together. However, if a test fails, the problem can be resolved, and the test can be restarted within 30 minutes without having to restart the entire test suite.
Memory/CPU Test Objective: Verifies that the CPU's and memory function correctly under stress for an extended amount of time. This test does not require any setup.
Hard Disk/RAID Test Objective: The Hard Disk/RAID device test will detect and test all Hard Disks and/or hardware RAID devices which have a mountable partition.
NIC Test Objective: Verify that the NIC's function correctly under stress for an extended amount of time. The Verify Setup NIC tests are required to be run and pass prior to starting this test.
Time Sync Test Objective: Verify that the time remains synced between the SUT and the TC for the duration of the stress test suite. Also indicates that log file time stamps are consistent between the TC and SUT machines.
Optical Read Test Objective: Verifies that Optical devices function correctly with the OS.
USB Test Objective: The USB test will detect and test all USB devices which have a mountable partition. This test is intended to exercise 2 USB ports. If the SUT has 2 USB ports, then plug a USB device into each port.
Workstation
Manual Test Suite
The Manual Test Suite contains tests that require user interaction to either setup or to run each test. These tests must be run individually.
Video Test Objective: Verify that the display supports screen saver and other functionality on the system and monitor combination.
Speaker Test Objective: Verify that the sound card and speakers function properly with the OS.
Microphone Test Objective: Verify that the sound card and microphone function properly with the OS.
Optical Verify/Write Test Objective: Verifies that the optical devices function correctly with the OS. YES Certification requires that the most capable functionality of each optical device be tested. The most capable function is always write-capability.
Hibernate Test Objective: Verify that the OS can correctly put the machine into and recover from hibernation mode.
Sleep Test Objective: Verify that the OS can correctly put the machine into and recover from sleep mode.
Kdump Test Objective: Verify that the OS and hardware can correctly perform a kernel crash dump (kdump)locally, and/or via the network.
GPU Compute Test Overview: This test is designed to test Accelerator/GPU functionality of video adapters and embedded GPU chipsets. At this time only NVIDIA adapters and GPUs are tested. All other GPU technologies will result in a NOT APPLICABLE test result.
Automated Test Suite
The automated tests will run without any user interaction. These tests can be run individually or queued up as the entire suite. When launched as a suite, they will still run individually, in sequence.
CPU Frequency Test Objective: Verify that the OS can change and monitor CPUs that support multiple frequencies.
Fan Thermal Test Objective: Verify that the SUT can properly monitor the change of temperature and fan states (on/off)
CPU Throttling Test Objective: Verify that the OS can properly change the throttling settings on the CPUs.
Firmware BIOS Test (optional) Objective: Verify that the BIOS/firmware interfaces properly with the OS.
Verify NIC Test Objective: Verify that the NIC configuration is correct including LAN speeds and IP addresses. This test must pass before starting the stress tests.
Verify Time Sync Setup Test Objective: Verify that the time is correctly synchronized between the SUT and the TC. This test must pass before starting the stress tests.
Stress Test Suite
The stress tests are run without any user interaction, but unlike the automated tests, they are run together as a group to test the system under high load conditions. All stress tests must start within 30 minutes of each other, and run to duration together as a group.
Memory/CPU Test Objective: Verifies that the CPU and memory function correctly under stress for an extended amount of time. This test does not require any setup.
Hard Disk/RAID Test Objective: The Hard Disk/RAID device test will detect and test all Hard Disks and/or hardware RAID devices which have a mountable partition. The hardware RAID testing is for hardware RAID only, it is not intended for testing software RAID.
NIC Test Objective: Verify that the NIC's function correctly under stress for an extended amount of time. The Verify Setup NIC tests are required to be run and pass prior to starting this test.
Time Sync Test Objective: Verify that the time remains synced between the SUT and the TC for the duration of the stress test suite. Also indicates that log file time stamps are consistent between the TC and SUT machines.
Optical Read Test Objective: Verifies that Optical devices function correctly with the OS.
USB Test Objective: The USB test will detect and test all USB devices which have a mountable partition. This test is intended to exercise 2 USB ports. If the SUT has 2 USB ports, then plug a USB device into each port.
Laptop/Tablet
Manual Test Suite
The Manual Test Suite contains tests that require user interaction to either setup or to run each test. These tests must be run individually.
Brightness Test Objective: Verify that the setting of the display brightness interfaces properly with the OS.
Brightness Keys Test Objective: Verify that the keyboard can be used to set the screen brightness.
Lid Close Test Objective: Verify that closing and opening the laptop lid interfaces properly with the OS.
Battery Test Objective: Verify the OS can properly monitor the usage of the battery on the system.
Video Test Objective: Verify that the display supports screen saver and other functionality on the system and monitor combination.
Touchscreen Test Objective: Verify that the touchscreen functions correctly with the OS.
Speaker Test Objective: Verify that the sound card and speakers' interface properly with the OS.
Microphone Test Objective: Verify that the sound card and microphone interface properly with the OS.
Volume Keys Test Objective: Verify that the volume keys interface properly with the OS.
External Mouse Test Objective: Verify that the OS can properly detect and use an external mouse on the system.
External Keyboard Test Objective: Verify the OS can detect and use an external keyboard with the system.
External Monitor Test Objective: Verify that the external monitor functions correctly with the OS. Start the test with no external monitor connected.
Optical Verify/Write Test Objective: Verifies that the optical devices function correctly with the OS. YES Certification requires that the most capable functionality of each optical device be tested. The most capable function is always write-capability.
Hibernate Test Objective: Verify that the OS can correctly put the machine into and recover from hibernation mode.
Sleep Test Objective: Verify that the OS can correctly put the machine into and recover from sleep mode.
Kdump Test Objective: Verify that the OS and hardware can correctly perform a kernel crash dump (kdump)locally, and/or via the network.
GPU Compute Test Overview: This test is designed to test Accelerator/GPU functionality of video adapters and embedded GPU chipsets. At this time only NVIDIA adapters and GPUs are tested. All other GPU technologies will result in a NOT APPLICABLE test result.
Automated Test Suite
The automated tests will run without any user interaction. These tests can be run individually or queued up as the entire suite. When launched as a suite, they will still run individually, in sequence.
CPU Frequency Test Objective: Verify that the OS can change and monitor CPUs that support multiple frequencies.
Fan Thermal Test Objective: Verify that the SUT can properly monitor the change of temperature and fan states (on/off)
CPU Throttling Test Objective: Verify that the OS can properly change the throttling settings on the CPUs.
Firmware BIOS Test (optional) Objective: Verify that the BIOS/firmware interfaces properly with the OS.
Verify NIC Test Objective: Verify that the NIC configuration is correct including LAN speeds and IP addresses. This test must pass before starting the stress tests.
Verify Time Sync Setup Test Objective: Verify that the time is correctly synchronized between the SUT and the TC. This test must pass before starting the stress tests. Also indicates that log file time stamps are consistent between the TC and SUT machines.
Stress Test Suite
The stress tests are run without any user interaction, but unlike the automated tests, they are run together as a group to test the system under high load conditions. All stress tests must start and run together. However, if a test fails, the problem can be resolved, and the test can be restarted within 30 minutes without having to restart the entire test suite.
Memory/CPU Test Objective: Verifies that the CPU's and memory function correctly under stress for an extended amount of time. This test does not require any setup.
Hard Disk/RAID Test Objective: The Hard Disk/RAID device test will detect and test all Hard Disks and/or hardware RAID devices which have a mountable partition. The hardware RAID testing is for hardware RAID only, it is not intended for testing software RAID.
NIC Test Objective: Verify that the NIC's function correctly under stress for an extended amount of time. The Verify Setup NIC tests are required to be run and pass prior to starting this test.
Time Sync Test Objective: Verify that the time remains synced between the SUT and the TC for the duration of the stress test suite. Also indicates that log file time stamps are consistent between the TC and SUT machines.
Optical Read Test Objective: Verifies that Optical devices function correctly with the OS.
USB Test Objective: The USB test will detect and test all USB devices which have a mountable partition. This test is intended to exercise 2 USB ports. If the SUT has 2 USB ports, then plug a USB device into each port.
POS System
Manual Test Suite
The Manual Test Suite contains tests that require user interaction to either setup or to run each test. These tests must be run individually.
Battery Test Objective: Verify the OS can properly monitor the usage of the battery on the system.
Video Test Objective: Verify that the display supports screen saver and other functionality on the system and monitor combination.
Touchscreen Test Objective: Verify that the touchscreen functions correctly with the OS.
Speaker Test Objective: Verify that the sound card and speakers' interface properly with the OS.
Microphone Test Objective: Verify that the sound card and microphone interface properly with the OS.
Optical Verify/Write Test Objective: Verifies that the optical devices function correctly with the OS. YES Certification requires that the most capable functionality of each optical device be tested. The most capable function is always write-capability.
Hibernate Test Objective: Verify that the OS can correctly put the machine into and recover from hibernation mode.
Sleep Test Objective: Verify that the OS can correctly put the machine into and recover from sleep mode.
Kdump Test Objective: Verify that the OS and hardware can correctly perform a kernel crash dump (kdump)locally, and/or via the network.
Automated Test Suite
The automated tests will run without any user interaction. These tests can be run individually or queued up as the entire suite. When launched as a suite, they will still run individually, in sequence.
CPU Frequency Test Objective: Verify that the OS can change and monitor CPUs that support multiple frequencies.
Fan Thermal Test Objective: Verify that the SUT can properly monitor the change of temperature and fan states (on/off)
CPU Throttling Test Objective: Verify that the OS can properly change the throttling settings on the CPUs.
Firmware BIOS Test (optional) Objective: Verify that the BIOS/firmware interfaces properly with the OS.
Verify NIC Test Objective: Verify that the NIC configuration is correct including LAN speeds and IP addresses. This test must pass before starting the stress tests.
Verify Time Sync Setup Test Objective: Verify that the time is correctly synchronized between the SUT and the TC. This test must pass before starting the stress tests.
Stress Test Suite
The stress tests are run without any user interaction, but unlike the automated tests, they are run together as a group to test the system under high load conditions. All stress tests must start and run together. However, if a test fails, the problem can be resolved, and the test can be restarted within 30 minutes without having to restart the entire test suite.
Memory/CPU Test Objective: Verifies that the CPU's and memory function correctly under stress for an extended amount of time. This test does not require any setup.
Hard Disk/RAID Test Objective: The Hard Disk/RAID device test will detect and test all Hard Disks and/or hardware RAID devices which have a mountable partition. The hardware RAID testing is for hardware RAID only, it is not intended for testing software RAID.
NIC Test Objective: Verify that the NIC's function correctly under stress for an extended amount of time. The Verify Setup NIC tests are required to be run and pass prior to starting this test.
Time Sync Test Objective: Verify that the time remains synced between the SUT and the TC for the duration of the stress test suite. Also indicates that log file time stamps are consistent between the TC and SUT machines.
Optical Read Test Objective: Verifies that Optical devices function correctly with the OS.
USB Test Objective: The USB test will detect and test all USB devices which have a mountable partition. This test is intended to exercise 2 USB ports. If the SUT has 2 USB ports, then plug a USB device into each port.
Virtualization (KVM/Xen/Third-Party Hypervisor) Test Suite
Multiple VM (MVM) Stress Tests
The MVM Stress Test Suite is run to stress the virtualization host environment from the hardware up through the entire virtualization stack using multiple VMs (Virtual machines)
Memory/CPU Test Objective: Load the vCPU by testing memory and low memory conditions with the host in an over-committed vCPU environment.
Hard Disk/RAID Test Objective (optional): Test the virtual disk channel within each VM.
NIC Out Test Objective: Tests the network connection from the VM to the TC.
NIC In Test Objective: Tests the network connection from the TC to the VM.
Optical Read Test Objective: Verifies that Optical devices function correctly within a VM.
USB Test Objective (optional): The USB test will detect and test all USB devices which have a mountable partition within a VM.
Time Sync Test Objective: Verifies that the TC and VM clocks are in sync.
Clock Test Objective: Verifies that the CPU clock does not drift.
VM Maximum (VMM) Stress Tests – KVM/Xen Only
The VMM Stress Test Suite is run to stress the virtualization host environment from the hardware up through the entire virtualization stack using a single maximum memory VM.
Memory/CPU Test Objective: Load the vCPU by testing memory and low memory conditions.
Hard Disk/RAID Test Objective: Test all the hard disks and RAIDs in the system.
NIC Out Test Objective: Tests the network connection from the VM to the TC.
NIC In Test Objective: Tests the network connection from the TC to the VM.
Time Sync Test Objective (optional): Verifies that the TC and VM clocks are in sync.
Clock Test Objective (optional): Verifies that the CPU clock does not drift.