Design Flow for Embedded System Device Driver

Development and Verification

Ross Dickson Jason Andrews Jakob Engblom

Virtutech, Inc Cadence Virtutech, AB
dickson@virtutech.com jasona@cadence.com jakob@virtutech.com
 Context and Motivation

• Bugs in device drivers are common

– The majority of embedded system bugs are attributed to software
• Possibly because software is less expensive to fix
• Cadence has proven utility of constrained random testing in the
Incisive Software Extensions to provoke bugs
• Virtutech has proven the efficiency of the Simics Virtual Platform to
nonintrusively isolate and identify bugs
• A combined environment provides the embedded device driver
writer with an improved environment to provoke, isolate, and
identify bugs in device drivers for embedded systems
• A driver for a hardware accelerator for the Rule 30 Cellular
Atomata on a virtual Freescale MPC8641D SoC running Linux
version 2.6.23 is used as a running example

2 Copyright @ 2009 Virtutech, Cadence

 Software Setup

Virtual MPC8641D Board

Virtual MPC8641D Board
MPC8641D SoC

Test program
Other
Busybox exercising
program
device driver

Device
Virtual serial driver under
console test
UART Linux 2.6.23 kernel
CPU cores 0 and 1
Ethernet
Virtual
network
MemCtrl Timers
Device model
RAM MPIC
under test
PCI PCIe

Simics

Host operating system

Host computer

3 Copyright @ 2009 Virtutech, Cadence

 Verification Planning

• Determine the items that must be exercised and

monitored
– Software
• the software functions to be called
• their input parameters, and their outputs
• Variables and data structures to be monitored
– Hardware
• Registers modified
• Interrupts triggered
• Examples:
– Software: device driver functions
• open and write
– Software: kernel functions
• interrupt handlers.

4 Copyright @ 2009 Virtutech, Cadence

 Rule 30 Verification Plan
Create
CreateVerification
VerificationPlan
Plan

Describe
Software to be
Exercised

• Done using Word

• Saved as XML file

5 Copyright @ 2009 Virtutech, Cadence

 Introduction to Coverage Driven Verification

Automatic Coverage Coverage-Driven

+ =
Generation Measurement Verification

Constrained Metrics
Randomization

Define Verification Goals and Let Machines do the Work

6 Copyright @ 2009 Virtutech, Cadence

 Directed Testing

An engineer writes directed test for each item in Test Plan:

Design

B1 B1 B1

B1

B2

B1 B3

It’s work determine and follow the paths

It’s work to verify the goal was reached
Redo if design
Poor coverage of non-goal scenarios changes

7 Copyright @ 2009 Virtutech, Cadence

 …so directed tests are not enough

• Tedious to write and maintain

– Tens of thousands of lines of test code
– Reuse to get to state quickly causes poor new coverage
– Design spec changes cause delays

• Difficult to think of all required verification cases

– Many different ways to reach a certain state

• Can't implement all identified tests in test plan within

project schedule

8 Copyright @ 2009 Virtutech, Cadence

 Where did CDV Come From?

• Automated Constrained
Can the same Random
technology Stimulus
be used for Generation
embedded software?
• Coverage Collection (Functional, Assertion, Code)
• Can
Independent Coverage
it be usedAssertion & Data Checking
by an embedded software verification engineer?

Coverage Scoreboard
Scoreboard
Automated Data
Data Check
Check
Generation
seed
23098432
38u48932
Self
Self Checking
Checking Self
Self Checking
Checking
23432239
17821961 Monitors
Monitors Monitors
Monitors
10932893
20395483
18902904
23843298
23432432
24324322
55252255
09273822
13814791 Stimulus
Stimulus
stimulus
4098e092 stimulus
Linux
Linux
Automated 23432424 Scenarios
Scenarios
Scenarios
24242355 Scenarios
Device
Test Scenario 25262622 BFM
test
BFM
test sw
sw Device
26452454
24524522
Driver
Driver

9 Copyright @ 2009 Virtutech, Cadence

 Software Setup with Coverage Driven
Verification

Set test
Virtual MPC8641D Board
Virtual MPC8641D Board program
MPC8641D SoC parameters
Coverage
Test program
Other
Driven
Busybox exercising Verification
program
device driver

…
Device e
Virtual serial driver under Helper
test Tests
console Observe
UART Linux 2.6.23 kernel
hardware/ Verification
CPU cores 0 and 1 software Planning &
Ethernet interface
Virtual Management
network
MemCtrl Timers
Device model Change
RAM MPIC
under test hardware
PCI PCIe
parameters

vPlan
Simics ISX
Simics

Host operating system

Host computer

10 Copyright @ 2009 Virtutech, Cadence

 Rule 30 Driver Execution and Coverage
Analysis
Run
Runsequences
sequencesand
andcollect
collect
coverage
coverage

Generated
Generatedsequences
sequenceswith
with
random data
random data

Achieved
AchievedCoverage
Coverage

11 Copyright @ 2009 Virtutech, Cadence

 Rule 30 Driver Regression and Results

Automated
AutomatedExecution
Executionand
andAnalysis
Analysis

• Reads XML file

• Analyzes Results

12 Copyright @ 2009 Virtutech, Cadence

 Rule 30 Results

• Coverage
– Ability to measure hardware model parameters
• time_to_result for algorithm completion time to make sure
Linux driver is timing independent

– Ability to measure generated stimulus

• Length of rule30 line

– Ability to measure coverage on software

• Mode of device open() system call: read, write, read/write
• Return values from system calls

– Ability to cross all of the above and find out, “Did I test short
line length with fast hardware response time?”

13 Copyright @ 2009 Virtutech, Cadence

 Rule 30 Results

• Bug Hunting: hitting corner cases not thought writing

manual tests
– Operations out of order
– Parameters outside of expected values
– Memory corruption: “serial8250: too much work for irq42”
• Learning
– Understanding driver behavior
– When does a read block waiting for results
– How interrupts work

14 Copyright @ 2009 Virtutech, Cadence

 Repeatability and Reverse Debugging
On
On hardware,
hardware, only
only
• Repeat any run trivially some
some runs
reproduce
runs
reproduce an
an error
error
– No need to rerun and hope
for bug to reoccur
• Stop & go back in time
– Instead of rerunning program
from start
– Breakpoints & watchpoints
backwards in time
– Investigate exactly what
happened this time
• This control and reliable
repeatability is very powerful
for parallel code!

On
On virtual
virtual hardware,
hardware,
debugging
debugging isis much
much
easier
easier

15 Copyright @ 2009 Virtutech, Cadence

Thank You
 What Virtutech Does

• Provider of Simics: a high-performance, high fidelity, full system

simulator

– High Performance – fast enough to run real software loads (typically

100’s of MIPS, up to multiple GIPS)

– High Fidelity – run full production software, including firmware, device

drivers, hypervisor, RTOS/OS, application software

– Full System – simulate entire systems, not just processor cores, or

SoCs, or boards
• Complete machines, networks, backplanes

• The true value of Simics is through enablement of process change:

Virtualized Software Development

17 Copyright @ 2009 Virtutech, Cadence

 Traditional Product/Project Development

Pre-Silicon Post-Silicon
Testing Testing
Production,
System
Manufacture
Integration
start time and duration
Hardware
Development

pes
t y Marketing /
roto
P Feature
Validation
Software
Product Development
Specification System
Test

End User
Documentation
& Translation
Board Bring-up Platform Application
Development Development

18 Copyright @ 2009 Virtutech, Cadence

 Agility, Higher Quality, Faster time to Market

True Iterative
Production,
Development
Manufacture
System
Bring-up

Hardware
Development
Marketing /
Feature
Software Validation
Development
Product
Specification System
Test

End User
Documentation
& Translation

19 Copyright @ 2009 Virtutech, Cadence

 What is a Virtual Platform?

• A piece of software
• Running on a regular PC,
server, or workstation
• Functionally identical to a
particular hardware
• Runs the same software as
the physical hardware system

Virtual HW

20 Copyright @ 2009 Virtutech, Cadence