The basic approach we are taking is as follows:
1) Input video signal into NI PXI system
2) Analyze the active video portion of the signal
3) Detect which “notes” appear
4) When “notes” are detected, send appropriate control signal to robot.
We will be analyzing a standard NTSC 480i composite video signal from the PlayStation2. Each line of video contains 640 pixels. The brightness of a pixel is determined by the signal’s voltage amplitude. By parsing the signal waveform of a single line into subsets the width of our detection area, we can monitor when a “note” passes through those pixels. Since the “notes” are the brightest images on the screen we can detect them by setting trigger points. If the signal voltage of our waveform subset rises above a trigger value, we know a “note” was detected. Once a detection is made, we enqueue a control signal and the send it to the robot after a specific delay (determined by the tempo of the song).
The image below describes the anatomy of a single line of video. We focus on the active video region. This waveform is divided into subsets that correspond to locations on the screen.